The Downside of ChatGPT for Coding, In Its Own Words

ChatGPT is disrupting human-machine interactions. OpenAI created the chatbot to enable people to interact with it conversationally and generate answers and content they ask for. Since ChatGPT’s free preview launch in November 2022, just about everyone has read ChatGPT news reports, heard a ChatGPT poem, or tried ChatGPT to “help” with homework. The development community has also tried ChatGPT for coding, with mixed reviews.

Improving ChatGPT for Coding

Developers who used ChatGPT shortly after its release were anxious to see if it could make development easier. Its training dataset, built from information on the internet, included code snippets and programming techniques, giving the model an understanding of syntax and best practices.

But just a few weeks after ChatGPT’s release, Meta Stack Overflow temporarily banned it. The company stated that the average rate of correct answers from ChatGPT was too low, and it was “substantially harmful to the site and to users who are asking and looking for correct answers.” Stack Overflow explained that some people were using the tool to try to answer questions without taking the time – or lacking the expertise – to verify them. Moreover, because ChatGPT is easy and fast to use, the volume created an overwhelming workload for moderators.

Additionally, developers that conducted early tests found errors. For example, one asking it to simulate AM modulation returned an expression that the developer had to be corrected, but asking ChatGPT a few days later resulted in a correct expression. Improved performance over time shouldn’t be a surprise, though. ChatGPT uses feedback on its outputs to refine them.

However, improving code quality and accuracy more quickly seems to be a priority for OpenAI, according to a report by Semafor. The company has hired hundreds of contractors to create data for its models to learn software engineering tasks. Semafor reports that a developer who took the coding test for OpenAI said they’re looking for developers who can explain their thought processes as well as write code to generate data to train its models.

OpenAI is the developer of Codex, launched in 2021, which translates natural language into code. The company trained that model with Microsoft’s GitHub data, and Microsoft uses it for Copilot, a service that helps generate code. Combined with ChatGPT’s language model, those capabilities could result in a tool that automates development with reliability and accuracy.

Can You Use ChatGPT for Coding Now?

So, although ChatGPT has the potential to save time and increase development efficiency and in light of feedback that it doesn’t always produce answers that developers expect, can you confidently use it? We decided to go to the source. We asked ChatGPT to list the cons of using it for development, and it generated this response:

1. 1. 1. Lack of Context Awareness: One of the biggest challenges of using ChatGPT for coding is that it lacks context awareness. This means that the generated code may not always be appropriate for the specific context in which it is being used. Developers need to be careful when using the model to generate code and ensure that the generated code meets their specific requirements and fits the context in which it is being used.
      2. Limited Creativity: While ChatGPT is capable of generating code that is syntactically correct and semantically meaningful, it is limited in its ability to generate creative or innovative code. The model is trained to generate code based on existing patterns and practices, and it is unlikely to generate new, unique solutions to complex problems.
      3. Need for Human Intervention: While ChatGPT can generate code quickly and accurately, it is still necessary for developers to review and validate the generated code. This is because the model by not always generate code that meets the specific requirements of a project or is appropriate for a particular context.
      4. Dependence on Training Data: The accuracy and reliability of ChatGPT for coding are directly tied to the quality and relevance of the training data used to train the model. If the model is trained on outdated or low-quality data, the generated code may not be accurate or relevant to current development practices.

The bottom line is that ChatGPT may be helpful in some situations, but, at least for now, the code requires a developer with expertise to verify and validate it. If ChatGPT hits the mark, it could save time. If it doesn’t, time savings could be negated. Worse, if a development team overlooks an error, it could mean more time and costs in the future.

What About the Future?

Generative artificial intelligence (GAI), natural language processing (NLP), and other technologies that make ChatGPT possible have advanced to the point where they are able to deliver value in a wide range of use cases. But remember that they are still in their early stages. ChatGPT for coding can provide value now and will likely improve over time, and new tools are undoubtedly on the horizon.

Stay tuned to the news on the subject – which is coming out rapidly – explore new options, and make the best decisions for your business.