ChatGPT’s Answers To Software Engineering Questions Were 52% Incorrect – News18

OpenAI’s ChatGPT answered about 52 per cent software program engineering questions incorrectly, in line with a examine, elevating questions concerning the standard language fashions accuracy.

Despite ChatGPT’s reputation, there hasn’t been an intensive investigation into the standard and usefulness of its responses to software program engineering queries, stated researchers from the Purdue University within the US.

To deal with this hole, the group undertook a complete evaluation of ChatGPT’s replies to 517 questions from Stack Overflow (SO).

“Our examination revealed that 52 per cent of ChatGPT’s solutions comprise inaccuracies and 77 per cent are verbose,” the researchers wrote in the paper, not peer-reviewed and published on a pre-print site.

Importantly, the team found that 54 per cent of the time the errors were made due to ChatGPT not understanding the concept of the questions.

Even when it could understand the question, it failed to show an understanding of how to solve the problem, contributing to a high number of conceptual errors, they said.

Further, the researchers observed ChatGPT’s limitation to reasoning.

“In many cases, we saw ChatGPT give a solution, code, or formula without foresight or thinking about the outcome,” they stated.

“Prompt engineering and human-in-the-loop fine-tuning could be useful in probing ChatGPT to know an issue to some extent, however they’re nonetheless inadequate with regards to injecting reasoning into LLM. Hence it’s important to know the elements of conceptual errors in addition to repair the errors originating from the limitation of reasoning,” they added.

Moreover, ChatGPT also suffers from other quality issues such as verbosity, inconsistency, etc. Results of the in-depth manual analysis pointed to a large number of conceptual and logical errors in ChatGPT answers. The linguistic analysis results showed that ChatGPT answers are very formal, and rarely portray negative sentiments.

Nevertheless, users still preferred ChatGPT’s responses 39.34 per cent of the time due to its comprehensiveness and articulate language style.

“These findings underscore the need for meticulous error correction in ChatGPT while also raising awareness among users about the potential risks associated with seemingly accurate answers,” the researchers stated.

(This story has been edited by News18 employees and is revealed from a syndicated news company feed – IANS)