Chatgpt seems to improvise when it is put through the old puzzle of Greek mathematics

The Chatbot of Artificial Intelligence, Chatgpt, seemed to improvise ideas and make mistakes like a student in a study that has restarted a mathematical challenge aged 2,400.

The experience, of two education researchers, asked the chatbot to solve a version of the problem “double the place” – a lesson described by Plato in about 385 BCE and, the document suggests, “perhaps the first documented experience in mathematical education”. The puzzle has triggered centuries of debate on the question of whether knowledge is latent in us, while waiting to be “recovered”, or something that we “generate” through experience and experiences lived.

The new study has explored a similar question on Chatgpt mathematics “knowledge” – as can be perceived by its users. The researchers wanted to know if it would solve the Plato problem using the knowledge that it has already “held” or by adapting its own solutions in an adaptive way.

Plato describes Socrates teacher to a boy without instruction how to double the area of ​​a square. At the beginning, the boy wrongly suggests double the length on each side, but Socrates finally leads him to understand that the sides of the new square should be of the same length as the diagonal of the original.

The researchers posed this problem to Chatgpt-4, first imitating Socrates’ questions, then deliberately introducing errors, requests and new variants of the problem.

Like other models of great language (LLMS), Chatgpt is formed on large text collections and generates responses by predicting the sequences of words learned during his training. The researchers expected him to manage their old Greek mathematical challenge by regurgizing his preexisting “knowledge” of the famous solution of Socrates. Instead, however, he seemed to improvise his approach and, at some point, also made a human type error.

The study was conducted by Dr. Nadav Marco, invited researcher at the University of Cambridge, and Andreas Stylianides, professor of mathematics at Cambridge. Marco is permanently based at the Hebrew University and David Yellin College of Education in Jerusalem.

Although they are cautious about the results, stressing that LLMs do not think like humans or “things”, Marco characterized Chatgpt behavior as “similar to the learner”.

“When we face a new problem, our instinct is often to try things according to our past experience,” said Marco. “In our experience, Chatgpt seemed to do something similar. Like a learner or a scholar, he seemed to offer his own hypotheses and solutions.”

Because Chatgpt is formed on the text and not on the diagrams, it tends to be lower with the type of geometric reasoning that Socrates used to double the square problem. Despite this, Plato’s text is so well known that the researchers expected the chatbot to recognize their questions and reproduced the solution of Socrates.

Curiously, he failed to do it. Invited to double the place, Chatgpt opted for an algebraic approach which would have been unknown during the time of Plato.

He then resisted attempts to make him make the boy’s mistake and stubbornly stuck in algebra even when the researchers complained that his response was an approximation. It was only when Marco and Stylianides told him that they were disappointed that, despite all his training, he could not provide an “elegant and exact” response, made Chatgpt produced the geometric alternative.

Despite this, Chatgpt has demonstrated a complete knowledge of Plato’s work when questioned on this subject. “If he had recalled only from memory, he would have almost certainly referenced the classic solution to build a new square on the diagonal of the original square,” said Stylianides. “Instead, he seemed to adopt his own approach.”

The researchers also posed a variant of the Plato problem, asking Chatgpt to double the area of ​​a rectangle while retaining its proportions. Even if this was now aware of their preference for geometry, Chatgpt stubbornly stuck in algebra. When he was in a hurry, he then wrongly said that, because the diagonal of a rectangle cannot be used to double its size, a geometric solution was not available.

The point on the diagonal is true, but a different geometric solution exists. Marco suggested that the chance that this false assertion came from the chatbot knowledge base was “missing”. Instead, Chatgpt seemed to improvise his answers according to their previous discussion on the square.

Finally, Marco and Stylianids asked him to double the size of a triangle. The cat returned to algebra once again, but after more incentive found a correct geometric response.

Researchers emphasize the importance of not interpreting these results, as they could not scientifically observe Chatgpt coding. From the point of view of their digital experience as users, which emerged at this surface level was a mixture of data recovery and reasoning on the fly.

They compare this behavior to the educational concept of a “proximal development zone” (ZPD) – the gap between what a learner already knows and what they could possibly know with support and advice. Perhaps, according to them, the generator AI has a metaphorical “chat ZPD”: in some cases, it will not be able to solve the problems immediately but could do so with incentive.

The authors suggest that working with Chatgpt in your ZPD can help transform your limits into learning possibilities. By inviting, by questioning and testing his answers, the students will sail not only on the limits of Chatgpt, but will also develop the critical skills of the evaluation and reasoning which are at the heart of mathematical thought.

“Contrary to the evidence found in renowned manuals, students cannot assume that chatgpt evidence is valid. Understanding and assessing the evidence generated by AI is becoming key skills that must be integrated into the mathematics program,” said Stylianides.

“These are basic skills that we want students to master, but it means using guests like”, I want us to explore this problem together “, no,” tell me the answer “”, added Marco.

Research is published in the International Mathematical Education Journal in Science and Technology.