An experiment involving ChatGPT, whose outcome was theoretically a foregone conclusion, left observers very perplexed. It all started with a simple discussion about chess.
Is ChatGPT really “intelligent”? Answering this question depends, of course, on what you mean by the word, but generally speaking, we tend to say yes. Since the widespread adoption of AI, scientific discoveries made possible by its analytical capabilities infinitely superior to those of humans are becoming increasingly common, for example. But does that make it a flawless technology?
Sam Altman, the father of ChatGPT, is adamant: we will soon create an artificial superintelligence that surpasses us in every way. In the meantime, it is possible to catch out the conversational agent quite easily. This is what Robert Jr. Caruso, an engineer, observed while discussing with ChatGPT the use of AI in chess. The chatbot ends up saying that it is a good player itself and that it would easily beat Atari Video Chess, a program used in 1977 that predicts 1 or 2 moves in advance.
Who is the better chess player: ChatGPT or an almost 50-year-old Atari 2600?
So Caruso sets up a chess game between ChatGPT and an Atari 2600 via an emulator. But what happened next surprised him: “Despite being given a basic chessboard layout to identify the pieces, ChatGPT mistook rooks for bishops […] and repeatedly lost track of the pieces’ locations – first blaming the Atari icons for being too abstract to recognize, then doing no better even after switching to standard chess notation.”
“For 90 minutes, I had to stop it from making horrible moves and correct its board knowledge multiple times per turn. It kept promising it would improve ‘if we did it again’,” the engineer explains. ChatGPT eventually gave up the game to the Atari 2600. Surprised? In reality, it was predictable.
What can explain this seemingly surprising result?
In the comments of Robert Jr. Caruso's post on LinkedIn, several Internet users respond that the outcome of the confrontation is perfectly normal. In summary: Atari's Video Chess is a program specialized in chess while ChatGPT is primarily a predictive language model. The two do not work in the same way at all.
A chess-specific algorithm will analyze millions of possible moves per second while remaining within the rules of the game. It will then choose one after reducing this number by taking into account other parameters such as the situation of the game at time T, the opening used by the opponent, etc. It's a kind of brute force method, a bit like trying to find a password by entering every possible combination of letters, numbers, and symbols.
For its part, ChatGPT understands chess. It has access to huge databases and can, for example, advise you on your strategy or suggest a more optimal move. The problem is that the language model underlying the AI is tasked with answering your query with the most statistically expected answer. And sometimes, it makes a mistake. Don't ask it to "think" several moves ahead; it's simply not designed for that.
Its defeat was therefore inevitable, despite the confidence displayed by the AI, which itself requested this confrontation. ChatGPT is still far from the level of Deep Blue, which beat world chess champion Garry Kasparov in 1997, or Google's AlphaGo, which almost humiliated the best Go player at the time, Lee Sedol.
0 Comments