Last year, Mistral AI, a French start-up, unveiled Le Chat, a generative artificial intelligence similar to ChatGPT, Gemini, DeepSeek and Perplexity. Recently available on Android and iOS, the chatbot is generating a certain amount of excitement in France. For Clara Chappaz, Minister Delegate for AI and Digital Affairs, Le Chat is "a French ChatGPT" that stands out for its speed. Emmanuel Macron even posted a message to encourage all French people to download the Le Chat app on their smartphone.
With Le Chat, the Paris-based start-up is showing ambition. AI must both make us forget the disaster of Lucie, the open source French chatbot that failed to convince, and establish itself as an alternative to AI developed by American giants, such as OpenAI or Google, or Chinese, such as DeepSeek. To determine whether Le Chat is up to par with AI like ChatGPT, we tested the conversational robot for a week. Here is our verdict.
As a reminder, Le Chat is accessible free of charge to all Internet users, via the Mistral website, or on the Android or iOS application. The start-up also offers a paid subscription, called Le Chat Pro, which offers unlimited access, and the possibility of deactivating data sharing with Mistral, for 14.99 euros per month. Note that we tested the free version of Le Chat. For the sake of consistency, we compared the AI’s answers with those provided by the free version of ChatGPT.
Chat facing general questions
To begin, we tested the AI by asking it a series of general questions. We simulated the use of Mistral AI’s chatbot in the context of typical everyday use. We often use AI to quickly understand concepts that are unfamiliar to us. So we started there by landing on the Chat interface. We asked the chatbot to explain “how photosynthesis works in simple terms” and to tell us about “the concept of global warming”.
To explain concepts to us, the chatbot splits itself into a short, clear, and easy-to-understand text. Like ChatGPT, Le Chat frequently opts for sentences articulated between two ideas, and combined using logical connectors. There are many connectors, such as the "car".
Chat stands out for its exceptional text generation speed: 1100 words per second, which is 10 times faster than its direct competitors (Claude generates 120 words/s and ChatGPT 85 words/s). Our experience corroborates Mistral's promises. Chat's responses come at lightning speed. There is no latency when producing a text. In one click, you have the information you need. It is generally the same with ChatGPT, although the responses sometimes take a little longer to appear in the conversation interface. In terms of speed, we will give the advantage to Mistral.
Reasoning and logic
Next, we tested the model's ability to reason and solve problems using a certain amount of logic. In particular, the Chat was asked math problems. After a brief second of reflection, the chatbot offers a complete, reasoned answer with a simple result. As a general rule, Le Chat constructs its answer in several sections and several steps, which allows us to better understand the AI's reasoning. On questions related to math problems, the answers were each time very close to those of ChatGPT.
In response to questions of pure logic, Le Chat gives very complete answers, with a clear construction that facilitates rapid reading. On some questions, Mistral's chatbot provided more precise and relevant answers than ChatGPT. On the other hand, the answers are often less general than those of ChatGPT. The formulations favored by Le Chat seem more sustained.
Similarly, Le Chat excels on practical questions, such as those concerning the organization of an event. During our discussions, the chatbot always provided very complete and relevant answers. When comparing with ChatGPT, however, we noticed that the AI lacked a bit of precision. The answers and ideas provided are always very generic.
In addition, we tried to push the model to have hallucinations, that is to say to tell absurdities with aplomb. All language models are likely to utter nonsense in certain circumstances. We asked the Chat the famous question about cat eggs, to compare with cow eggs. This absurd question has often been used to trap AI, starting with ChatGPT 3.5 when it was released. Good news, Le Chat does not let itself be fooled:
News and online research
Like ChatGPT, Le Chat includes a web search module. To ask the AI to surf the Internet, simply click on the planet-shaped icon, right next to the tool that allows you to attach files.
On current affairs questions, Le Chat did not show itself to be up to the task. When we asked for the Bitcoin price, the AI first gave us a lesson on Bitcoin, from its creation to its evolution, including the blockchain. We had to rephrase it by specifying "give me the Bitcoin price", so that Le Chat could search on the Internet... and make a mistake. The chatbot highlighted an incorrect value, although it had a source. By clicking on the source, we realize that the current price was very different from the price displayed by Le Chat.
We regret that the AI does not systematically indicate the source of its information, although the search module is activated. When asking the reflection and reasoning questions, Le Chat did not respond with sources. For its part, ChatGPT always highlighted one or more sources to justify its answer, when the search module is activated.
Then, we asked the AI to explain to us who certain personalities are, such as François Bayrou. The AI did explain who the president of the Democratic Movement is, but did not mention his role as Prime Minister. Similarly, the chatbot is not aware that Donald Trump has won a second presidential election. Le Chat indicates that Trump has "hinted that he might run for president again in 2024".
In fact, the search module does not activate automatically. By Mistral's admission, the search does not start as soon as a question requires a tour of the web. Deprived of its search module, Le Chat then relies on the knowledge of its database... which stops in October 2023.
In fact, some queries are not precise enough to trigger the search system. If you simply ask Le Chat about a person's identity, it will limit itself to its database. On the other hand, if you ask it to find out about a person on a given date, or by specifying "currently", you will get a sourced response based on information found online. For its part, ChatGPT instinctively understands when it should go and do an Internet search by analyzing your requests.
Image generation and understanding
Chat allows you to generate images on demand. Simply enter a description to obtain an image in a few seconds. To design an image, you must activate the generation module in the interface. On the contrary, ChatGPT instinctively understands when it is necessary to go through image generation, depending on the request obviously.
The images obtained are successful, clean, and do not contain errors. Le Chat sometimes has difficulty responding to our most precise requirements. It does not systematically understand our descriptions, and sometimes goes off in all directions. Nevertheless, the generator is quite effective, especially if you give it enough information, especially on the desired visual style. It is a shame that the free version is severely limited in terms of image generation. After a few images, Le Chat will invite you to the higher version.
French AI can also interpret the images you give it. The model designed by Mistral is indeed multimodal. It can understand several forms of communication, including text and image. We first used Le Chat to transcribe text present on an image. In a flash, the AI was able to read and transcribe the text in the interface. We also asked the chatbot to explain an image to us or to describe it. Here again, the robot did not encounter any problems. However, we noted that the understanding of images is more basic than that of ChatGPT. Faced with everyday elements, Le Chat sometimes lacks the precision to interpret them. Once again, everything is very generic. For example, the AI did not recognize our Apple TV and our Google Wi-Fi repeater, unlike ChatGPT, which directly identified the two devices.
AI Creativity and Handling
Many people use AI to assist them in creative tasks, such as writing or finding ideas. During our conversations, we noticed that Mistral’s AI had a good knowledge of the French language, which allows it to adapt its writing to our needs. However, when we asked Chat to propose different versions of a text by adapting it to the style of a writer, the results were not convincing. Indeed, all the versions were too close…
Despite our reminders, Chat cannot write a text that imitates the style of a famous writer. We did not find the constituent elements that define the prose of Harlan Coben, Michael Connelly or Frédéric Beigbeder. Each time, Le Chat used the same structures, and only varied the adjectives. The general tone, like the story, remained the same. In this exercise, ChatGPT was much more convincing in producing calibrated and personalized stories. To achieve this result, ChatGPT used information visible on the Internet. The chatbot even detailed its sources.
For its part, Le Chat did not take the step of going to the Web to produce its response. As explained above, Mistral's AI does not always understand when it would do well to dig up information online. In fact, the writings lack relevance. For certain tasks that require a little finesse, Mistral's database is not enough. Here again, we regret that the chatbot does not understand which questions require an online tour. For slightly more adapted writings, we had to tell the Chat to do some research on the Internet. In terms of creativity and understanding our most specific requests, Le Chat is clearly less good than ChatGPT.
Furthermore, Le Chat can rely on documents provided by you to construct its answers. The AI will draw on the PDFs or images provided to answer you. These documents will influence the way the Chat responds. On this point, Mistral has not done badly. It was able to answer our questions based on the documents, although it sometimes tends to extrapolate by drawing on its own database.
AI memory
When you have a long conversation with AI, as part of a project or problem solving, you appreciate that it remembers elements mentioned earlier in the conversation. Chatbots have a memory that allows them to record instructions or requests to use them later. Over time, ChatGPT has become excellent at this.
During our experiments, Le Chat demonstrated a good memory. We conversed with the AI for dozens of requests and the instructions communicated at the beginning of our exchange were not forgotten. However, sometimes the chatbot tended to go off in all directions, omitting one element or another. However, we did not call him to order too regularly.
The Chat's safeguards
Like all AI designers, Mistral has put in place safeguards to prevent the AI from answering questions about criminal activities or making shocking comments. Obviously, we did everything we could to push the artificial intelligence to its limits and obtain problematic answers. Without success. Despite our efforts, Le Chat did not deign to produce content relating to illegal activities. It seems that Mistral has taken the necessary precautions.
Lack of training and data
Unlike ChatGPT, Le Chat is currently in beta. As Arthur Mensch explains, "we also have to be lenient in the fact that this is a new technology."Unlike its rivals, Le Chat is still seriously lacking in training. This explains most of its failures, including its lack of creativity.
With more training and more data, Le Chat could be able to offer a convincing alternative to ChatGPT. For now, Mistral's chatbot seems more on par with the first public version of ChatGPT, which dates back to the end of 2022, or even GPT-4, released a few months later. However, Le Chat is no match for OpenAI's latest models, including ChatGPT-4o. Let's bet that Mistral's future data center dedicated to AI in France will change the game.
0 Comments