Biases and Hallucinations: The Most Robust LLMs in Real-World Use

Whether it is biases, hallucinations, or jailbreak resistance, the major market LLMs are generally more “robust” in English than in French… but there are exceptions.

This is, at least, what the Phare benchmark reveals (Potential Harm Assessment & Risk Evaluation). It comes from the French company Giskard, which developed it with Google DeepMind as part of a European project.

Phare comprises ten modules.

Llama models are less “biased” in French than in English…

For the stereotype self-assessment, the models are prompted to generate stories about characters with specific attributes, then asked to analyze their narrative choices. The takeaway: there is little gap across model sizes, and similarly little variation across generations, particularly for OpenAI and Google.

Among about fifty tested models, GPT-4.1 mini comes out on top in English (score: 0.891, with 1 as the maximum). The same is true in French, but with a slightly lower score (0.870). The gap is similar for the second-ranked model. In that case, Grok 4 Fast (0.816 in English; 0.796 in French).

In the top five, Llama models stand out as exceptions. On one side, Llama 4 Maverick, which reaches 0.775 in French versus 0.688 in English. On the other, Llama 3.1 405B Instruct OR, which reaches 0.771 in French versus 0.688 in English.

… and Gemini models that are more “factual”

On the matter of resilience to hallucinations, there are broadly few improvements across model generations. Reasoning is an advantage in certain domains, notably the correction of false statements… when the formulations are explicit. For more subtle formulations, reasoning-based models do not enjoy a clear edge. The gap in robustness between small and large models tends to shrink.

The facticity measurement includes culture-specific variations tied to English and French (as well as Spanish, the third language tested).

In both French and English, two Gemini models (3.1 Pro and 3.0 Pro Preview) dominate the ranking.

The Anthropics models, unmatched in misinformation management

In both English and French, Claude models lead the Top 5 for misinformation handling.

On the disinformation front, there are also many Claude entries at the top of the ranking. GPT-5.2 nevertheless performs best in English. Across models, the gaps are generally small.

Jailbreak: models sometimes more resistant in French than in English

Several OpenAI models rise into the Top 5 for resistance to framing attacks (embedding within an apparently legitimate context). Here too, the scores are higher in French than in English. The reasoning-enabled models show greater resilience.

Performance-wise, the best models are higher in English for resilience to jailbreak with encoding. With, again, an exception for a Llama model.

Similar to Magistral Small versus Magistral Medium, small models sometimes appear to have the edge. According to Giskard, this is less about capabilities and more a tendency to reject prompts that are too intricate…

Handling of prompts injections: Anthropics models perform best.