A cheaper model at the token level isn’t necessarily cheaper for the request.
BPCE was able to verify this on customer support moderation—thanks to its internal benchmark.
Mikaël Le Bars (photo) leads the cross-disciplinary AI expertise team within the group’s data science center of excellence. At the Big Data & AI Paris 2026 expo, he revisited the development of this benchmark.
At its core, there was the desire not to “pay for evaluation on every project […] without owning anything at the end.” There were also the limits of public benchmarks.
In parallel, many benchmarks were reaching saturation: they were becoming too “simple” for the main models, which reached the maximum scores. In Stanford’s latest AI Index, the phenomenon affects nearly half of the 60 most popular benchmarks.
Models also tend to overfit: they “stick” to certain benchmarks, so that even a minor modification can be enough to drop performance. Apple researchers illustrated this on a derivative of GSM8K.
There is also the issue of governance of rankings. Model vendors can test several versions before releasing one… which isn’t necessarily the one offered to customers. We’ve also seen some fund benchmarks financed (OpenAI with FrontierMath, for example). Add to that a certain “Anglocentrism.” MMLU bears this out. More than a quarter of the questions require a “Western way of thinking” and 85% of geographic references focus on North America or Europe.
An Object-Oriented Framework
To identify the business scopes, Le Bars’ team based its work on all BPCE projects—including PoC and ideation phases. It determined the skills expected in each project (moderation, translation, intent detection…) and broke them down into tasks on which to evaluate the models (classification, speech recognition, entity extraction…).
Second step: identify the available datasets. And, if needed, proceed with labeling. “On one of the projects, we recorded text that we had to read,” explains Mikaël Le Bars.
In addition to state-of-the-art metrics (BLEU and METEOR for translation, false positives and false negatives for extracting sensitive data…), BPCE added metrics tailored to its operations. For example, for audio transcription, the error rate on terms specific to the banking sector.
The prioritization is based on usage, available internal data and criticality, taking into account cost, latency and the robustness of the systems in place.
The implementation was carried out with an object-oriented framework. It enables developing scalable components to receive new datasets, models or metrics.
The code is deployed across several environments depending on data criticality and model availability. Each instance is timestamped and identified by model version, dataset and prompt. A common protocol enables exporting results as CSV into a dashboard for the business units.
Configuration Is More Important Than the Model
Beyond what public benchmarks may reveal, there is the impact of each component in a pipeline on the accuracy of the responses. In one use case for information extraction from financial documents, BPCE observed that configuration matters more than the model. The model itself has only an impact of about 1–4 percentage points, versus 12–17 points for embeddings, 10–20 points for the chunking/retrieval phase and 14–21 for OCR.
The value of internal metrics became especially apparent in audio transcription. Consider processing a phone conversation that contains the instruction “transfer €45,000 to account 8837.” One model might drop the word “the,” while another might interpret 40,000 instead of 45,000. In practice, the two mistakes are not equal. But a standard metric like WER (Word Error Rate), widely used in public benchmarks, would yield the same score for both models. Adding aspects such as punctuation and banking terms can make a real difference.
Agentic AI, a Major Challenge on the Horizon
As the assessments progressed, BPCE concluded that the rankings obtained under nominal conditions do not hold up to noise, accents, or out-of-distribution data.
Regarding the cost of the approach: “Not bad, mainly in terms of human time,” due to labeling, coding and deployment, according to Mikaël Le Bars. But “we exchange incurred costs for a chosen cost.” And 70% of the code is reusable.
BPCE is considering involving more group entities (the solution is available on the internal repository). In particular by allowing users to upload their own models and configure prompts. A move to open source is contemplated in the medium to long term. Another perspective is to broaden the benchmark to agentic systems—not yet deployed internally. The challenge is unique: it will not suffice to evaluate only the final result, but also the sequence of actions taken to reach it.
