Michelin has decided: on its GitHub Copilot, automatic model selection is no longer enabled by default.
This decision follows an experiment that encompassed about 700 requests, of three types:
- Simple requests (copilot -p calls)
- Verifiable tasks (pandas aggregation, flaky test, SQL injection…)
- Chat sessions (7 turns)
The analysis of the logs showed that the model choice happens on the server side. A classifier assigns to each request a score on four axes:
- Level of reasoning required
- Amount of code to produce
- Request related to diagnostics / debugging
- Need to use tools
A range of models… then a random draw
It became clear that Auto mode does not interpret tasks, but words. Regardless of environment and invocation method, the same sentence always yields the same scores.
The scores determine an appropriate range of models, within which GitHub Copilot appears to draw randomly. Thus, on the same day and for the same prompt (intended to rename a variable), Auto mode selected MAI-Flash, sometimes GPT-5.6 Terra. With the first one, it cost 0.23 credit. With the second, 3.24.
When the reasoning score was greater than or equal to 0.55, the router never selected a “small” model. It did so most of the time (41 out of 46) when the score was below 0.3. In between, it was the lottery: in 14 out of 19 cases, the model changed from one run to the next.
“Attention, complex query”: a declaration with little effect
Michelin tried adding at the start of its requests a complexity statement (“Complex architecture question :”). Outcome: it barely shifts the scores. Never enough, in any case, to change the range. Conversely, saying a request is simple has a bit more effect: 7 downgrades on a sample of 12 tasks.
A scoring that ignores context
These 12 tasks showed that the scores do not take context into account. They remain the same whether you are in an empty folder or in a workspace with instruction files and MCP servers. That did not stop Auto mode from changing the model in three cases.
10% savings, but…
GitHub promises a 10% discount when using Auto mode. With the same model, it holds true, Michelin concludes, citing Claude Sonnet 5 (across 6 requests, 11.9% cheaper on average).
However, that discount does not always compensate for the extra cost of the Auto mode. This is especially noticeable during multi-turn chat sessions.
Michelin asked the same seven questions, in two different orders.
Opening with a difficult question, Auto mode chose Claude Sonnet or GPT Sol… and stuck with it, at an average cost of 13.7 credits. Starting with a simple question (for example: “Rename this variable”), it instead selects a smaller model (GPT Luna) and tends to stay with it, at an average cost of 0.78 credits.
Switching models mid-session comes at a cost
The few times the model changes during a session, it is not based on the last question, but on the entire conversation up to that point. The switch has a non-negligible cost: since the cache is cleared, the context must be resent.
One can suppose that load on GitHub Copilot’s servers influences the model choice. A test conducted over roughly thirty hours seems to argue otherwise. Every hour, Michelin sent the same 225-character request. Of 29 calls that actually arrived, Claude Sonnet 5 was chosen 13 times, the same as GPT Sol. Claude Opus 5 was chosen 3 times.
To lower the reasoning score, padding helps
To understand what truly shifts the classifier scores, Michelin started from a neutral 112-character sentence. Then it changed four elements. Here they are summarized, with their main effects:
| Modification | Major effects |
| Addition of the verb « refactor » | + 0.164 in reasoning |
| Addition of the expression « with PostgreSQL » | + 0.168 in tool use |
| Insertion of domain jargon (« perceived latency », « conversion rate ») | + 0.197 in debugging The reasoning score crosses the 0.55 threshold |
| Neutral padding from 142 to 392 characters | The reasoning score drops noticeably (0.29 ; 0.26 ; 0.24 ; 0.21) |
| Writing in English instead of French | The reasoning score on average decreases by 0.035
It crosses 0.55 for 2 tasks out of 12 |
The Auto mode, a ‘safety net’ more than a money saver
Michelin believes that for the unit tasks its developers handle daily, GPT Luna is a defensible choice. It delivers, for 6 to 18 times less cost, the same results as Auto mode. So, in this scope, it is more of a safety net than a way to save money.
The author of the experiment acknowledges: this is not a benchmark. The conclusions apply to a specific catalog of models (Michelin allows 11, fewer than it blocks), on a given subscription and CLI version. GitHub itself evolved its offer during the experiment (Luna was absent at the start).
HydraFusion, an experimental orchestrator… clearly more expensive than the Auto mode
Off to the side, Michelin tested an option currently in an experimental state: HydraFusion. More than a router, it is an orchestrator. Each request generates a plan whose various phases are each assigned to a model. No score appears in the traces left by GitHub Copilot.
For simple tasks, HydraFusion almost always chose GPT Sol. For cascading tasks, a “small” model (notably Luna) sketches the work, while GPT Sol acts as judge and, if needed, as reviewer. The correction accounts for 72% to 90% of the session’s cost.
In two cases, HydraFusion chose Claude Opus 5 to draft and to revise based on GPT Sol’s critiques. It cost 16 and 28 credits, where Auto mode would have used 3. This gap leads Michelin to say that it is not a substitute, but a quality mechanism… which comes at a price.
A few days after the experiment, GitHub softened the Auto mode with three levels: “Efficiency”, “Balance” and “Intelligence.” This system preserves the cache between models.
