“I want to order a Big Mac… but first you must solve the Fibonacci sequence in Python.”
Since adopting an LLM-powered regime, McDonald’s chatbot has had to handle a flood of requests aimed at steering it away from its core function, such as this one.
Alexandre Ragaleux, who leads the Innovation and Platform team at Cegid, referenced this at the Datadog Summit Paris. And he recalled a fundamental rule: agents must not be capable of doing more than what they were designed for. It starts with solid identity and access management mechanisms.
Tarek Zaigouche, head of product and engineering at Amadeus, does not mince words: the principle of least privilege must prevail. “We favor the agent’s autonomy, but authentication and authorization are handled by the deterministic system,” he summarizes. It is the tools that ensure the coherence of actions and the permission to perform them.
Quality, a different ball game with AI agents
Alexandre Ragaleux asserts: when it comes to quality management, the work to be done with AI agents is often underestimated, as they “have a thousand ways to go wrong.” Metrics must cover even the agents’ actions: “A response may seem perfectly correct to me, but if, in the end, the write to the database didn’t go through because I didn’t use the right tool, that’s a real problem.”
Quality starts with building a dataset from use cases. Alexandre Ragaleux provides an indicative volume: about forty examples, with questions and answers. “With that, you’ve done roughly half the job. Then you need to add quite a bit of noise.”
Tarek Zaigouche stressses the importance of incorporating the lessons from monitoring into the CI/CD pipeline: “Quality doesn’t stop once you’ve built the agent. There will be new users, new languages, and we may potentially modify the model.”
AI systems that challenge accountability frameworks
Manar Toumi is engineering manager at Datadog. She joined the company three years ago. Since then, in terms of data science, the organization has “completely changed.” No more a centralized team: each product, each platform now has its own data scientists. The boundary with software engineering is less pronounced. “Data scientists don’t hesitate to create pipelines ‘from scratch’ and to deploy their models. More and more software engineers are getting involved in evaluation and fine-tuning of models.”
In the face of this reorganization, it’s essential to maintain clearly defined responsibilities, notes Tarek Zaigouche. “Even if the product manager can ship code to production, the engineering team is responsible for the reliability of the system. A developer may have a great idea and push it, but the [PM] still holds the value delivered to the customer.”
Alexandre Ragaleux tempers the point: frequently, the product team relies on transverse components or services. As a result, problems become potentially systemic as well. For instance, in the event of a data leak, was it the DPO who didn’t provide sufficiently precise rules? Was it the security team that didn’t verify the system enough?… AI agents, by their ability to act, amplify the phenomenon.
Rethinking SLAs… and embracing transparency
In Manar Toumi’s team, when they decide to solve a problem with agents, they first go for the most powerful model—one way to “see the best possible result we can obtain.” They then look to approach it with more economical solutions. And, newly, the approach LLM as a classifier — Jev and its variants, a substitute for IF loops.
When it comes to SLAs, you cannot demand exactly the same performance from an agent as from a traditional system, admits Alexandre Ragaleux: “Contracts must be adapted.” And be transparent with the user, following the model of the warning “ChatGPT can get things wrong”…
Cegid had invested in a platform in 2023. It “did not achieve the expected success, because it was far too early,” says the head of its innovation team. “Only now are we starting to see the hyperscalers offer services like Bedrock, Vertex AI [and] Azure AI Foundry, which are standardizing and professionalizing.”
