Building the Foundations of Trust

On 9 December 2025, Nicolas Cavallo, Head of Generative AI at OCTO Technology, delivered a detailed presentation on one of the most important topics in the current AI landscape: how to put AI agents into production with confidence.
Many organisations now showcase AI prototypes and internal demos. Yet very few actually deploy agentic systems at industrial level. The gap between a working prototype and a robust production system remains large. The conference addressed this gap head on and proposed a structured way to think about trust in AI agents.

Nicolas articulated his talk around three main pillars: evaluation, security and stability. Before exploring these elements, he used a practical example to illustrate the complexity behind seemingly simple agentic workflows.

1. Why AI Agents Are Difficult To Industrialise

AI agents can read unstructured text, connect to internal tools, call APIs, synthesise information and orchestrate complex workflows. These new abilities create strong value for organisations. At the same time, they introduce new forms of uncertainty.
Unlike deterministic systems, agents are probabilistic, non deterministic and context dependent. Their behaviour may vary across identical inputs, and this variability complicates testing, validation and risk management.

This context has created a paradox:

On the one hand, companies are multiplying AI initiatives.
On the other hand, few are willing to let autonomous agents act on real systems without strong guarantees.

Building trust is therefore not optional. It is the foundation that allows an organisation to move from “it works in a demo” to “we can rely on this daily, at scale, with acceptable risk.”

2. A Simple Use Case That Reveals Hidden Complexity

To illustrate his ideas, Nicolas selected a use case based on managing candidate emails. The idea seems straightforward. You receive many emails from applicants, and an AI agent should help classify the messages, extract relevant information and trigger next steps.

The agent could, for example:

Detect whether an email is a job application.
Extract the candidate’s name, skills and desired role.
Add the candidate to a structured system.
Check your calendar for possible interview slots.
Identify HR staff or hiring managers.
Generate and send a first email proposing a meeting.

At prototype scale, such a system often works well enough to impress stakeholders. With a few test cases, everything appears solid. The danger is to assume this early success means the system is ready for production.

Nicolas’s central message is that this assumption is wrong. A successful demo hides many challenges:

How does the agent behave with ambiguous or noisy emails?
What happens if a candidate unknowingly includes malicious content?
Can the agent reveal private information?
Can it send messages to the wrong recipients?
Can it be manipulated?
How do you monitor or reproduce its decisions later?

Behind the apparent simplicity of the workflow, real deployment requires a disciplined approach. Nicolas structured this approach into three pillars.

3. Three Key Takeaways

Takeaway 1 — Evaluation must be continuous and multi layered

Evaluation is not something that happens once before deployment. AI agents require evaluation at three levels, which apply throughout their lifecycle.

a. Evaluation in the loop (during design)

In this first phase, you integrate “evaluation zones” directly inside the agent’s workflow. These zones serve as checkpoints that verify intermediate results or decisions before the agent proceeds. Evaluation in the loop may use:

A second LLM acting as a judge.
A human reviewer (human in the loop).
Simple rule based checks.

For example, after the agent proposes an email reply, another component verifies if the answer is consistent with the candidate’s profile and with internal job offers. If not, the workflow loops back and searches for more data or escalates to a human.

b. Offline evaluation (during testing)

Once the workflow stabilises, you run structured evaluations outside of live environments. This mode relies on three elements:

Clear criteria: relevance of final answers, correctness of tool usage, coherence of action sequences.
Evaluation datasets: examples representing normal, difficult and adversarial cases.
Scoring methods: humans, LLM judges or custom code.

Offline evaluation produces metrics that serve as quality gates. If a change in prompt, model or configuration reduces accuracy, the system should not be deployed.

c. Online evaluation (during production)

After deployment, evaluation does not stop. The system must be monitored in real time. This includes:

Latency and response times
Error rates and failure patterns
User feedback signals
Frequency and types of actions the agent executes

Online evaluation helps detect regressions, unexpected behaviour and cost anomalies. Production data also enriches offline evaluation datasets.

Overall, this takeaway shows that evaluation must be an ongoing activity embedded in design, testing and production.

Takeaway 2 — Security requires defence in depth

Security is a central challenge for agentic systems. Nicolas framed the problem through the lens of injection attacks. Traditional SQL injections exploit weaknesses in query construction. LLM applications face similar threats, expressed through natural language.

a. Prompt injection

Prompt injection can be direct or indirect.

Direct prompt injection occurs when a user explicitly instructs the model to ignore previous instructions and change its behaviour.
Indirect prompt injection occurs when the agent reads external content and treats hidden instructions as legitimate commands.

Both forms can lead the agent to reveal confidential data, execute unauthorised actions or behave unpredictably.

b. Broader vulnerabilities

The OWASP list for LLM applications highlights further risks:

Information leakage
Data poisoning
Excessive autonomy
Unsafe tool usage

These vulnerabilities affect even high profile systems, showing how difficult the problem is.

c. Defensive strategies

To reduce risks, Nicolas recommended a combination of approaches.

Limited power
The agent should have only the minimum rights required. Restrict access to sensitive systems and distinguish between read and write permissions.

Controllers
Controllers validate actions before the agent executes them. Humans can handle high risk actions. Automatic controllers can block unsafe behaviour in real time.

Guardrails
Guardrails inspect inputs and outputs around each model call. They identify toxic content, private data, off topic requests or injection attempts. If detected, the system blocks or redirects the request.

AI red teaming
AI red teaming introduces systematic attack simulations. Teams write adversarial prompts, multi step attack sequences and edge case scenarios. They run these tests regularly and track how the agent responds. When a vulnerability appears, the team adjusts guardrails or controllers and repeats the test.

Observability
Observability ensures that every significant action is logged and traceable. Modern agent frameworks may distribute decisions across several components. Centralised traces help understand what happened, investigate incidents and refine defences.

These defence mechanisms form a layered structure that reduces the probability and impact of security failures.

Takeaway 3 — Stability depends on DevOps fundamentals adapted to AI agents

The third takeaway underlines that operationalising AI agents does not eliminate the need for good engineering discipline. On the contrary, it requires more of it.
Nicolas emphasised that classical DevOps practices remain the foundation:

Versioning of all agent components
Continuous integration and deployment
Automated tests
Centralised monitoring
Clear rollback strategies

The difference is that these practices now apply not only to code, but also to prompts, model settings, retrieval parameters and tool definitions.

a. Version control for prompts and configurations

Prompts evolve over time, and even small variations can change agent behaviour significantly. Treating prompts as versioned artefacts is therefore essential. The same applies to configurations that control tool availability, model temperature or retrieval logic.

b. CI/CD pipelines that integrate evaluation and security tests

When something changes, such as a prompt or a model, the CI/CD pipeline should:

Run offline evaluation datasets
Run AI red teaming scenarios
Compare metrics to historical baselines
Block deployment if regressions appear

This automation ensures stability across releases and protects production environments from unintended changes.

c. Monitoring production behaviour

Stability also depends on detailed monitoring. For AI agents, this includes:

Functional monitoring of actions taken
Detection of shifts in behaviour
Trends in human override rates
Tracking of cost evolution
Identification of unusual tool usage patterns

These signals help detect slow drifts or unexpected changes that are not visible through classical infrastructure metrics.

d. Balancing investment and risk

Building these foundations has a cost. For some use cases, simpler rule based systems may be more appropriate. Organisations should evaluate the business value, the risk level and the lifetime of each use case before choosing an agentic approach.

4. Steps To Strengthen Trust In AI Agents

Below is a concise list of steps organisations can take to move toward production ready AI agents. These items synthesise the practical recommendations from the conference.

[ ] Evaluate the effectiveness of AI agents using relevant metrics for each phase (design, testing, production)
[ ] Implement security mechanisms such as controllers, guardrails and security tests
[ ] Ensure full observability of agents in production
[ ] Establish a balance between automation and human validation based on the level of risk
[ ] Apply DevOps best practices adapted to the specificities of AI agents

how to put AI agents into production with confidence