What does it take to build self-maintaining AI infrastructure in production? We sat down with Valentin Buchner, Chief Architect at Chapter Two, to get a practical look at it.
When Valentin Buchner, Chief Architect at Chapter Two, gave his coding agents read access to production logs, “an hour of reading logs turned into one prompt.” Having worked across machine learning research, healthcare, EQT’s Motherbrain, and music finance, he saw firsthand how much senior engineering time disappears into routine pipeline maintenance.
Ahead of his session at the upcoming NDSML Summit, Valentin discusses how to build self-maintaining AI infrastructure without creating dangerous feedback loops.
In this interview, he explains why machine-readable telemetry beats traditional dashboards, how to set safety guardrails for coding agents, and the simple mental trick to automate future debugging.
At the upcoming NDSML Summit, your session highlights how maintenance eats building time. What does instrumenting pipelines for self-maintenance involve, and why does it matter?
It matters because it frees up engineering hours for new features, product decisions, and other high-value work. Software needs maintenance because the world around it changes. This is especially true for AI agents. The AI landscape changes quickly, and the best practices of building AI solutions are still being established. This results in AI solutions requiring a disproportionate amount of maintenance work, sometimes taking up an engineer’s full attention. 
The good part is that maintenance mostly does not require high-stakes decisions. We already know the specifications, have unit and integration tests, and can often pinpoint the cause. This is work that AI agents themselves can perform, with minimal human involvement.
Self-maintenance involves giving access and autonomy to AI agents in a safe and responsible manner. Ideally, you want a maintenance agent with read access to system logs, databases, and write access to the codebase. In reality, you have to find the right balance, considering sensitive user data and the risk of production failures. Often, this involves read access to desensitized data and the ability to propose code changes that are approved by a human before release. You also need to define how the maintenance agent should be triggered: on a regular schedule, through error alerts, or both.
To make that self-maintenance work, you advocate for agent-readable data over dashboards. Why are traditional dashboards insufficient here, and how do you structure machine-readable telemetry?
Traditional dashboards are a great start, but they are often optimized for human readability and don’t go into details. Keeping telemetry in simple JSON logs gets you far, ideally paired with metadata such as severity, microservice, what data is being processed, etc. This allows the agent to quickly filter through the logs to find the relevant ones. What matters more than the structure are the access rights – we experienced a huge productivity gain once we gave our coding agents read access to desensitized production logs. An hour of reading logs turned into one prompt.
Your session covers empowering agents to trace failures and ship autonomous fixes. How do you grant that level of authority without risking failure loops in production?
Never let an agent ship code straight to production. Code should always be reviewed by independent agents using capable models, and in high-stakes domains such as finance and healthcare, by human developers. To avoid failure loops, limit how many times the agent may attempt to fix the same issue, roll back automatically when a fix makes things worse, and escalate to a human developer as soon as an attempt fails.
Apart from code reviews, the same guardrails are needed as we had in pre-LLM times. Unit and integration tests, CI/CD pipelines, and test deployments. For AI agents, one should create evaluation pipelines that estimate the agent’s performance. This can be done with deterministic evaluations when the agent creates structured output, or with an LLM-as-a-judge for the case of conversational agents.
Drawing on your experience across EQT’s Motherbrain, healthcare, and IP, what core telemetry pattern works regardless of the stack?
What really helps is tracing each unit of work end-to-end. A unit could be a customer, a file that is being processed, or a website that is being scraped. Each such unit should receive an ID, and that ID should be tracked in every log line, together with input and output, the version of the code, and other metadata – anything that is needed to reproduce failures.
How do you safely validate that an agent’s autonomous code or config fix won’t introduce hidden regressions before it hits production?
It’s all about testing, as with human-written code. Test-driven development is also a good idea for autonomous agents: they should first write the tests defining the desired behavior, and then the code that actually fixes it, until the tests pass. Ideally, writing the tests and fixing the code is even done by two independent agents.
When a coding agent makes changes to an AI agent in production, it is important to have a good evaluation pipeline for the AI agent. It is good practice to continuously look for new test cases to cover a variety of scenarios. For instance, we set up a nightly cron job that analyzes the logs of a specific agent. Edge cases, or cases the agent didn’t answer optimally, are included in the evaluation set, and each time we make a change to the agent, we run the evaluation to ensure performance improves.
Here, it is important to avoid “overfitting” the AI agent on the examples in the evaluation set. This can happen when the coding agent optimizes the AI agent on exactly the examples in the evaluation set, at the cost of out-of-distribution performance. For this, it helps to have a “hold-out” set that the coding agent never sees, and that can be used to evaluate the agent’s true performance.
Looking ahead as self-maintaining systems become standard, what single piece of advice would you give AI engineers transitioning from manual debugging to building self-improving infrastructure?
There is a simple strategy. After debugging something, I ask two questions: Could an AI agent have made the decisions I made? If yes, what information/tools would the agent have needed to do this autonomously? It’s important to continuously re-assess these questions because AI improves quickly and a no today might be a yes a month from now.
Valentin will dive further into these patterns at the upcoming NDSML Summit through his session, ‘Stop Babysitting Your Agents: Self-Maintaining & Self-Improving Agentic Systems.
