Ciao, over the past month, while Italy was deep into the holiday season, things at HikmaAI and in our learning process went quite the opposite way.

We started iterating faster on the platform because we got some very good feedback from design partners and pilot users. We also ran our first installations based on the new deployment model.

Yes, we moved from SaaS to on-premise. That’s what the market had been screaming for.

Of course, none of the above came for free.

We started finding edge cases and the usual distributed systems problems. Because, agents or not, at the end you still got a bunch of containers competing for resources, sharing information, processing data, interacting with external services and, last but not least, dealing with failures.

As with every “cloud native” distributed system, you have a lot of points that can fail and generate a waterfall of failures across the synchronous and asynchronous parts of the system.

Some of you may already be asking:

Why am I reading about this basic stuff in a series about coding agent harnesses?

And here is the thing.

As mentioned in the first article, we believe, like many others, that to get the most out of agentic or automated programming you need strong feed-forward and feedback loops, with deterministic computation and LLM gates working together.

The coding agent is not the software factory. The complete loop is.

One of the first additions I made to our harness was the ability to spawn on-demand environments on k3d, perform end-to-end tests, and collect, measure and address errors coming from both the application and the infrastructure.

As always happens, the tendency is to leave reliability tests for later. Or, as modern naming conventions like to call it: technical debt.

But the faster you go, and the faster you want to go, the longer product qualification can take. This time also increases as soon as you start engaging with customers, for two simple reasons:

1. No one likes building and shipping shit code.
2. No user likes wasting time and feeling like they are your tester.

With those two criteria in mind, and given that we now have AI helping us, the question got an immediate answer in my head:

Why should we keep those tickets in the backlog? Let’s start addressing them now.

We had already automated the end-to-end qualification tests against real target environments, thanks to Xference.ai for providing model access.

We also had a proper improvement loop in our harness:

1commit
2run the CI tests
3deploy into an ephemeral environment
4run the end-to-end tests
5send me an update on Telegram every ten minutes
6identify and fix errors
7repeat

At that point, it was time to add Chaos Engineering to the pipeline.

Agentic coding allows the team to focus on establishing solid practices and foundations without waiting for a technical debt reduction sprint.

This does not mean asking the coding agent to “make the system resilient”. That kind of prompt is too vague and will probably produce a lot of plausible code without giving you any proof that the system is actually more resilient.

The real value comes from giving the agent an executable hypothesis, a controlled environment, clear safety boundaries and a measurable expected result.

Avoiding failures is not an option

Equipped with memories from my SRE days, I started looking at what was available in the open-source space.

The goal was to generate and inject failures as part of our qualification process, then have pi, my coding agent, investigate those failures and add or fix the resiliency mechanisms exposed by the tests, just as it already did for our normal CI and end-to-end tests.

Chaos Engineering means introducing failures deliberately to test the system’s resilience inside a controlled environment. In my case, that controlled environment is an ephemeral k3d qualification environment.

The idea was to take some of those annoying “edge case” and “what if?” questions and transform them into rehearsed responses.

It is not about generating chaos for the pleasure of breaking things. It is about:

1. Making the system more resilient to failures.
2. Reacting with calm and confidence in production, which only happens after you have already dealt with a similar failure.
3. Creating operational runbooks. Yes, this one is closely related to number two.

I am not going to write another post explaining the principles and history of Chaos Engineering. Google has already made a great effort there:

1. Google SRE books
2. Getting Started with Chaos Engineering

What I want to focus on is how Chaos Engineering fits into an agentic coding workflow.

Choosing the framework

AI makes it easier to describe a steady state, generate a hypothesis, parse outputs and create automation scripts.

Still, the human needs to decide what success means and how much of the system the experiment is allowed to touch.

My first task was to evaluate frameworks that could help us:

• inject failures into the messaging layer;
• generate memory pressure and OOM conditions;
• add latency and connection problems between workers and dependencies;
• interrupt callbacks and external calls;
• validate readiness and liveness behaviour;
• collect evidence that an agent could inspect.

The framework I opted for is Chaos Toolkit, an open-source framework written in Python.

Chaos Toolkit gives us a modular architecture where we can plug in extensions and custom drivers to execute experiments against the parts of the system we want to test.

As the first extension, we added Toxiproxy.

Toxiproxy sits between a worker and a dependency, allowing us to introduce network problems without modifying either side. We use it to introduce latency, timeouts, bandwidth limits, interrupted responses and connection problems across the messaging layer, databases, callbacks and controlled external targets.

For Kubernetes failures, we can terminate workers or temporarily remove a dependency. For memory failures, we can apply bounded pressure inside a worker and observe how its admission logic reacts.

The important part is that all of this happens only inside the dedicated k3d environment.

Start from the steady state

Before introducing a failure, you need to define what the system should look like when it is healthy.

“Everything is green” is not a steady state.

“HTTP returned 200” is not a steady state either.

For a long-running asynchronous operation, our steady-state checks look more like this:

 1{
 2  "terminal_status": "completed",
 3  "expected_outputs": 3,
 4  "physical_external_calls": 3,
 5  "successful_external_calls": 3,
 6  "ambiguous_external_calls": 0,
 7  "worker_restarts": 0,
 8  "oom_events": 0,
 9  "failure_codes": []
10}

The exact fields are not important. The important part is that the expected result is machine-verifiable.

A worker did not recover simply because it started answering health checks again. The operation must resume, persist its outputs, avoid duplicate external calls and eventually reach a stable terminal state.

We require repeated identical terminal observations before accepting convergence. This prevents the harness from mistaking a stale or intermediate response for a recovered system.

This is one of the most useful ways to work with a coding agent:

Do not ask the agent to make the system “reliable”. Give it invariants that can fail.

Turning “what if?” into experiments

Once we had the steady-state contract, we started converting failure hypotheses into experiments.

Some examples:

• What happens if a worker disappears halfway through a task?
• What happens if the messaging layer becomes slow?
• What happens if the database resets a connection?
• What happens if an internal callback times out?
• What happens if a target accepts a request but its response never reaches us?
• What happens when there is no memory headroom for another job?
• What happens when two workers disagree about the active execution configuration?
• What happens when an old message is delivered after a newer generation has started?

These scenarios cover different failure classes.

Worker failures

We terminate workers while an operation is active, then verify that another worker can safely continue the work.

The experiment does not pass because Kubernetes accepted a delete command. We record the original worker identity, prove that it disappeared, wait for its replacement, and then verify that the operation continues.

Dependency failures

We temporarily make the messaging or persistence layer unavailable.

We also inject latency and timeouts because a slow dependency is often more interesting than a dead one. A hard failure is usually visible. A dependency that responds after several seconds can consume worker slots, trigger retries and create backlogs before anyone notices.

Memory pressure

Containers compete for finite memory. This sounds obvious, but concurrency settings are often based on a number someone picked months ago on a laptop.

We created experiments where a worker sees progressively less memory headroom.

The expected behaviour is not always “keep processing”. Sometimes the correct behaviour is to stop accepting new work, let the active work finish, and resume only when there is enough headroom again.

Zero admission capacity is valid. An OOM is not a better answer.

We also keep an isolated legacy configuration that intentionally reproduces the old OOM behaviour. This lets us prove both sides:

• the failure is real and reproducible;
• the protected worker reacts differently under the same pressure.

External call ambiguity

One of the most interesting cases happens when an external target accepts a request but the response gets interrupted.

At that point, the worker does not know whether the target executed the operation.

Automatically retrying can produce a duplicate action. Not retrying means accepting that the result is uncertain.

Without target-side idempotency, you cannot honestly claim exactly-once execution.

Our response is conservative:

1. Persist the call intention before touching the target.
2. Bind it to a stable operation and case identity.
3. Persist the result when completion is known.
4. Mark the call as ambiguous when completion cannot be established.
5. Never replay the ambiguous call automatically under a different identity.

The corresponding experiment verifies both the expected number of physical calls and their outcomes.

This is the kind of detail that normal happy-path tests rarely cover.

Injecting a fault is the easy part

Creating latency with Toxiproxy takes a few lines of configuration.

For example:

1{
2  "type": "latency",
3  "stream": "downstream",
4  "toxicity": 1.0,
5  "attributes": {
6    "latency": 750,
7    "jitter": 50
8  }
9}

The hard part is making sure that:

• the fault targets the correct environment;
• the fault is actually active;
• the experiment cannot affect shared resources;
• rollback happens after both success and failure;
• the agent cannot call a partial run successful;
• the evidence does not contain credentials or customer data.

This is where most of our implementation work went.

Safety first, especially with an agent

Every chaos run happens inside a new, dedicated k3d namespace.

Before introducing a fault, the harness verifies the cluster, namespace, worker identities, candidate images and experiment configuration.

Names are not enough. The harness works with exact runtime identities and refuses to proceed if something changed after the configuration was reviewed.

Each mutation is also revalidated immediately before execution. If the target changed between preflight and mutation, the experiment stops.

This may sound paranoid. It should.

A coding agent that can delete a worker or interrupt a dependency needs stronger boundaries than an agent that can run a unit test.

The rule is simple:

If the harness cannot prove that the target belongs to the current experiment, it does nothing.

Prove that the fault happened

Another easy mistake is to assume that the fault is active because the mutation command succeeded.

We do not start the observation window after sending the mutation. We start it after proving activation.

Depending on the experiment, this means verifying that:

• the original worker disappeared;
• the dependency reached the expected unavailable state;
• the network fault exists with the requested attributes;
• traffic crossed the faulted connection;
• the worker observed the expected memory pressure.

Without activation proof, a passing experiment may simply mean that the fault never happened.

And yes, an LLM can happily summarize that as a successful resilience test if you only give it the command output.

Rollback is part of the experiment

Rollback is not cleanup added at the end. It is part of the experiment contract.

Before starting, the harness writes a restricted rollback journal containing the original state and the exact resource owned by the run.

The journal follows explicit transitions:

1prepared
2revalidated
3mutation attempted
4mutation confirmed
5restoration attempted
6restored

If the process crashes, recovery uses that exact journal. It does not look for the most recent file and hope it belongs to the same run.

We also avoid broad cleanup operations. If the experiment creates one named network fault, rollback removes that specific fault. It does not reset the complete proxy configuration.

If rollback fails, the harness keeps the state and evidence for diagnosis. It does not remove everything and return green.

Evidence instead of vibes

The coding agent needs structured evidence, not ten thousand lines of logs.

For each experiment, we collect a bounded snapshot containing:

• the current operation status;
• stable output identifiers;
• external call counts and outcomes;
• worker restart and OOM observations;
• recovery transitions;
• failure codes;
• proof that the injected fault carried traffic;
• proof that rollback completed.

The retained result is signed and redacted.

We exclude credentials, raw prompts, responses, payloads, endpoints and unfiltered logs.

This gives the agent enough information to investigate a failure without giving it unnecessary access to sensitive data.

It also prevents a common problem in automated qualification: accepting a plausible summary without being able to reproduce how that summary was produced.

Fixtures are not runtime evidence

We built deterministic fixtures for every experiment.

They help us validate the harness itself:

• configuration parsing;
• scenario generation;
• expected-result matching;
• rollback transitions;
• evidence generation;
• signature and tamper detection.

But fixture evidence is always marked as non-qualifying.

A mocked worker that survives a simulated network timeout does not prove that the real worker survived it inside Kubernetes.

This distinction becomes even more important with coding agents because they are very good at making local tests pass. If the boundary is not explicit, a green fixture can slowly become a “qualified” feature in the conversation history.

We do not allow that.

The improvement loop

Once the safety controls were in place, pi could work inside the loop.

A task for the agent looks more like this:

 1Run one reviewed scenario inside the dedicated k3d environment.
 2
 3Do not touch shared environments.
 4Do not broaden selectors or resource ownership.
 5Stop if preflight, fault activation, rollback or evidence verification fails.
 6Inspect only the redacted evidence.
 7Identify the smallest defect that violates the steady-state contract.
 8Fix it.
 9Run the unit and contract tests.
10Build a new immutable candidate.
11Repeat the same scenario.
12Do not weaken the experiment to make it pass.

This is very different from:

1Make the workers resilient.

The agent can generate experiment definitions, write probes, run tests, inspect failures, implement fixes and update runbooks.

The human still decides:

• which environment can be touched;
• which failure is safe to introduce;
• what the expected behaviour should be;
• whether the collected evidence is enough.

Agentic coding does not remove engineering responsibility. It increases the speed at which engineering decisions become running code, including bad decisions.

How to start applying this

If you want to introduce Chaos Engineering into your agentic coding process, start with one important workflow.

Do not start by randomly killing workers.

1. Pick one business operation

Choose something with a clear start and end, such as:

• processing a job;
• generating a report;
• importing a file;
• executing an external action.

2. Describe the steady state

Write down:

• the expected terminal state;
• the expected persisted outputs;
• the allowed number of retries;
• the number of external calls;
• the accepted failure codes;
• the convergence deadline;
• the allowed worker restarts and OOMs.

Turn those expectations into code.

3. Build an isolated environment

Create a disposable local cluster or namespace.

Use separate credentials and make ownership explicit. The experiment should fail closed if the environment does not match what was reviewed.

4. Introduce one reversible failure

Start with something simple:

• terminate one worker;
• add latency to the messaging layer;
• interrupt one callback;
• temporarily remove memory headroom.

Do not combine faults until you understand the recovery path for each one.

5. Prove activation and rollback

Do not trust the mutation command.

Measure that the fault became active and that the original state was restored afterwards.

6. Produce structured evidence

Give the coding agent IDs, counts, outcomes and failure codes.

Do not ask it to infer success from raw logs.

7. Keep the agent inside the contract

The agent can change the implementation, but it should not silently change:

• the steady state;
• the fault intensity;
• the allowed failure codes;
• the qualification boundary.

Those changes require human review.

8. Repeat until the same experiment passes

The final loop is simple:

1reproduce
2inspect
3fix
4validate
5rebuild
6repeat

Simple does not mean easy, but it gives the agent a concrete job and the team a reproducible result.

Where we are

At the time of writing, the experiment definitions, deterministic fixtures and syntax checks are in place. We have also completed a real baseline run against the ephemeral environment.

The full runtime Chaos Engineering qualification remains a separate gate. We are not treating fixture results or a healthy baseline as proof that every recovery scenario works.

This distinction is part of the engineering practice.

AI makes generating code and automation faster. It does not make partial evidence complete, and it does not turn a simulated failure into a production result.

For me, this has been the most useful part of bringing Chaos Engineering into an agentic coding loop.

The interesting question is no longer:

Can the agent write a chaos experiment?

Of course it can.

The real question is:

Can your harness stop the agent from running it against the wrong environment, accepting the wrong result or hiding a failed recovery?

That is the part worth engineering.