Simulation: the key to reliable agents

By Charlie Feng, Co-founder and CEO, Runner — 2026-10-29

Good agents don't practice on your customers

Runner Simulation is how we test agents before they go live. It builds a copy of your business, populates it with simulated people, runs weeks of activity in hours and grades every conversation the agent has.

We built it because of the way agents fail. Consider an agent that helps patients coordinate care between their insurer and their doctors. It performs well in development and in its first days of use. Two weeks after launch, it tells a patient something false about a diagnosis. The team replays the conversation with the same inputs 11 times, and the agent answers correctly each time.

In a case like this, there is usually no bug to find. The agent made an error that it makes about once in 100,000 runs, and reproducing the error would take about that many attempts. That volume of testing isn't practical on a developer's laptop, and it can't be done on real customers.

Our answer is to build a copy of your environment and test agents there, at volume, before they handle real work. This is how we keep agents reliable at scale in regulated industries.

Why agents need simulation

Traditional software is reliable in large part because it is deterministic. The same input produces the same output, and every unit test depends on that property.

Agents built on language models don't have it. The same input can produce a different answer on the next run, so a single passing test says little about how the agent will behave later. Reliability has to be measured statistically, by running many trials and counting how often the agent fails.

If you run a probabilistic system, you might have to run it 1,000, 10,000, or 100,000 times before you see the failure. Teams miss this because the agent is so easy to deploy. — Zac Smith, Runner engineering

Those trials can't be run on real people. A hospital won't allow testing on its patients, and a rideshare company won't allow it on its drivers.

The failures that matter most are also hard to catch with conventional tests. A fixed rule such as "no texts after 9 p.m." can be enforced in code. The more costly mistakes are matters of judgment in conversation, such as an agent that texts too often, sounds automated or keeps messaging someone who has threatened to sue. Finding those mistakes requires putting the agent in conversations with people, or with something that behaves like them.

How it works

Runner Simulation runs agents inside a World, which is a sealed copy of your business. Building and running a World involves four steps.

We copy your systems

The first step is to copy the systems the agent works in, including inboxes, text messages, payments and records, into the World.

Each outside system is replaced by a double that looks and behaves like the real one. Doubles exist today for Gmail, Outlook, Microsoft Teams, Google Calendar, Slack, SMS, WhatsApp, Google Sheets, Stripe and DocuSign. Inside the World, the agent sends email, collects payments and requests signatures with the same tools it uses in production, but nothing leaves the World. If an agent depends on an app that doesn't have a double yet, Runner reports that before the run starts.

Every World starts from a snapshot of your real data taken at a set moment. Each run begins from the same state, which makes it possible to compare one run with the next.

Each live system gets a double inside the sealed World.

We fill it with people who try to break the rules

Each World is populated with simulated customers, drivers or patients, depending on the business. Every simulated person has a private persona, a phone number or inbox, and a goal. They behave the way real people do when a conversation gets difficult: They stop responding, push back, refuse to pay or ask to speak with a person.

When the agent escalates a problem to your team, a simulated teammate answers so the run can continue.

One scenario, 240 simulated people.

We speed up time

Much of the work agents do plays out over days or weeks. A driver uploads a license on Tuesday and doesn't respond again until Friday. An invoice comes due in 30 days.

A World runs on a single clock that Runner controls. While the agent is working, the clock runs at normal speed. When every participant is waiting, the clock jumps ahead. This allows a month of conversations to run in an afternoon, and it means the amount of time you can test depends mainly on how much you want to spend.

Real time moves steadily. World time jumps ahead whenever everyone is waiting.

We grade every conversation

Each fleet of agents has a set of pass conditions. These are the rules of your business, written in plain English, and each one has a test attached. Examples include:

• Don't text a patient so often that it annoys them.
• Sound like a person, not a bot.
• Stop and tell the team when someone threatens to sue.
• Hand a problem to a person when you can't solve it.
• Answer each message within 30 minutes.

When a run ends, a grader reviews every conversation. Rules that can be measured, such as message counts and response times, are checked in code. For rules that require judgment, a judge model reads each message the agent sent along with the earlier messages from that person. The result is a grade for each person on each rule.

When the agent fails a rule, we fix the cause and run the scenario again. We repeat that process until the agent passes.

Each run is graded person by person. We fix what broke and run again.

Simulation runs in production

Worlds run inside Runner's production system and use the same agents, tools and automations that run live. The only differences are the doubles, the clock and the seal that keeps activity from leaving the World. As a result, a fix that passes in a World can go live without changes.

Live incidents also become scenarios. When something goes wrong in production, we capture your data from just before the failure and replay it as a new scenario. That scenario stays in the test set, so the same failure is caught if it comes back.

We use the same process to build Runner itself. Our engineers write the scenario and pass conditions first, confirm that the current agent fails them and then make the fix. Our coding agents test their own changes in simulation before a person reviews them, and new features run in simulation before customers see them.

Pilots train on the ground

Other fields adopted this approach long ago. Pilots train in flight simulators before they carry passengers, and surgeons practice on manikins before they operate on patients. Where mistakes are expensive, people practice in a safe setting first. We think agents that talk to your customers should meet the same standard.

What it means for your business

• You define what passing means. You write the pass conditions, and an agent goes live only after it meets them.
• Launches take less time. Because a month of activity runs in an afternoon, testing doesn't require a three-month pilot.
• Test coverage grows over time. Each live incident is added as a scenario that later versions of the agent must pass.
• There is a record of every run. Each run keeps every message and decision, and your team, an auditor or a customer can review them.

The limit

A simulation doesn't capture everything about the real world, and passing one doesn't prove that an agent will work in production. It is a minimum bar: An agent that can't pass a realistic simulation is unlikely to do well with real customers. Because each live failure is added as a new scenario, the World becomes a closer match to real conditions over time.

By the time a Runner agent goes live, it has already worked through weeks or months of simulated conversations. To see your own workflow run in a World, talk to us.