Read Figure 2 from left to right. Before , a run that has gone is expected to finish sooner than a relaunch plus a fresh run, so a cutoff there only adds time. Past , every cutoff in the shaded region beats never restarting, and the best of them is .
Agent Reliability: Restarting Stalled Agents and Estimating Rare Failures
We study two questions about tool-using LLM agents that run for a long time and sometimes fail. When a run stalls, when is restarting it worth more than waiting, and what in the distribution of run times decides that? And how can we estimate the probability of a failure too rare for direct simulation to reach? Here we set out both problems and the classical baseline for each; our approach and results are held back until publication.
In Experimentation and Development
Introduction
Agents that run for a long time sometimes stall and sometimes fail. That raises two questions:
-
Restart or wait: when a run stalls, should we restart it or keep waiting, and what in the distribution of run times decides?
-
Rare failures: how do we estimate the probability of a failure too rare for direct simulation to reach?
We ask both questions of real tool-using LLM agents. It is a research project in Advanced Topics in Artificial Intelligence (COMS 6720), Department of Computer Science, Iowa State University, advised by Prof. Yang Li. We are in the experimentation and development phase, starting from two pilot designs on real agents, one on restart policies and one on rare failures, each with its kill criteria registered before any data. The approach and results are held back until publication.
Restart or Wait
A run has been going for time and has not finished. Should we kill it and start again, or keep waiting? In the textbook setting, with a known distribution of run times and independent restarts, the decision has an exact answer. Waiting costs, in expectation, the time the run still has to go: its mean residual life . Restarting costs a relaunch plus a fresh run, . So restart if and only if
The same inequality decides whether a standing policy of restarting every run at cutoff beats never restarting, as we show below. Classically, then, what in the distribution of run times decides is the mean residual life, compared with a fresh run plus the cost of a relaunch.
Figure 1. Question 1 in its classical form. A run has been going for without finishing. Waiting costs, in expectation, the time it still has to go, ; restarting costs the relaunch plus a fresh run. Restart if and only if . This assumes a known run-time law and independent restarts. We ask the question of real agents; our approach is held back.
The Classical Baseline
In this section we assume a run-time law that is known exactly and restarts that are independent and identically distributed. Nothing is fitted to agent data. No public source of the project states these derivations: we derived them for this page and checked them numerically (see Checks, below).
Notation
is the run time of one attempt, with CDF , survival function , density and hazard . A cutoff- policy kills any attempt still running at age and relaunches it at cost , paid on each relaunch but not on the first launch. is the total time until the first attempt finishes under that policy.
Expected Time Under a Cutoff
Write for the truncated mean:
The number of attempts is geometric with success probability , so the expected number of failed attempts is , and each costs . The attempt that succeeds costs given , and . Hence
An Identity, and the Rule
The mean residual life of a run that has reached age is
Splitting on whether the run finishes by gives
By the previous display, . Adding gives the same quantity, so, whenever ,
So is a weighted average of , with weight , and , with weight . For and finite , the first lies below the average exactly when the second lies above it:
The same inequality decides the one-shot question for a run stalled at age : restarting once and never again costs , and waiting costs .
The Best Cutoff
Differentiating the expected time under a cutoff,
At an interior optimum , which gives
Where restarting gains almost nothing, is flat to machine precision, and a search on its values, such as golden-section search, cannot locate . Bisection on the sign of the numerator still can, and that is how the widget below finds it.
Lognormal Closed Forms
Let with standard normal, , and the standard normal CDF. Then
comes from the site's normal-tail routine (an erf series and the Laplace continued fraction), always evaluated on the tail that avoids cancellation. With median 1 () the two main quantities take a few lines:
// Lognormal run time with median 1: T = exp(sigma * Z).
// upper(x) = P(Z > x) and lower(x) = P(Z <= x), each on its accurate tail.
function restartTime(tau, sigma, c) {
const z = Math.log(tau) / sigma;
const F = lower(z), S = upper(z);
const A = Math.exp(sigma * sigma / 2) * lower(z - sigma) + tau * S; // E[min(T, tau)]
return (A + c * S) / F; // E[T_{tau,c}]
}
function meanResidual(tau, sigma) { // m(tau)
const z = Math.log(tau) / sigma;
return (Math.exp(sigma * sigma / 2) * upper(z - sigma) - tau * upper(z)) / upper(z);
} Figure 2. What in the distribution decides it, classically: the mean residual life. Lognormal run time with and relaunch cost , in multiples of the median. Restarting at cutoff beats never restarting exactly where exceeds . The two crossings fall at the same , because . Best cutoff , where expected time is 1.85 against 3.08. Classical baseline, not project data.
Why both crossings share : with weights and , averages and , so one equals exactly when the other does.
How Much Restarting Buys
The rule says when restarting helps; how much it helps depends on the shape of the law. Figure 3 holds the lognormal family at median 1 and varies , with free relaunches on one side of the slider and a relaunch cost of half a median run time on the other. Drag the divider to compare. Each curve is , and the dots mark the best cutoffs.
Figure 3. Classical baseline, not project data. Lognormal run times with median 1. With free restarts (left), restarts help high-variance runs enormously: at the best cutoff, about a twentieth of the median, cuts expected time below a tenth. They help low-variance runs essentially not at all: at the gain is 3 parts in 10⁸. With a relaunch cost of half a median run time (right), every best cutoff moves later and every gain shrinks: now saves three quarters (ratio 0.248), not nine tenths. Exact closed form, checked against quadrature and, at , a seeded simulation.
A lognormal is one shape among many. Figure 4 has two others, chosen to make a point rather than fitted to anything: a law with two modes, and runs that can hang forever.
Figure 4. Classical baseline, not project data: two illustrative laws, not fitted to any agent (use the dots to switch). Two-mode: 80% of runs are fast (median 1, ) and a fifth are slow (median , about 20, ), at relaunch costs , 0.25 and 1. With free relaunches, killing any run still going at about 1.8 fast medians cuts expected time from 5.46 to 1.72. Runs that hang: each run hangs forever with probability = 0.01, 0.05 or 0.2, and finishing runs are lognormal with . Never restarting then takes infinitely long, so some cutoff is mandatory. With free relaunches the best cutoff stays between about 1.1 and 1.7 medians as goes from 0.2 to 0.01.
Restart or Wait, Interactively
Figure 5 computes the classical quantities for a law you choose: , , , , whether restarting at helps, and the best cutoff . Drag across the plot or use the sliders. Beside the exact value of , a seeded simulation of the restart process runs to 200,000 replications, and its running mean, with one standard error, converges onto the exact value. At cutoffs so early that one replication would need too many attempts to run live, the widget says so instead.
Two cases need care. If runs can hang, and are infinite, the identity and the rule do not apply, and any finite cutoff beats never restarting. And where the best attainable gain is below of , is numerically ill-defined, so the widget reports that restarting gains nothing measurable instead of printing one.
Figure 5. Classical baseline, not project data: a known run-time law and independent restarts. Exact values come from the closed forms above, with from the site's normal tail; comes from bisection on the sign of . The simulation is seeded, so the same settings give the same digits. The defaults are the case of Figure 2, and , where and against . The two-mode law keeps for both modes with the fast median at 1, so its times are in fast medians; the hanging law keeps finishing runs lognormal with median 1 and .
Reference Values
The lognormal family with median 1, at the cutoffs that minimise expected time. At the best cutoff is ill-conditioned (Checks, below), so the table reports the gain instead.
| (medians) | ||||
|---|---|---|---|---|
| 0.5 | 1.133 | 0 | no measurable gain | 1.0000000 (gain 3.3 × 10⁻⁸) |
| 1.0 | 1.649 | 0 | 1.73 | 0.892 |
| 1.0 | 1.649 | 0.5 | 3.68 | 0.955 |
| 1.5 | 3.080 | 0 | 0.349 | 0.413 |
| 1.5 | 3.080 | 0.5 | 1.56 | 0.601 |
| 2.0 | 7.389 | 0 | 0.0512 | 0.0976 |
| 2.0 | 7.389 | 0.5 | 0.886 | 0.248 |
| 2.5 | 22.76 | 0 | 0.00502 | 0.0128 |
| 2.5 | 22.76 | 0.5 | 0.592 | 0.0752 |
Classical baseline (known lognormal run-time law, independent restarts); not project data. Every row was also reproduced by an independent implementation.
Checks
These are checks of this page's classical arithmetic only, not of the project. The closed forms were checked numerically before publication, against pass criteria recorded in a configuration file as fixed before the first run. Of the 69 registered checks, 68 passed and one failed.
| Check | Cases | Worst observed | Result |
|---|---|---|---|
| Closed form against numerical quadrature | 9 laws × 6 cutoffs | 4.5 × 10⁻¹⁵ relative | pass |
| The identity | 6 finite-mean laws × 3 values of , 8001-point grid | 5.9 × 10⁻¹⁶ relative | pass |
| The sign rule | 18 cases | 0 sign mismatches | pass |
| First-order condition at | 20 cases | 1.06 × 10⁻⁴ against a 10⁻⁵ criterion (, ) | 19 pass, 1 fail |
| Seeded simulation of the restart process, 200,000 replications per case | 5 cases | |z| = 2.76 against a criterion of 4 | pass |
The failure is one of solver conditioning, not a counterexample. At
the objective is flat to machine precision, so
golden-section search cannot locate ; bisection on the numerator
satisfies the condition to 7.8 × 10⁻¹⁶, and an independent bisection
agrees, with and a gain of 8.0 × 10⁻¹¹. That diagnosis
was made after the fact and is not a registered check; the registered failure stands as
recorded. The , check passes only
narrowly (5.8 × 10⁻⁶ against 10⁻⁵), for the same reason. Also after the fact, 10 further seeds
for the simulation case with the largest gave a mean of
−0.031. Every reference value on this page was reproduced independently in Python, with
math.erfc and bisection.
Failures Too Rare to See
The second question is how to estimate the probability of a failure too rare for direct simulation to reach. Picture an agent run as a loop of model and tool calls that, rarely, ends in failure. With per-run failure probability and independent runs, the number of failures is , and the classical arithmetic is short. The chance of seeing no failure at all is
the relative standard error of the estimate is
and when the one-sided upper bound is
So when failures are much rarer than one in the number of runs we can afford, direct simulation usually sees none, and zero failures in runs supports only below about : for , at 95%. Crude Monte Carlo then stops being an estimator: it returns zero occurrences and no information about the magnitude.
Figure 6. Question 2. When failures are much rarer than one in the number of runs you can afford, direct simulation usually sees none, and zero failures in runs supports only below about . Drawing a failure as reachable from every call is illustrative. This is the problem only; this project's approach is held back.
Figure 7 does that arithmetic for from to 0.1 and from 10 to , with a seeded direct simulation beside the exact value: whole experiments of runs, counting how many saw no failure. Push well below and nearly every experiment comes back empty.
Figure 7. Classical arithmetic, not project data. The curve is the exact chance of no failure in runs, , evaluated as ; the hollow dot is the fraction of seeded simulated experiments that saw none, with one binomial standard error at the exact value. The bound after zero failures is evaluated as . At the defaults, and , the exact chance of seeing nothing is 0.3677.
Status
The project has been in the experimentation and development phase since Aug. 2026, and Figure 8 shows how it is set up. The approach and any results are held back until publication. That includes the methods, models, agents, tools, environments, benchmarks and estimators, which restart policies a pilot would test, the pilot designs and the content of their kill criteria, and which direction the experiments take. No number on this page is a project result: every one is classical arithmetic computed for this page. No paper, preprint or repository for this project is public.
Figure 8. How the project is set up. The pilot designs and the content of the kill criteria are not public, and the figure implies no preference between the two pilots.
Both questions have textbook versions. For the first, the run-time law is known and restarts are independent of each other; for the second, failures are independent across runs. The first then has an exact answer, and for the second the classical arithmetic shows why direct simulation usually sees nothing. In the sections that follow, we work out those versions and let you vary them. Every quantitative statement from here on is about that classical arithmetic, computed and checked for this page: none of it is project data, and none of it describes our method.
Bears on: survival analysis · rare-event estimation · agent evaluation.
Advisor: Yang Li, Computer Science.
Status: experimentation and development, Aug. 2026 – present.