Agent Reliability: Restarting Stalled Agents and Estimating Rare Failures

by Akshat Valse

Advised by Yang Li, Department of Computer Science, Iowa State University

We study two questions about tool-using LLM agents that run for a long time and sometimes fail. When a run stalls, when is restarting it worth more than waiting, and what in the distribution of run times decides that? And how can we estimate the probability of a failure too rare for direct simulation to reach? Here we set out both problems and the classical baseline for each; our approach and results are held back until publication.

In Experimentation and Development

Three thick concentric arcs with rounded ends on a pink-to-violet background, all starting from the same point and running clockwise for different distances. The middle arc, pale blue deepening to indigo, stops before half way, and a pale band marks the rest of its lap.

Introduction

Agents that run for a long time sometimes stall and sometimes fail. That raises two questions:

  1. Restart or wait: when a run stalls, should we restart it or keep waiting, and what in the distribution of run times decides?

  2. Rare failures: how do we estimate the probability of a failure too rare for direct simulation to reach?

We ask both questions of real tool-using LLM agents. It is a research project in Advanced Topics in Artificial Intelligence (COMS 6720), Department of Computer Science, Iowa State University, advised by Prof. Yang Li. We are in the experimentation and development phase, starting from two pilot designs on real agents, one on restart policies and one on rare failures, each with its kill criteria registered before any data. The approach and results are held back until publication.

Both questions have textbook versions. For the first, the run-time law is known and restarts are independent of each other; for the second, failures are independent across runs. The first then has an exact answer, and for the second the classical arithmetic shows why direct simulation usually sees nothing. In the sections that follow, we work out those versions and let you vary them. Every quantitative statement from here on is about that classical arithmetic, computed and checked for this page: none of it is project data, and none of it describes our method.

Bears on: survival analysis · rare-event estimation · agent evaluation.
Advisor: Yang Li, Computer Science.
Status: experimentation and development, Aug. 2026 – present.

Restart or Wait

A run has been going for time τ\tau and has not finished. Should we kill it and start again, or keep waiting? In the textbook setting, with a known distribution of run times and independent restarts, the decision has an exact answer. Waiting costs, in expectation, the time the run still has to go: its mean residual life m(τ)=E[T−τ∣T>τ]m(\tau) = \mathbb{E}[T-\tau \mid T>\tau]. Restarting costs a relaunch cc plus a fresh run, c+E[T]c + \mathbb{E}[T]. So restart if and only if

m(τ)>E[T]+c.m(\tau) > \mathbb{E}[T] + c.

The same inequality decides whether a standing policy of restarting every run at cutoff τ\tau beats never restarting, as we show below. Classically, then, what in the distribution of run times decides is the mean residual life, compared with a fresh run plus the cost of a relaunch.

1. A run that is already τ old 0.1 1 10 now: age τ, not finished shaded: P(T > τ) = S(τ) run time T, multiples of the median (log axis) curve: density of ln T, lognormal with σ = 1.5 m(τ) = E[T − τ | T > τ], the time still to go 2. Wait, or restart? run stalled at age τ wait restart expected to go: m(τ) relaunch cost c, then a fresh run: E[T] restart iff m(τ) > E[T] + c the same inequality decides whether always restarting at cutoff τ beats never restarting Classical baseline for a known run-time law and independent restarts; drawn for a lognormal with σ = 1.5, τ = 2.

Figure 1. Question 1 in its classical form. A run has been going for τ\tau without finishing. Waiting costs, in expectation, the time it still has to go, m(τ)m(\tau); restarting costs the relaunch cc plus a fresh run. Restart if and only if m(τ)>E[T]+cm(\tau) > \mathbb{E}[T] + c. This assumes a known run-time law and independent restarts. We ask the question of real agents; our approach is held back.

The Classical Baseline

In this section we assume a run-time law that is known exactly and restarts that are independent and identically distributed. Nothing is fitted to agent data. No public source of the project states these derivations: we derived them for this page and checked them numerically (see Checks, below).

Notation

T≥0T \ge 0 is the run time of one attempt, with CDF FF, survival function S=1−FS = 1 - F, density ff and hazard h=f/Sh = f/S. A cutoff-τ\tau policy kills any attempt still running at age τ\tau and relaunches it at cost c≥0c \ge 0, paid on each relaunch but not on the first launch. Tτ,cT_{\tau,c} is the total time until the first attempt finishes under that policy.

Expected Time Under a Cutoff

Write A(τ)A(\tau) for the truncated mean:

A(τ)=E[min⁡(T,τ)]=∫0τS(t) dt.A(\tau) = \mathbb{E}[\min(T,\tau)] = \int_0^\tau S(t)\,dt.

The number of attempts is geometric with success probability F(τ)F(\tau), so the expected number of failed attempts is S/FS/F, and each costs τ+c\tau + c. The attempt that succeeds costs TT given T≤τT \le \tau, and E[T 1{T≤τ}]=A(τ)−τS(τ)\mathbb{E}[T\,\mathbf 1\{T\le\tau\}] = A(\tau) - \tau S(\tau). Hence

E[Tτ,c]=A(τ)+c S(τ)F(τ).\mathbb{E}[T_{\tau,c}] = \frac{A(\tau) + c\,S(\tau)}{F(\tau)}.

An Identity, and the Rule

The mean residual life of a run that has reached age τ\tau is

m(τ)=E[T−τ∣T>τ]=E[(T−τ)+]S(τ).m(\tau) = \mathbb{E}[T - \tau \mid T > \tau] = \frac{\mathbb{E}[(T-\tau)^+]}{S(\tau)}.

Splitting E[T]\mathbb{E}[T] on whether the run finishes by τ\tau gives

E[T]=(A(τ)−τS(τ))+S(τ)(τ+m(τ))=A(τ)+S(τ) m(τ).\mathbb{E}[T] = \big(A(\tau) - \tau S(\tau)\big) + S(\tau)\big(\tau + m(\tau)\big) = A(\tau) + S(\tau)\,m(\tau).

By the previous display, F(τ) E[Tτ,c]=A+cSF(\tau)\,\mathbb{E}[T_{\tau,c}] = A + cS. Adding S(τ)(m(τ)−c)S(\tau)\big(m(\tau) - c\big) gives the same quantity, so, whenever E[T]<∞\mathbb{E}[T] < \infty,

E[T]=F(τ) E[Tτ,c]+S(τ) (m(τ)−c).\mathbb{E}[T] = F(\tau)\,\mathbb{E}[T_{\tau,c}] + S(\tau)\,\big(m(\tau) - c\big).

So E[T]\mathbb{E}[T] is a weighted average of E[Tτ,c]\mathbb{E}[T_{\tau,c}], with weight F(τ)F(\tau), and m(τ)−cm(\tau) - c, with weight S(τ)S(\tau). For 0<F(τ)<10 < F(\tau) < 1 and finite E[T]\mathbb{E}[T], the first lies below the average exactly when the second lies above it:

E[Tτ,c]<E[T]  ⟺  m(τ)>E[T]+c.\mathbb{E}[T_{\tau,c}] < \mathbb{E}[T] \iff m(\tau) > \mathbb{E}[T] + c.

The same inequality decides the one-shot question for a run stalled at age τ\tau: restarting once and never again costs c+E[T]c + \mathbb{E}[T], and waiting costs m(τ)m(\tau).

The Best Cutoff

Differentiating the expected time under a cutoff,

ddτ E[Tτ,c]=SF−f (A+c)F2.\frac{d}{d\tau}\,\mathbb{E}[T_{\tau,c}] = \frac{S F - f\,(A + c)}{F^2}.

At an interior optimum A+c=SF/fA + c = SF/f, which gives

E[Tτ∗,c]=1h(τ∗)−c.\mathbb{E}[T_{\tau^*,c}] = \frac{1}{h(\tau^*)} - c.

Where restarting gains almost nothing, E[Tτ,c]\mathbb{E}[T_{\tau,c}] is flat to machine precision, and a search on its values, such as golden-section search, cannot locate τ∗\tau^*. Bisection on the sign of the numerator SF−f(A+c)SF - f(A + c) still can, and that is how the widget below finds it.

Lognormal Closed Forms

Let T=eμ+σZT = e^{\mu + \sigma Z} with ZZ standard normal, z=(ln⁡τ−μ)/σz = (\ln\tau - \mu)/\sigma, and Φ\Phi the standard normal CDF. Then

A(τ)=eμ+σ2/2 Φ(z−σ)+τ (1−Φ(z)),A(\tau) = e^{\mu+\sigma^2/2}\,\Phi(z-\sigma) + \tau\,\big(1-\Phi(z)\big),
E[(T−τ)+]=eμ+σ2/2 (1−Φ(z−σ))−τ (1−Φ(z)).\mathbb{E}[(T-\tau)^+] = e^{\mu+\sigma^2/2}\,\big(1-\Phi(z-\sigma)\big) - \tau\,\big(1-\Phi(z)\big).

Φ\Phi comes from the site's normal-tail routine (an erf series and the Laplace continued fraction), always evaluated on the tail that avoids cancellation. With median 1 (μ=0\mu = 0) the two main quantities take a few lines:

// Lognormal run time with median 1: T = exp(sigma * Z).
// upper(x) = P(Z > x) and lower(x) = P(Z <= x), each on its accurate tail.
function restartTime(tau, sigma, c) {
  const z = Math.log(tau) / sigma;
  const F = lower(z), S = upper(z);
  const A = Math.exp(sigma * sigma / 2) * lower(z - sigma) + tau * S;  // E[min(T, tau)]
  return (A + c * S) / F;                                              // E[T_{tau,c}]
}

function meanResidual(tau, sigma) {                                   // m(tau)
  const z = Math.log(tau) / sigma;
  return (Math.exp(sigma * sigma / 2) * upper(z - sigma) - tau * upper(z)) / upper(z);
}
Mean residual life m(tau) of a lognormal run (sigma 1.5) against E[T] + c, with the expected time of the cutoff-tau restart policy; restart helps exactly where m(tau) exceeds E[T] + c. Exact. sigma = 1.5, c = 0.5. E[T] = 3.0802. Break-even tau0 = 0.30917; optimum tau* = 1.5630 with E[T_tau*,c] = 1.8526. 0.010.111010010000246810 age of the current run τ (also the cutoff), in multiples of the median time, in multiples of the median E[T] + c E[T] m(τ): expected time still to go for a run already τ old E[Tτ,c], best at τ* = 1.56 shaded: restart helps, τ > τ₀ = 0.309 both crossings happen at τ₀ (dotted)

Figure 2. What in the distribution decides it, classically: the mean residual life. Lognormal run time with σ=1.5\sigma = 1.5 and relaunch cost c=0.5c = 0.5, in multiples of the median. Restarting at cutoff τ\tau beats never restarting exactly where m(τ)m(\tau) exceeds E[T]+c\mathbb{E}[T] + c. The two crossings fall at the same τ0=0.309\tau_0 = 0.309, because E[T]=F(τ) E[Tτ,c]+S(τ) (m(τ)−c)\mathbb{E}[T] = F(\tau)\,\mathbb{E}[T_{\tau,c}] + S(\tau)\,(m(\tau) - c). Best cutoff τ∗=1.56\tau^* = 1.56, where expected time is 1.85 against 3.08. Classical baseline, not project data.

Read Figure 2 from left to right. Before τ0\tau_0, a run that has gone τ\tau is expected to finish sooner than a relaunch plus a fresh run, so a cutoff there only adds time. Past τ0\tau_0, every cutoff in the shaded region beats never restarting, and the best of them is τ∗\tau^*.

Why both crossings share τ0\tau_0: with weights FF and SS, E[T]\mathbb{E}[T] averages E[Tτ,c]\mathbb{E}[T_{\tau,c}] and m−cm - c, so one equals E[T]\mathbb{E}[T] exactly when the other does.

How Much Restarting Buys

The rule says when restarting helps; how much it helps depends on the shape of the law. Figure 3 holds the lognormal family at median 1 and varies σ\sigma, with free relaunches on one side of the slider and a relaunch cost of half a median run time on the other. Drag the divider to compare. Each curve is E[Tτ,c]/E[T]\mathbb{E}[T_{\tau,c}]/\mathbb{E}[T], and the dots mark the best cutoffs.

Expected completion time with restarts at cutoff tau, divided by the expected time without restarts, for lognormal run times with sigma from 0.5 to 2.5 and relaunch cost 0 median run times. Classical baseline, exact (closed form via the normal CDF). Dots: optimal cutoffs sigma 0.5: tau* 11.5, ratio 1.00; sigma 1: tau* 1.73, ratio 0.892; sigma 1.5: tau* 0.349, ratio 0.413; sigma 2: tau* 0.0512, ratio 0.0976; sigma 2.5: tau* 0.00502, ratio 0.0128. 0.0010.010.111010010000.00.20.40.60.81.01.21.4 restart cutoff τ, in multiples of the median run time E[Tτ,c] ÷ E[T] never restart σ = 0.5 σ = 1 σ = 1.5 σ = 2 σ = 2.5 c = 0
c = 0
Expected completion time with restarts at cutoff tau, divided by the expected time without restarts, for lognormal run times with sigma from 0.5 to 2.5 and relaunch cost 0.5 median run times. Classical baseline, exact (closed form via the normal CDF). Dots: optimal cutoffs sigma 0.5: tau* 20.1, ratio 1.00; sigma 1: tau* 3.68, ratio 0.955; sigma 1.5: tau* 1.56, ratio 0.601; sigma 2: tau* 0.886, ratio 0.248; sigma 2.5: tau* 0.592, ratio 0.0752. 0.0010.010.111010010000.00.20.40.60.81.01.21.4 restart cutoff τ, in multiples of the median run time E[Tτ,c] ÷ E[T] never restart σ = 0.5 σ = 1 σ = 1.5 σ = 2 σ = 2.5 c = 0.5
c = 0.5

Figure 3. Classical baseline, not project data. Lognormal run times with median 1. With free restarts (left), restarts help high-variance runs enormously: at σ=2\sigma = 2 the best cutoff, about a twentieth of the median, cuts expected time below a tenth. They help low-variance runs essentially not at all: at σ=0.5\sigma = 0.5 the gain is 3 parts in 10⁸. With a relaunch cost of half a median run time (right), every best cutoff moves later and every gain shrinks: σ=2\sigma = 2 now saves three quarters (ratio 0.248), not nine tenths. Exact closed form, checked against quadrature and, at σ=2\sigma = 2, a seeded simulation.

A lognormal is one shape among many. Figure 4 has two others, chosen to make a point rather than fitted to anything: a law with two modes, and runs that can hang forever.

Expected completion time against restart cutoff for a two-mode run-time law, most runs healthy and a minority slow, at three relaunch costs. Exact. Mixture 0.8 x lognormal(0, 0.5) + 0.2 x lognormal(3, 0.5); E[T] = 5.4585. Optima: c 0: tau* 1.826, E 1.722; c 0.25: tau* 1.963, E 1.820; c 1: tau* 2.229, E 2.076. 0.1110100100001234567 restart cutoff τ, in multiples of the healthy median expected time to finish E[Tτ,c] never restart: E[T] = 5.46 relaunch costc = 0c = 0.25c = 1 density of T (log axis): 80% fast, 20% slow 0.211050

Figure 4. Classical baseline, not project data: two illustrative laws, not fitted to any agent (use the dots to switch). Two-mode: 80% of runs are fast (median 1, σ=0.5\sigma = 0.5) and a fifth are slow (median e3e^3, about 20, σ=0.5\sigma = 0.5), at relaunch costs c=0c = 0, 0.25 and 1. With free relaunches, killing any run still going at about 1.8 fast medians cuts expected time from 5.46 to 1.72. Runs that hang: each run hangs forever with probability qq = 0.01, 0.05 or 0.2, and finishing runs are lognormal with σ=1\sigma = 1. Never restarting then takes infinitely long, so some cutoff is mandatory. With free relaunches the best cutoff stays between about 1.1 and 1.7 medians as qq goes from 0.2 to 0.01.

Restart or Wait, Interactively

Figure 5 computes the classical quantities for a law you choose: E[T]\mathbb{E}[T], F(τ)F(\tau), m(τ)m(\tau), E[Tτ,c]\mathbb{E}[T_{\tau,c}], whether restarting at τ\tau helps, and the best cutoff τ∗\tau^*. Drag across the plot or use the sliders. Beside the exact value of E[Tτ,c]\mathbb{E}[T_{\tau,c}], a seeded simulation of the restart process runs to 200,000 replications, and its running mean, with one standard error, converges onto the exact value. At cutoffs so early that one replication would need too many attempts to run live, the widget says so instead.

Two cases need care. If runs can hang, E[T]\mathbb{E}[T] and m(τ)m(\tau) are infinite, the identity and the rule do not apply, and any finite cutoff beats never restarting. And where the best attainable gain is below 10−610^{-6} of E[T]\mathbb{E}[T], τ∗\tau^* is numerically ill-defined, so the widget reports that restarting gains nothing measurable instead of printing one.

Restart or wait: the classical question, for a run-time law you know

classical baseline, not project data · exact: closed form in Φ · check: seeded simulation of the restart process
τ
E[T], never restarting 3.08
F(τ), finished by τ —
m(τ), time still to go —
E[T] + c —
—
E[Tτ,c], exact —
simulated —
—
best cutoff τ* —
E[Tτ*,c] —

Figure 5. Classical baseline, not project data: a known run-time law and independent restarts. Exact values come from the closed forms above, with Φ\Phi from the site's normal tail; τ∗\tau^* comes from bisection on the sign of SF−f(A+c)SF - f(A+c). The simulation is seeded, so the same settings give the same digits. The defaults are the case of Figure 2, σ=1.5\sigma = 1.5 and c=0.5c = 0.5, where τ∗=1.563\tau^* = 1.563 and E[Tτ∗,c]=1.853\mathbb{E}[T_{\tau^*,c}] = 1.853 against E[T]=3.080\mathbb{E}[T] = 3.080. The two-mode law keeps σ=0.5\sigma = 0.5 for both modes with the fast median at 1, so its times are in fast medians; the hanging law keeps finishing runs lognormal with median 1 and σ=1\sigma = 1.

Reference Values

The lognormal family with median 1, at the cutoffs that minimise expected time. At σ=0.5\sigma = 0.5 the best cutoff is ill-conditioned (Checks, below), so the table reports the gain instead.

σ\sigma E[T]\mathbb{E}[T] cc τ∗\tau^* (medians) E[Tτ∗,c] / E[T]\mathbb{E}[T_{\tau^*,c}]\,/\,\mathbb{E}[T]
0.51.1330no measurable gain1.0000000 (gain 3.3 × 10⁻⁸)
1.01.64901.730.892
1.01.6490.53.680.955
1.53.08000.3490.413
1.53.0800.51.560.601
2.07.38900.05120.0976
2.07.3890.50.8860.248
2.522.7600.005020.0128
2.522.760.50.5920.0752

Classical baseline (known lognormal run-time law, independent restarts); not project data. Every row was also reproduced by an independent implementation.

Checks

These are checks of this page's classical arithmetic only, not of the project. The closed forms were checked numerically before publication, against pass criteria recorded in a configuration file as fixed before the first run. Of the 69 registered checks, 68 passed and one failed.

CheckCasesWorst observedResult
Closed form against numerical quadrature9 laws × 6 cutoffs4.5 × 10⁻¹⁵ relativepass
The identity E[T]=F E[Tτ,c]+S (m−c)\mathbb{E}[T] = F\,\mathbb{E}[T_{\tau,c}] + S\,(m - c)6 finite-mean laws × 3 values of cc, 8001-point grid5.9 × 10⁻¹⁶ relativepass
The sign rule E[Tτ,c]<E[T]  ⟺  m(τ)>E[T]+c\mathbb{E}[T_{\tau,c}] < \mathbb{E}[T] \iff m(\tau) > \mathbb{E}[T] + c18 cases0 sign mismatchespass
First-order condition at τ∗\tau^*20 cases1.06 × 10⁻⁴ against a 10⁻⁵ criterion (σ=0.5\sigma = 0.5, c=0.5c = 0.5)19 pass, 1 fail
Seeded simulation of the restart process, 200,000 replications per case5 cases|z| = 2.76 against a criterion of 4pass

The failure is one of solver conditioning, not a counterexample. At σ=0.5\sigma = 0.5 the objective is flat to machine precision, so golden-section search cannot locate τ∗\tau^*; bisection on the numerator SF−f(A+c)SF - f(A+c) satisfies the condition to 7.8 × 10⁻¹⁶, and an independent bisection agrees, with τ∗=20.13\tau^* = 20.13 and a gain of 8.0 × 10⁻¹¹. That diagnosis was made after the fact and is not a registered check; the registered failure stands as recorded. The σ=0.5\sigma = 0.5, c=0c = 0 check passes only narrowly (5.8 × 10⁻⁶ against 10⁻⁵), for the same reason. Also after the fact, 10 further seeds for the simulation case with the largest ∣z∣|z| gave a mean zz of −0.031. Every reference value on this page was reproduced independently in Python, with math.erfc and bisection.

Failures Too Rare to See

The second question is how to estimate the probability of a failure too rare for direct simulation to reach. Picture an agent run as a loop of model and tool calls that, rarely, ends in failure. With per-run failure probability pp and NN independent runs, the number of failures is K∼Bin(N,p)K \sim \mathrm{Bin}(N, p), and the classical arithmetic is short. The chance of seeing no failure at all is

Pr⁡(K=0)=(1−p)N≈e−Np,\Pr(K = 0) = (1-p)^N \approx e^{-Np},

the relative standard error of the estimate p^=K/N\hat p = K/N is

sd(p^)p=1−pNp,\frac{\mathrm{sd}(\hat p)}{p} = \sqrt{\frac{1-p}{Np}},

and when K=0K = 0 the one-sided upper bound is

pU=1−α1/N≈−ln⁡αN≈3Nat α=0.05.p_{\mathrm{U}} = 1 - \alpha^{1/N} \approx \frac{-\ln \alpha}{N} \approx \frac{3}{N} \quad\text{at } \alpha = 0.05.

So when failures are much rarer than one in the number of runs we can afford, direct simulation usually sees none, and zero failures in NN runs supports only pp below about 3/N3/N: for N=1000N = 1000, p<0.00299p < 0.00299 at 95%. Crude Monte Carlo then stops being an estimator: it returns zero occurrences and no information about the magnitude.

Diagram: an agent run as a chain of model and tool calls, each with a rare chance of failure; N simulated runs all finish, so the crude estimate is zero and supports only p below about 3 over N. Mechanism diagram for the rare-failure estimation problem. Not project data; symbols only. 1. One agent run: a loop of model and tool calls that can, rarely, end in failure model call tool call model call tool call … done failure, probability p per run, too small to see 2. Direct simulation: N independent runs every run ends in done 0 failures in N runs estimate of p: 0 / N = 0 chance of this: (1 − p)^N ≈ e^(−Np) all it supports (95%): p < 1 − 0.05^(1/N) ≈ 3/N relative s.e.: √((1 − p)/(Np)), ≫ 1 when Np ≪ 1 The problem only; this project's approach is held back.

Figure 6. Question 2. When failures are much rarer than one in the number of runs you can afford, direct simulation usually sees none, and zero failures in NN runs supports only pp below about 3/N3/N. Drawing a failure as reachable from every call is illustrative. This is the problem only; this project's approach is held back.

Figure 7 does that arithmetic for pp from 10−1210^{-12} to 0.1 and NN from 10 to 10710^7, with a seeded direct simulation beside the exact value: whole experiments of NN runs, counting how many saw no failure. Push pp well below 1/N1/N and nearly every experiment comes back empty.

How many runs before you see a failure? (classical arithmetic)

not project data · exact: (1 − p)N · reference: p = 0.001, N = 1000 gives 0.3677 · check: seeded direct simulation
N = 1/p
expected failures, Np —
P(no failure in N runs), exact —
simulated —
—
relative s.e. of K/N —
if none seen, 95% bound on p —
3/N, for comparison —

Figure 7. Classical arithmetic, not project data. The curve is the exact chance of no failure in NN runs, (1−p)N(1-p)^N, evaluated as exp⁡(N log1p(−p))\exp(N\,\mathrm{log1p}(-p)); the hollow dot is the fraction of seeded simulated experiments that saw none, with one binomial standard error at the exact value. The bound after zero failures is evaluated as −expm1(ln⁡(0.05)/N)-\mathrm{expm1}(\ln(0.05)/N). At the defaults, p=0.001p = 0.001 and N=1000N = 1000, the exact chance of seeing nothing is 0.3677.

Status

The project has been in the experimentation and development phase since Aug. 2026, and Figure 8 shows how it is set up. The approach and any results are held back until publication. That includes the methods, models, agents, tools, environments, benchmarks and estimators, which restart policies a pilot would test, the pilot designs and the content of their kill criteria, and which direction the experiments take. No number on this page is a project result: every one is classical arithmetic computed for this page. No paper, preprint or repository for this project is public.

Two questions about long-running agents Pilot A design restart policies Pilot B design rare failures kill criteria registered before any data kill criteria registered before any data experimentation and development

Figure 8. How the project is set up. The pilot designs and the content of the kill criteria are not public, and the figure implies no preference between the two pilots.


Early-Feedback Calibration    All research