Skip to content

Agent Foresight

Can agents predict the effects of their own modifications?

Author
Han Luo
Date
Status
Research note · preliminary results

From proposing a change to anticipating its effects

An agent can propose a plausible improvement to its prompts, tools, or memory. It can explain why the change should help. The harder question is whether that explanation predicts what will actually happen. A modification aimed at fixing one failure may leave the failure untouched, or may disrupt behavior that previously worked.

In an iterative improvement loop, execution provides the answer: generate a candidate, evaluate it, and use the feedback to choose the next version. We are interested in the judgment that comes before that evaluation. Could an agent identify a useful change in advance? Could it recognize a regression? If several changes are available, could it select the most promising one?

We call this capability Agent Foresight: predicting the performance consequences of a concrete modification by inspecting the original system, the proposed change, and the task. Reliable foresight could help prioritize evaluations and identify risky changes. Whether it can do so is an empirical question.

The modification is fixed before the prediction is made. This separates two abilities that are easy to conflate: generating a good candidate and judging a given candidate. A model that performs well on one need not perform well on the other. In particular, our pilot compares several predictors on candidates generated by the same model; it does not test whether each predictor can generate better modifications itself.

What is being modified?

Our initial setting concerns agent artifacts: the persistent instructions and software around a model. These include prompts, skills, tool descriptions, long-term memory, middleware, tool implementations, and sub-agent organization. Together, these artifacts form the agent’s harness.

For a shared parent harness, we consider four candidate types. Structural candidates change middleware, tools, or sub-agents. Guidance candidates change prompts, skills, tool descriptions, or memory. A Free candidate is independently designed and may change either category or both. A Merge candidate combines the Structural and Guidance patches, resolving conflicts when necessary. Free and Merge therefore represent different ways of obtaining a candidate.

Structural, Guidance and Free candidates are independently generated from the same parent harness. Merge combines the Structural and Guidance patches.
Figure 1. Four candidate modifications of the same parent harness. Structural, Guidance, and Free are generated independently; Merge combines the first two.

The broader research direction also includes changes to model weights. The predictor would inspect the weights and the actual proposed parameter changes. We describe this exploratory setting below; it is not part of the pilot results.

Two prediction tasks

Let A be the original agent, m a proposed modification, and J its performance on a specified task or task set. The quantity we would like to anticipate is:

ΔJ = J(m(A)) − J(A).

The prediction must be made without access to the modified agent’s evaluation outcome. The intended protocol allows inspection of the change; running the target evaluation to obtain the answer would defeat the purpose. As an additional experiment, we plan to test whether prior interaction trajectories and the feedback that motivated a change improve prediction accuracy.

Task 1

Task outcome prediction

Given a task and a proposed modification, predict whether performance will improve, remain unchanged, or regress. In a binary pass/fail setting, two events are especially useful: a fix, where a failure becomes a pass, and a regression, where a pass becomes a failure. Predicting the magnitude of a performance change is a natural extension, but no magnitude-prediction results are reported here.

A proposed modification and a given task lead to a prediction of improvement, unchanged performance, or regression.
Figure 2. Predicting the direction of the performance change for one proposed modification and a given task.

Task 2

Candidate comparison

Given several modifications of the same parent, select a preferred candidate and rank the candidates by expected performance. Ties are allowed. Selecting any best-scoring candidate counts as a correct selection. For pairwise comparison, the prediction must distinguish which candidate scores higher or whether the two have equal observed scores.

Four candidates from the same parent harness are compared to choose a preferred candidate and a ranking that permits ties.
Figure 3. Comparing a fixed set of candidates. Choosing a best candidate and recovering the full ordering are distinct evaluation targets.

A small harness pilot

The pilot setup identifies two tasks from Terminal-Bench 2.1 [1]Terminal-Bench
A benchmark for agents in terminal environments.
: db-wal-recovery for fix prediction (fail → pass), and cancel-async-tasks for regression prediction (pass → fail). Candidate comparison considers performance across both tasks.

GPT-5.4 mini handles task execution, trajectory diagnosis, and candidate generation. We use GPT-5.4 mini, GPT-5.6 Luna, and GPT-6 Astra for outcome prediction and candidate comparison. These are the model labels used in the pilot figures.

Pilot setup: GPT-5.4 mini executes tasks and generates candidates; three models predict and compare. A two-round generation, evaluation and parent-selection sequence is marked times three.
Figure 4. The pilot setup diagram. Each round generates Structural, Guidance, Free, and Merge candidates, followed by evaluation and parent selection. The diagram depicts three two-round sequences; the comparison results below report five groups.

Results

These are preliminary observations from two tasks and a small set of harness modifications. The figures report point estimates, without uncertainty intervals or repeated-run summaries. We report the available denominators alongside the results and distinguish ranking unequal scores from recognizing ties.

Pilot result 1

Predicting fixes is not the same as anticipating regressions

On db-wal-recovery, GPT-6 Astra achieved 75% fix precision and 75% fix recall. GPT-5.4 mini and GPT-5.6 Luna each achieved 100% recall, with precision of 26.7% and 25%, respectively. The reported random-fix precision baseline is 20%.

Precision measures the fraction of predicted events that occurred; recall measures the fraction of observed events that were predicted. In this sample, mini and Luna identified all observed fixes but also predicted many fixes that did not occur. Astra’s fix predictions were more precise, at the cost of missing some observed fixes. Because the outcome-prediction summary does not give the number of observed fixes or predicted fixes, we retain the reported percentages rather than reconstruct unverified counts.

Fix precision is 26.7%, 25%, and 75% for mini, Luna, and Astra; fix recall is 100%, 100%, and 75%. All three have regression recall of 0 out of 6 and regression precision marked N/A.
Figure 5. Fix and regression prediction. The dashed line is the pilot’s reported 20% random-fix precision baseline. All three models miss the six observed regressions. Fix-event counts and the total outcome-prediction denominator are not supplied in this summary.

On cancel-async-tasks, all three models missed all 6 observed regressions: regression recall was 0/6 (0%) for each model. Regression precision is N/A, not zero, because no regressions were predicted. The pilot therefore exposes a specific failure mode: useful judgments about possible fixes did not translate into warnings about the observed harmful changes.

Pilot result 2

Selecting a best candidate

Across 5 groups of 4 candidates, Astra selected a best-scoring candidate in 4/5 groups (80%), compared with 3/5 (60%) for mini and 2/5 (40%) for Luna. The reported random-choice baseline is 50%. With ties, more than one candidate can count as best; selecting one of four candidates does not necessarily imply a 25% success baseline.

Astra has the highest observed selection accuracy in this sample. With only five groups, however, a single group changes accuracy by 20 percentage points. These results do not establish a general model ranking or a statistically reliable advantage.

Pilot result 3

Ordering unequal scores is easier here than recognizing ties

Each group of four candidates contributes six pairs, giving 30 pairs in total. Of these, 14 have unequal observed scores and 16 are tied. On the unequal-score subset, Astra correctly ordered 12/14 pairs (85.7%); mini and Luna each ordered 7/14 (50%) correctly, matching the reported 50% random-order baseline.

Candidate selection accuracy is 60%, 40%, and 80% for mini, Luna, and Astra across 5 groups. All-pair accuracy is 23.3%, 23.3%, and 46.7% across 30 pairs. Unequal-score accuracy is 50%, 50%, and 85.7% across 14 pairs.
Figure 6. Selecting and ranking candidates across two tasks. The three panels have different denominators: 5 groups, 30 pairs including ties, and 14 unequal-score pairs. Their baselines answer different questions.

The picture changes when ties are included. Across all 30 pairs, accuracy is 14/30 (46.7%) for Astra and 7/30 (23.3%) for both mini and Luna. Astra recognizes only 2/16 ties (12.5%). Subtracting the unequal-pair counts from the all-pair counts implies that mini and Luna recognize none of the 16 ties.

An always-tied prediction would be correct on 16/30 pairs, or 53.3%—higher than all three models’ full-pair accuracy. Astra’s stronger performance on unequal-score pairs therefore does not establish reliable ranking when ties matter. Also, these are ties in the observed two-task scores, not proof that the candidates are behaviorally equivalent more broadly.

Candidate comparison: counts and percentages
MeasureGPT-5.4 miniGPT-5.6 LunaGPT-6 Astra
Best-candidate selection3/5 · 60%2/5 · 40%4/5 · 80%
All pairs, including ties7/30 · 23.3%7/30 · 23.3%14/30 · 46.7%
Unequal-score pairs7/14 · 50%7/14 · 50%12/14 · 85.7%
Observed ties0/16 · 0%0/16 · 0%2/16 · 12.5%

Counts are reported directly where available, or recovered arithmetically from the displayed percentages and denominators. Mini and Luna’s tie counts are derived by subtraction. The 30 pairs share candidates within five groups and should not be treated as 30 independent experiments.

Original annotated pilot slides

The original slides below accompany the two result plots. They summarize the same observations, rather than additional experiments.

Original annotated task-outcome slide, showing precision and recall definitions and the six missed regressions.
Original task-outcome prediction slide.
Original annotated comparison slide, reporting five groups, thirty pairs, fourteen unequal-score pairs, and sixteen ties.
Original candidate-comparison slide.

What the pilot tells us—and what remains open

The most useful observation is the mismatch between different judgments. In this sample, Astra can distinguish many unequal-scoring candidates and select a best candidate in most groups, while all three models fail to anticipate the observed regressions. Recognizing ties is another weakness. A single aggregate score would obscure these differences.

The evidence is narrow: two tasks, five reported comparison groups, a shared candidate-generation model, and incomplete outcome-prediction counts. The summaries do not establish how stable the outcomes are across repeated executions, how predictions vary with prompts or context, or how the results transfer to other harnesses. They also do not establish that foresight improves an end-to-end self-improvement loop.

A larger evaluation should document the full candidate pool and scoring rule, separate generation from prediction, retain regressions and ties, and evaluate across more tasks and repeated runs. We also want to test whether providing the earlier trajectories and feedback changes the quality of predictions. These are next experiments, not conclusions of the present pilot.

Next: foresight through direct weight inspection

Agent artifacts are one modification space. Model weights are another. Inspired by Weight-Space Programming, which explores language models reading and editing neural-network weights [2]Takehi, Teoh, Zhang & Khattab · 2026
Direct inspection and editing of neural-network weights.
, we plan to study whether an agent can anticipate the effects of a concrete weight change.

Here, direct access to weights is central to the setting. The predictor inspects the original parameters and the proposed update, or the original and modified parameter tensors. It may use tools to examine relevant layers and numerical changes; the goal is not to force every parameter into a single prompt. It must predict the effect without running the modified model’s target evaluation.

θ′ = θ + Δθ     →     predict J(Mθ′) − J(Mθ).

The same two tasks apply: predict the direction of the outcome for one update, or compare multiple concrete updates to a shared base model. The exact models, modification mechanisms, and evaluation environments remain under exploration. No weight-inspection results are reported in this post.

References

  1. Terminal-Bench. Benchmark website. The pilot setup specifies version 2.1; the linked website may describe a newer release.
  2. Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. November 2, 2026. Blog post.

The editorial layout is inspired by the linked research blog. All experimental figures on this page are the author’s supplied pilot figures.

Cite this research note

Han Luo. “Agent Foresight: Can agents predict the effects of their own modifications?” November 2026. Preliminary research note.

BibTeX
@misc{luo2026agentforesight,
  author = {Luo, Han},
  title = {Agent Foresight: Can agents predict the effects of their own modifications?},
  year = {2026},
  month = nov,
  howpublished = {\url{https://lhannnn.github.io/blog/agent-foresight/}},
  note = {Preliminary research note}
}

← All posts