Agent Foresight
Can agents predict the effects of their own modifications?
From proposing a change to anticipating its effects
An agent can propose a plausible improvement to its prompts, tools, or memory. It can explain why the change should help. The harder question is whether that explanation predicts what will actually happen. A modification aimed at fixing one failure may leave the failure untouched, or may disrupt behavior that previously worked.
In an iterative improvement loop, execution provides the answer: generate a candidate, evaluate it, and use the feedback to choose the next version. We are interested in the judgment that comes before that evaluation. Could an agent identify a useful change in advance? Could it recognize a regression? If several changes are available, could it select the most promising one?
We call this capability Agent Foresight: predicting the performance consequences of a concrete modification by inspecting the original system, the proposed change, and the task. Reliable foresight could help prioritize evaluations and identify risky changes. Whether it can do so is an empirical question.
The modification is fixed before the prediction is made. This separates two abilities that are easy to conflate: generating a good candidate and judging a given candidate. A model that performs well on one need not perform well on the other. In particular, our pilot compares several predictors on candidates generated by the same model; it does not test whether each predictor can generate better modifications itself.
What is being modified?
Our initial setting concerns agent artifacts: the persistent instructions and software around a model. These include prompts, skills, tool descriptions, long-term memory, middleware, tool implementations, and sub-agent organization. Together, these artifacts form the agent’s harness.
For a shared parent harness, we consider four candidate types. Structural candidates change middleware, tools, or sub-agents. Guidance candidates change prompts, skills, tool descriptions, or memory. A Free candidate is independently designed and may change either category or both. A Merge candidate combines the Structural and Guidance patches, resolving conflicts when necessary. Free and Merge therefore represent different ways of obtaining a candidate.
The broader research direction also includes changes to model weights. The predictor would inspect the weights and the actual proposed parameter changes. We describe this exploratory setting below; it is not part of the pilot results.
Two prediction tasks
Let A be the original agent, m a proposed modification, and J its performance on a specified task or task set. The quantity we would like to anticipate is:
The prediction must be made without access to the modified agent’s evaluation outcome. The intended protocol allows inspection of the change; running the target evaluation to obtain the answer would defeat the purpose. As an additional experiment, we plan to test whether prior interaction trajectories and the feedback that motivated a change improve prediction accuracy.
Task 1
Task outcome prediction
Given a task and a proposed modification, predict whether performance will improve, remain unchanged, or regress. In a binary pass/fail setting, two events are especially useful: a fix, where a failure becomes a pass, and a regression, where a pass becomes a failure. Predicting the magnitude of a performance change is a natural extension, but no magnitude-prediction results are reported here.
Task 2
Candidate comparison
Given several modifications of the same parent, select a preferred candidate and rank the candidates by expected performance. Ties are allowed. Selecting any best-scoring candidate counts as a correct selection. For pairwise comparison, the prediction must distinguish which candidate scores higher or whether the two have equal observed scores.
A small harness pilot
The pilot setup identifies two tasks from Terminal-Bench 2.1 [1]Terminal-Bench
A benchmark for agents in terminal environments.: db-wal-recovery for fix prediction (fail → pass), and cancel-async-tasks for regression prediction (pass → fail). Candidate comparison considers performance across both tasks.
GPT-5.4 mini handles task execution, trajectory diagnosis, and candidate generation. We use GPT-5.4 mini, GPT-5.6 Luna, and GPT-6 Astra for outcome prediction and candidate comparison. These are the model labels used in the pilot figures.
Results
These are preliminary observations from two tasks and a small set of harness modifications. The figures report point estimates, without uncertainty intervals or repeated-run summaries. We report the available denominators alongside the results and distinguish ranking unequal scores from recognizing ties.
Pilot result 1
Predicting fixes is not the same as anticipating regressions
On db-wal-recovery, GPT-6 Astra achieved 75% fix precision and 75% fix recall. GPT-5.4 mini and GPT-5.6 Luna each achieved 100% recall, with precision of 26.7% and 25%, respectively. The reported random-fix precision baseline is 20%.
Precision measures the fraction of predicted events that occurred; recall measures the fraction of observed events that were predicted. In this sample, mini and Luna identified all observed fixes but also predicted many fixes that did not occur. Astra’s fix predictions were more precise, at the cost of missing some observed fixes. Because the outcome-prediction summary does not give the number of observed fixes or predicted fixes, we retain the reported percentages rather than reconstruct unverified counts.
On cancel-async-tasks, all three models missed all 6 observed regressions: regression recall was 0/6 (0%) for each model. Regression precision is N/A, not zero, because no regressions were predicted. The pilot therefore exposes a specific failure mode: useful judgments about possible fixes did not translate into warnings about the observed harmful changes.
Pilot result 2
Selecting a best candidate
Across 5 groups of 4 candidates, Astra selected a best-scoring candidate in 4/5 groups (80%), compared with 3/5 (60%) for mini and 2/5 (40%) for Luna. The reported random-choice baseline is 50%. With ties, more than one candidate can count as best; selecting one of four candidates does not necessarily imply a 25% success baseline.
Astra has the highest observed selection accuracy in this sample. With only five groups, however, a single group changes accuracy by 20 percentage points. These results do not establish a general model ranking or a statistically reliable advantage.
Pilot result 3
Ordering unequal scores is easier here than recognizing ties
Each group of four candidates contributes six pairs, giving 30 pairs in total. Of these, 14 have unequal observed scores and 16 are tied. On the unequal-score subset, Astra correctly ordered 12/14 pairs (85.7%); mini and Luna each ordered 7/14 (50%) correctly, matching the reported 50% random-order baseline.
The picture changes when ties are included. Across all 30 pairs, accuracy is 14/30 (46.7%) for Astra and 7/30 (23.3%) for both mini and Luna. Astra recognizes only 2/16 ties (12.5%). Subtracting the unequal-pair counts from the all-pair counts implies that mini and Luna recognize none of the 16 ties.
An always-tied prediction would be correct on 16/30 pairs, or 53.3%—higher than all three models’ full-pair accuracy. Astra’s stronger performance on unequal-score pairs therefore does not establish reliable ranking when ties matter. Also, these are ties in the observed two-task scores, not proof that the candidates are behaviorally equivalent more broadly.
| Measure | GPT-5.4 mini | GPT-5.6 Luna | GPT-6 Astra |
|---|---|---|---|
| Best-candidate selection | 3/5 · 60% | 2/5 · 40% | 4/5 · 80% |
| All pairs, including ties | 7/30 · 23.3% | 7/30 · 23.3% | 14/30 · 46.7% |
| Unequal-score pairs | 7/14 · 50% | 7/14 · 50% | 12/14 · 85.7% |
| Observed ties | 0/16 · 0% | 0/16 · 0% | 2/16 · 12.5% |
Counts are reported directly where available, or recovered arithmetically from the displayed percentages and denominators. Mini and Luna’s tie counts are derived by subtraction. The 30 pairs share candidates within five groups and should not be treated as 30 independent experiments.
What the pilot tells us—and what remains open
The most useful observation is the mismatch between different judgments. In this sample, Astra can distinguish many unequal-scoring candidates and select a best candidate in most groups, while all three models fail to anticipate the observed regressions. Recognizing ties is another weakness. A single aggregate score would obscure these differences.
The evidence is narrow: two tasks, five reported comparison groups, a shared candidate-generation model, and incomplete outcome-prediction counts. The summaries do not establish how stable the outcomes are across repeated executions, how predictions vary with prompts or context, or how the results transfer to other harnesses. They also do not establish that foresight improves an end-to-end self-improvement loop.
A larger evaluation should document the full candidate pool and scoring rule, separate generation from prediction, retain regressions and ties, and evaluate across more tasks and repeated runs. We also want to test whether providing the earlier trajectories and feedback changes the quality of predictions. These are next experiments, not conclusions of the present pilot.
Next: foresight through direct weight inspection
Agent artifacts are one modification space. Model weights are another. Inspired by Weight-Space Programming, which explores language models reading and editing neural-network weights [2]Takehi, Teoh, Zhang & Khattab · 2026
Direct inspection and editing of neural-network weights., we plan to study whether an agent can anticipate the effects of a concrete weight change.
Here, direct access to weights is central to the setting. The predictor inspects the original parameters and the proposed update, or the original and modified parameter tensors. It may use tools to examine relevant layers and numerical changes; the goal is not to force every parameter into a single prompt. It must predict the effect without running the modified model’s target evaluation.
The same two tasks apply: predict the direction of the outcome for one update, or compare multiple concrete updates to a shared base model. The exact models, modification mechanisms, and evaluation environments remain under exploration. No weight-inspection results are reported in this post.
References
- Terminal-Bench. Benchmark website. The pilot setup specifies version 2.1; the linked website may describe a newer release.
- Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. November 2, 2026. Blog post.
The editorial layout is inspired by the linked research blog. All experimental figures on this page are the author’s supplied pilot figures.
Cite this research note
Han Luo. “Agent Foresight: Can agents predict the effects of their own modifications?” November 2026. Preliminary research note.
BibTeX
@misc{luo2026agentforesight,
author = {Luo, Han},
title = {Agent Foresight: Can agents predict the effects of their own modifications?},
year = {2026},
month = nov,
howpublished = {\url{https://lhannnn.github.io/blog/agent-foresight/}},
note = {Preliminary research note}
}

