Agentic Abstention

// 01

Knowing when
to stop.

Agents need a stopping policy, not only a task policy.

Tool-using LLM agents often face goals that are ambiguous, underspecified, or impossible in the current environment. A reliable agent should recognize when continued interaction is unlikely to help and stop with an abstention.

Timing matters: an agent can stop as soon as infeasibility becomes clear, delay after unnecessary tool calls, or fail to abstain within its interaction budget.

A web shopping request has no matching product. Timely abstention stops after the first search, delayed abstention makes three unnecessary calls, and failure to abstain exhausts the ten-turn budget. — Abstention is a decision about timing. Correct eventual abstention can still waste actions after the blocking evidence is available. Open any figure to inspect it at full size.

01 / Answer

Complete the task when the evidence supports a valid response or action.

02 / Act

Search, browse, or inspect when another observation can reduce uncertainty.

03 / Abstain

Stop when the task is infeasible or essential information cannot be recovered.

// 02

Three settings.
One stopping problem.

More than 28,000 instructions across web, terminal, and interactive QA settings.

The benchmark combines solvable tasks with abstention-warranted variants. Some blockers are visible in the request; others only emerge as the agent interacts with the environment.

Web

WebShop

Ambiguous shopping requests and missing catalog targets that become clear through interaction.

Terminal

Terminal-Bench

Missing prerequisites, false premises, contradictions, and underspecified goals in terminal tasks.

Question answering

Interactive QA

AbstentionBench datasets adapted into a multi-turn setting where agents can answer, abstain, or search.

Terminal task construction and rewritten abstention instructions. — Constructing abstention-warranted tasks. Solvable tasks are rewritten into cases requiring abstention while preserving the interaction interface.

// 03

Often too late.
Sometimes never.

Most agents abstain too late, if they abstain at all.

Abstention recall improves with additional turns, but timely abstention remains low across settings. Agents often discover infeasibility only after unnecessary interactions.

Web26.7%

Timely recall for the strongest baseline, despite much higher eventual recall.

Terminal21.6%

Best timely recall under the tested GPT-5.4-mini configurations.

All settings<40%

Average timely recall for every evaluated model group in the reported comparisons.

Abstention recall curves across Web, Terminal, and QA settings. — More turns do not guarantee timely decisions. Recall increases with larger interaction budgets, while early abstention remains difficult.

Missing targets and prerequisites are particularly difficult because their absence must be established through interaction. Larger models mainly improve eventual recall, while the same model can behave differently under different agent scaffolds.

Explore results by task category, reasoning, and model size

Abstention recall by abstention category. — Missing targets and prerequisites create delayed-abstention failures.

Reasoning effects on AbsRec@1 and AbsRec@10. — Reasoning can improve early recall, but may reduce overall recall.

Cumulative over-abstention rate by turn in Web and Terminal scenarios. — Over-abstention increases with interaction, while reasoning helps mitigate it.

Timely and overall recall by model parameter count. — Scale mainly helps eventual recall, not necessarily timely recall.

// 04

Learning to stop
from experience.

CONVOLVE turns full interaction trajectories into reusable stopping rules.

It distills observed failures and timely abstention evidence into a playbook appended to the agent context. The model parameters and action interface stay fixed.

01 / Roll out

Interact

Run agents in WebShop using the original action interface.

02 / Reflect

Find the evidence

Analyze the full trajectory for evidence that made abstention warranted.

03 / Curate

Build a playbook

Compress repeated lessons into concise, reusable stopping rules.

04 / Reuse

Apply the lessons

Append the playbook to the context of future agent runs.

WebShop · CONVOLVE learns from 20 trajectories · Values in %
Model / playbook source	AbsRec@1 ↑	AbsRec@10 ↑	SPL ↑
Base model
Llama-3.3-8B	6.9	92.1	39.4
Llama-3.3-70B	26.7	83.2	55.3
Base model + in-context learning
Llama-3.3-8B	11.2	81.1	39.1
Llama-3.3-70B	55.1	97.0	77.2
Base model + CONVOLVE
Llama-3.3-8B / 8B	7.9	94.1	39.5
Llama-3.3-8B / 70B	12.9	94.1	39.7
Llama-3.3-70B / 8B	55.3	99.0	76.4
Llama-3.3-70B / 70B	57.4	100.0	78.9

Source: Table 1 of the paper. AbsRec measures abstention recall; SPL also accounts for interaction length. The value after “/” identifies the model used to produce the playbook. Bold marks the best value in each column.

Lessons learned by smaller models can transfer to larger models. The playbook makes the stopping evidence available in context without updating model weights.

// 05 / Related research

What if the agent and its challenges could improve together?

Agentic Abstention studies when agents should stop and learns stopping rules from trajectories. HERA extends this research direction by co-evolving the agent harness and executable environments, with matched feasible–infeasible task pairs.

// 06

Read, explore,
and cite.

Paper on arXiv Code & data NeurIPS 2026

BibTeX

@article{luo2026agentic,
  title={Agentic Abstention: Do Agents Know When to Stop Instead of Act?},
  author={Luo, Han and Wen, Bingbing and Wang, Lucy Lu},
  journal={arXiv preprint arXiv:2606.28733},
  year={2026}
}

Knowing when to stop.

Three settings. One stopping problem.

WebShop

Terminal-Bench

Interactive QA

Often too late. Sometimes never.

Learning to stop from experience.

Interact

Find the evidence

Build a playbook

Apply the lessons

From stopping rules to co-evolution.

HERA.

Read, explore, and cite.

Knowing when
to stop.

Three settings.
One stopping problem.

Often too late.
Sometimes never.

Learning to stop
from experience.

From stopping rules
to co-evolution.

Read, explore,
and cite.