acceptodds
Under review as a conference paper at ICLR 2027

RIGOR: Dual Governance for Research Intent Integrity in Long-Horizon Autonomous Research

Abstract

Long-horizon autonomous research requires agents to maintain project-level scientific intent while repeatedly designing, executing, and interpreting experiments. Persistent single-agent loops often mix long-term research objectives with transient implementation details and debugging histories, creating substantial context accumulation. Decomposing the workflow into a persistent Research Agent and transient Execution Agents can reduce this burden, but it also introduces new handoff boundaries where execution fidelity may be compromised and unsupported evidence may enter the scientific state. We formalize Research Intent Integrity (RII) as the requirement that material deviations become part of the authoritative research state only when they are authorized, justified, and provenance-preserving. Based on this principle, we introduce a dual-governance architecture consisting of an Intent-Grounded Research Loop (IGRL) for governing evidence-to-state updates and research termination, and Scientific Execution Governance (SEG) for binding experiment briefs with candidate implementations, formal executions, and sealed evidence receipts. We instantiate these mechanisms in RIGOR (Research Intent Governance for Open-ended Research), an end-to-end role-decomposed AutoResearch system for long-horizon, evidence-driven discovery. Execution-level evaluations show that SEG improves experimental fidelity and prevents unsupported deviations from entering canonical evidence with modest computational overhead. Long-horizon studies further demonstrate that role decomposition reduces persistent context accumulation, while integrating IGRL and SEG preserves research intent throughout iterative research updates compared with ungoverned architectures. Evaluations on public AutoResearch benchmarks show that RIGOR maintains competitive end-to-end performance while providing stronger control over scientific state transitions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous research agents, long-horizon LLM agents, multi-agent systems, agent governance, research intent integrity

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_M4gyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RIGOR, a framework for preserving research intent in long-horizon autonomous research. It separates a persistent Research Agent from transient Execution Agents and introduces two governance modules: SEG checks execution fidelity and provenance, while IGRL governs planning, evidence admission, scientific-state updates, and stopping decisions. The paper defines Research Intent Integrity and evaluates the framework on fixed-brief execution episodes, multi-cycle research tasks, and a subset of FML-Bench. The reported results show improved experimental fidelity and fewer realized integrity violations, with additional orchestration costs.

Critical strengths
  1. The paper addresses a relevant problem: autonomous research agents may gradually deviate from the intended scientific objectives.
  2. The distinction between local execution errors and errors that propagate into authoritative research records is useful.
Critical weaknesses
  1. Limited methodological novelty and unclear incremental contribution. RIGOR combines role decomposition, structured experiment specifications, independent review, controlled execution, provenance tracking, and evidence-gated state updates. These are established agent-system design patterns, and the paper does not clearly identify what technically novel mechanism emerges from their combination. The RII formulation provides useful terminology, but primarily defines and categorizes failures rather than establishing new guarantees or an algorithmic contribution. The distinction from existing specification-driven and evidence-gated workflows therefore remains insufficiently developed.

  2. Insufficient technical and empirical depth to establish a substantial contribution. The paper describes a relatively elaborate architecture, but does not examine its central mechanisms in sufficient depth. For example, the practical reliability of authorization checks, semantic evidence admission, and stopping review is not quantitatively characterized at the component level. Canonical-123 and Long-50 are useful adaptations of public research resources, but their additional evaluation value needs stronger validation. In particular, the paper describes construct-validation cases without reporting sufficient quantitative results or human-expert agreement to establish that the proposed integrity metrics reliably capture scientific failures. Consequently, neither the methodological contribution nor the evaluation contribution is sufficiently substantiated.

  3. Missing strong, lightweight, and compute-matched comparisons. The included baselines cover several architectural variants, and Compact Generic Review is a relevant comparator. However, the experiments do not establish whether the full harness is necessary compared with simpler alternatives: a single agent equipped with a reusable research-protocol skill, an explicit experiment checklist, persistent structured records, or an equally budgeted independent-review loop. Such baselines could implement many of the same scientific safeguards with substantially less orchestration. Additionally, the FML comparison uses published baseline results rather than a unified rerun, leaving backbone and inference-budget comparability insufficiently established.

  4. Insufficient component ablations and cost-effectiveness analysis. The paper does not provide a systematic ablation separating SEG from IGRL or isolating mechanical gating, semantic review, provenance binding, blind evidence review, and stopping review. The current architectural comparisons therefore cannot identify which mechanisms account for the gains. Although calls and token usage are reported, the long-horizon comparison is not compute-matched: RIGOR uses approximately 171.63 calls per trajectory, versus 7.82 for Single Agent and 25.60 for Compact RA–EA. Equal cycle limits do not imply equal computational budgets. Without matched-budget evaluations and performance–cost trade-offs, the reported improvements cannot be cleanly attributed to the proposed governance design rather than additional computation.

  5. Missing Reproducibility Statement and Ethics Statement.

Questions for authors

See Weeknesses.

7 comments

W1. Methodological novelty and incremental contribution.

Prior autonomous-research systems have devoted substantial attention to completing research workflows, improving experimental performance, and producing high-quality research artifacts. However, preserving the authorized research intent across successive planning, execution, and evidence-integration steps has received less explicit attention. Successful execution or a convincing final artifact does not establish whether the research has retained the required comparisons, interventions, and evidential standards. RIGOR addresses this gap by making research-intent preservation an explicit object of both governance and evaluation.

The example in the revised Figure 1(c) illustrates this distinction. Suppose the authorized task requires comparing X against Y, but the Research Agent replaces Y with Z without authorization or justification. An Execution Agent that faithfully implements this Brief can achieve EFR = 1, while the substitution constitutes a realized integrity violation once it enters formal execution. A check concerned only with implementation–Brief consistency would accept the execution and miss the upstream planning deviation.

RIGOR operationalizes this distinction through two complementary governance mechanisms. IGRL reviews planning, evidence interpretation, scientific-state updates, and stopping decisions against the research objective. SEG checks execution against the Brief and preserves the links between candidate implementations, executions, and evidence. Candidate binding identifies the implementation associated with an execution, execution receipts record its factual outcomes, and evidence-admission rules govern how those outcomes support subsequent research updates.

The RII formulation and associated metrics make the corresponding failure modes measurable: where a deviation originates, whether it remains local, and whether it enters authoritative research records. The contribution therefore connects a specified research problem to an executable governance protocol and trajectory-level evaluation, beyond adding review agents or scoring final artifacts.

W2. Technical depth and validation of the evaluation framework.

We have supplemented the evaluation with controlled plan-review experiments and a repair audit.

As shown in Table 22, the controlled experiment compares the RIGOR Plan Reviewer with a Generic Reviewer using 50 legitimate Briefs and 50 Briefs containing constructed deviations. The initial cases were manually checked against the original Tasks. Each case has three independent trajectories per reviewer, with the same model, inputs, and review protocol. Final plan validity is assessed separately by independent Judges.

MeasureGeneric ReviewerRIGOR Plan Reviewer
Correct initial rejection of constructed deviations148/150 (98.67%)150/150 (100.00%)
Initial false rejection of legitimate Briefs ↓18/150 (12.00%)8/150 (5.33%)
Final Task-consistent Briefs: constructed-deviation condition147/150 (98.00%)150/150 (100.00%)
Final reviewer approval: constructed-deviation condition150/150 (100.00%)146/150 (97.33%)

The Generic Reviewer already detects most constructed deviations. RIGOR improves discrimination by reducing unnecessary rejection of legitimate plans while producing Task-consistent final Briefs in all constructed-deviation trajectories. The difference between approval and independently assessed validity is also informative: the Generic Reviewer approves all final Briefs, although three remain Task-inconsistent. These results provide quantitative evidence about both initial review decisions and subsequent plan correction.

As shown in Table 23, we additionally audited 2,387 EA repair records and 64 RA plan revisions from one trajectory for each of the 50 tasks. Repairs resolve 90.77% of verified original issues on the EA side and all 64 on the RA planning side. The audit separately records newly introduced problems and their downstream status. For example, 144 EA repair records introduce new scientific deviations, of which 135 (93.75%) are blocked.

Canonical-123 and Long-50 serve complementary evaluation purposes. Canonical-123 freezes the Brief and initial conditions to isolate execution fidelity from changes in research planning. Long-50 preserves the evolving research process, allowing deviations to be traced into later experiments, findings, and stopping decisions. Their additional value lies in exposing these process-level failures alongside task performance. The controlled cases provide known initial task-consistency conditions, while the trajectory audits examine how the mechanisms behave during actual research runs.

W3. Stronger, lightweight, and resource-aware comparisons.

The fixed-Brief experiment provides a direct comparison with an independent-review loop that already includes inspection and iterative feedback. SEG improves EFR from 66.67% for Compact Generic Review to 89.70%, while using fewer average agent calls, 3.69 versus 4.27, and fewer input tokens, 801.49k versus 1,034.20k. This comparison supports the value of structured execution governance beyond simply increasing the number of review calls (Table 4).

The long-horizon structural baselines examine shared-context staging, role decomposition, and structured communication. We have also expanded the existing-system comparison to include Agent Laboratory, AutoResearch, TAS v2, AIRA-MCTS, and AIDE (as shown in Table 5). AutoResearch achieves relatively high EFR (80.89%) but still exhibits substantial trajectory-level RII-V (66.67%), showing that faithful execution alone does not ensure research-intent preservation. Agent Laboratory separates planning and execution across agents, yet records an EFR of 11.60% and RII-V of 95.33%, further illustrating that role decomposition alone is insufficient to ensure faithful execution and preserve research intent.

A reusable skill or checklist can express research requirements, and structured records can help retain them. RIGOR specifies how those requirements are checked against implementations and execution evidence, and how the resulting evidence informs subsequent research. The contribution concerns these operational checks and evidence relationships rather than the number of agents alone. The component ablations and cumulative token-budget analysis below examine their benefits and resource implications.

For FML, the baseline values are explicitly identified as published results under the benchmark’s task protocol. We use this comparison as supplementary evidence of end-to-end task performance. The controlled comparisons supporting the governance claims are the fixed-Brief and long-horizon experiments, which use a common research model and specified evaluation procedures.

W4. Component ablations and cost-effectiveness analysis.

We have added the following ablation study to the main text (Table 6). It evaluates the two governance branches separately and removes each of the four reviewers individually from full RIGOR. Rates are percentages.

VariantCompletion ↑RII-V ↓EFR ↑RII-cycle ↓Avg CyclesAvg Calls
Full RIGOR69.3339.3385.388.3114.57167.12
SEG-only60.0050.0071.7620.197.2460.26
IGRL-only56.0066.0073.1518.0616.4094.88
w/o CAR58.0078.0069.0320.6718.00110.36
w/o Plan Review60.0046.0082.7314.2915.24147.42
w/o Blind Review54.0058.0073.6723.2217.08158.12
w/o Stop Review58.0046.0083.1212.3815.44180.06

Neither branch alone matches the complete system. Removing CAR produces the largest increase in trajectory-level violations and the largest reduction in execution fidelity. Removing Blind Review produces the lowest completion and highest RII-cycle rate among the component-removal variants. Removing Plan Review or Stop Review also reduces completion and increases realized violations.

We have also added Figure 5 to evaluate outcomes at common cumulative research-token thresholds from 0.5M to 30M. Completion counts confirmed completed trajectories whose total expenditure falls within the threshold. Fidelity and integrity are assessed over the eligible trajectory prefixes within that threshold.

RIGOR achieves the highest completion at the upper end of the evaluated budget range while maintaining strong execution fidelity and substantially lower realized violations. These curves provide a performance–token comparison beyond the shared cycle limit, showing where governance expenditure translates into improved research outcomes.

Revised Figure 4 and Appendix Table 21 further report planning-context growth, recorded compactions, component-level input consumption, and cached, uncached, output, and reasoning-token costs. RIGOR uses more input tokens than the ungoverned RA–EA baselines, but fewer than the persistent single-agent methods. Most of its input consumption is concentrated in EA execution and CAR, while its persistent RA planning context grows gradually with few recorded compactions.

W5. Reproducibility Statement and Ethics Statement.

We have added dedicated Reproducibility and Ethics Statements to the revised manuscript. We will publicly release the code and constructed task collections. The Ethics Statement addresses evidence reliability and traceability in autonomous research, human verification of consequential scientific claims, and source attribution and applicable access conditions for the released task resources.

Follow-up to Authors' Response

I thank the authors for their response. I have read the added ablation study and the two statements, and W5 is resolved. However, the following core concerns have not been substantively addressed:

1. The metrics lack human validation. The paper's main conclusions rely on LLM-as-a-Judge. The construct validation in E.6 provides no quantitative results and no agreement with human experts. The newly added Table 22 is also assessed by LLM Judges, and the Generic Reviewer already detects 98.67% of the constructed deviations, so the set offers limited discriminative power.

2. Lightweight baselines are still missing. The comparisons requested in my original review, such as a single agent with a research-protocol skill or checklist and an equally budgeted independent-review loop in the long-horizon setting, have not been run. The response only argues the point conceptually. The FML comparison still relies on published results, which is inconsistent with the claim of "under matched conditions" in the Introduction.

3. Compute matching remains unresolved. Figure 5 is obtained by truncating existing trajectories at token thresholds, rather than by running each method under a prespecified budget cap. Even so, RIGOR's completion rate is lower than that of the baselines at low and medium budgets, and RIGOR leads only at the high end. According to Table 21, RIGOR uses about 5× the total tokens of Raw RA–EA, and its median per-trajectory cost (34.29M) exceeds the largest evaluated budget (30M). These results do not establish that the gains come from the governance design rather than from additional computation. A budget-matched comparison is still needed.

Overall, I maintain my original rating (Rating: 1).

Reviewer_Zr7COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RIGOR, a dual-governance framework for preserving research intent in long-horizon autonomous scientific research. The framework separates a persistent research agent, responsible for scientific planning and interpretation, from transient execution agents, responsible for individual experiments. RIGOR introduces two complementary mechanisms: scientific execution governance, which enforces experimental fidelity and provenance preservation, and the intent-grounded research loop, which governs evidence interpretation, scientific-state updates, and termination decisions. The authors also formalize research intent integrity and introduce metrics for measuring material deviations from authorized research objectives. Experiments demonstrate improvements in experimental fidelity, reduced integrity violations, and competitive end-to-end task performance.

Critical strengths
  1. Preserving scientific intent across long-horizon agent workflows is a meaningful challenge. The paper identifies two concrete failure points, brief-to-evidence and evidence-to-state transitions, and provides a coherent framework for addressing them.
  2. The separation between research planning, experimental execution, and independent evidence review is well motivated. The use of structured experiment briefs, execution validation, and provenance-preserving receipts offers a practical approach to improving the reliability of autonomous research.
  3. The authors evaluate the system at multiple levels, including fixed-brief execution episodes, long-horizon research tasks, and executable benchmark tasks. The results demonstrate promising improvements in experimental fidelity and research-intent preservation.
Critical weaknesses
  1. The paper introduces two major governance mechanisms, SEG and IGRL, but does not adequately isolate their individual contributions in the long-horizon setting. Comparisons against existing architectural baselines do not establish whether the improvements primarily arise from execution governance, evidence review, stopping control, or their combination.
  2. In Table 5, RIGOR requires an average of 171.63 model calls per trajectory, compared with 7.82–37.62 calls for the baselines. RIGOR also executes more research cycles on average. Although it achieves higher task completion, these differences make it difficult to determine how much of the improvement comes from the proposed governance mechanisms rather than additional computation.
  3. Experimental fidelity rate and RII violations are largely assessed through an LLM-as-a-Judge protocol. Although the authors employ three independent judging instances and describe construct-validation cases, quantitative human-validation results and judge agreement statistics are not provided. Since these metrics constitute the central evidence supporting the paper's contributions, stronger validation of the evaluation protocol is necessary.
  4. The FML-Balanced-9 comparison uses previously published baseline results rather than rerunning all methods under fully matched conditions. This limits the extent to which the performance differences can be attributed to RIGOR.
Questions for authors
  1. Can the authors provide ablations evaluating SEG-only, IGRL-only, and the complete RIGOR framework on the same long-horizon tasks? Which components contribute most to experimental fidelity, integrity preservation, and task completion?
  2. How does RIGOR compare with the baselines under matched inference budgets or total token consumption? Can the authors provide performance-versus-cost curves to demonstrate that the governance mechanisms improve efficiency rather than simply benefiting from additional model calls?
  3. Has the LLM-as-a-Judge protocol been validated against independently annotated human judgments? What are the agreement rates among the three judges, and how sensitive are the reported RII metrics to the choice of evaluation model?
  4. Were the model backbones, execution environments, and computational budgets fully comparable between RIGOR and the published FML-Balanced-9 baselines?
  5. How does RIGOR handle legitimate scientific discoveries that require substantial deviations from the original experiment brief? Can the authors provide an example demonstrating how the system distinguishes productive, evidence-driven changes from unauthorized research-intent drift?
6 comments

W1 / Q1 — Complementary Contributions of SEG, IGRL, and Their Components

We have added a long-horizon ablation study evaluating SEG-only, IGRL-only, and full RIGOR on the same research tasks. We additionally remove CAR, Plan Review, Blind Review, and Stop Review individually to examine their respective contributions. The results are included in the revised main text as Table 6 and reproduced below. Rates are percentages.

VariantCompletion ↑RII-V ↓EFR ↑RII-cycle ↓Avg CyclesAvg Calls
Full RIGOR69.3339.3385.388.3114.57167.12
SEG-only60.0050.0071.7620.197.2460.26
IGRL-only56.0066.0073.1518.0616.4094.88
w/o CAR58.0078.0069.0320.6718.00110.36
w/o Plan Review60.0046.0082.7314.2915.24147.42
w/o Blind Review54.0058.0073.6723.2217.08158.12
w/o Stop Review58.0046.0083.1212.3815.44180.06

Neither governance branch alone matches the complete system. Full RIGOR achieves the highest task completion and execution fidelity and the lowest trajectory-level and cycle-level realized violations.

The reviewer-level ablations reveal distinct contributions. Removing CAR produces the largest reduction in EFR and the largest increase in trajectory-level RII-V, highlighting its role in checking candidate implementations against the Brief. Removing Blind Review produces the lowest completion and highest RII-cycle rate among the component-removal variants, indicating the importance of independently assessing execution evidence before it informs subsequent research. Plan Review and Stop Review also improve completion and integrity.

W2 / Q2 — Research Performance under Cumulative Token Budgets

We have added Figure 5 to compare research outcomes at common cumulative research-token thresholds ranging from 0.5M to 30M. This analysis evaluates the performance–resource relationship beyond the shared research-cycle limit.

At each threshold, completion counts trajectories whose final task completion is confirmed and whose total research-token expenditure falls within the budget. EFR and RII-cycle are calculated over completed, assessable cycles within the budget, while trajectory-level RII-V records whether a realized violation has occurred in the evaluated prefix. The analysis therefore accounts for the amount of computation used rather than treating cycles or calls as equal-sized units of work.

RIGOR achieves the highest completion at the upper end of the evaluated budget range. As the budget increases, it also combines high execution fidelity with substantially lower cycle-level and trajectory-level integrity violations. Thus, the benefit is visible not only in eventual task completion but also in the quality and integrity of research conducted within comparable cumulative token thresholds.

We have also revised Figure 4 and added the detailed long-horizon cost breakdown in Appendix Table 21. The accounting includes underlying LLM requests across research agents and internal reviewers, including repair attempts, tool-interaction loops, and recorded compactions; post-run evaluation-Judge costs are excluded.

Figure 4 shows that RIGOR consumes more input tokens than the ungoverned RA–EA baselines but fewer than the persistent Single Agent and Staged Single methods. Most of its input consumption comes from EA execution and CAR. Meanwhile, its persistent RA planning context grows gradually with few recorded compactions, illustrating how execution-side computation is localized rather than fully accumulated in the persistent research context.

W3 / Q3 — Inter-Judge Agreement and Evaluation Reliability

We performed manual spot-checks of selected cases and complemented these checks with quantitative analyses of inter-Judge agreement and sensitivity to evaluator choice.

For the sensitivity analysis, we evaluated the same set of 50 recorded research trajectories, one per task, using GPT-5.6-Sol and GPT-5.6-Terra. The primary results are summarized below; all values are percentages.

MetricGPT-5.6-SolGPT-5.6-Terra
Task completion62.0072.00
RII-V42.0014.00
EFR82.5690.74
RII-cycle11.021.88

Terra assigns higher completion and fidelity scores while identifying fewer integrity violations on the same trajectories, indicating a more lenient assessment pattern. This comparison makes the sensitivity to evaluator choice explicit. Our main results consistently use GPT-5.6-Sol under the same evaluation protocol across methods.

We have added quantitative inter-Judge agreement statistics for the 150 RIGOR trajectories used in the paper’s 50-task evaluation, covering 2,185 research cycles. This complements the manually checked initial cases in the controlled plan-review experiment.

The table reports agreement among the three Judge instances before aggregation. For each metric, all agreement statistics are calculated on the same subset of samples for which all three Judges provide definite binary judgments.

MetricSamples with three definite judgmentsUnanimous agreementMean pairwise agreementFleiss’ κ
EFR1,926 cycles86.55% (1,667)91.03%0.650
RII-cycle2,184 cycles83.15% (1,816)88.77%0.396
Completion150 trajectories75.33% (113)83.56%0.644
RII-V150 trajectories58.00% (87)72.00%0.437

Fleiss’ κ accounts for chance agreement. Reporting it alongside raw agreement and the sample counts makes the consistency of each judgment type explicit. Mean pairwise agreement is 91.03% for execution fidelity and 88.77% for cycle-level realized violations; the trajectory-level measures are reported separately rather than summarized by a single overall agreement score.

W4 / Q4 — FML-Balanced-9: Comparison Protocol and Scope

The FML-Balanced-9 comparison uses published baseline results under the benchmark’s task and scoring protocols. We followed the official FML configuration as closely as possible, including NVIDIA A100-80GB GPUs, a search budget of 100 optimization steps per trajectory, and three independent trajectories per method and task, with the final test result obtained from the candidate selected based on validation performance.

These conditions align the task-level experimental and scoring procedures. The baseline results are nevertheless published reference results rather than unified reruns with verified identical model backbones, execution environments, and inference expenditure. We therefore present FML as a benchmark-reference comparison of end-to-end task performance.

The controlled evidence for the governance mechanisms comes from Canonical-123 and Long-50, where the compared research systems use a common model and specified evaluation procedures. The new component ablations and cumulative token-budget curves further examine which mechanisms contribute and how research quality varies with computational expenditure.

We will make this distinction explicit in the FML discussion: its role is to demonstrate end-to-end research capability, while the controlled execution and long-horizon experiments support the analysis of governance effects.

Q5 — Distinguishing Evidence-Driven Adaptation from Research-Intent Drift

RIGOR preserves the scientific commitments of the authorized Task, while allowing hypotheses and experimental routes to evolve with evidence. A substantial change from an earlier Brief is therefore not automatically research-intent drift. The relevant questions are what changes, what evidence motivates the change, and whether it remains within the authorized research scope.

We distinguish three situations. Implementation choices within the authorized scope can be made by the EA while preserving required comparisons and evidence standards. When new evidence motivates a different hypothesis, intervention, or research route, the EA reports the finding and the RA incorporates it into subsequent planning through a new Brief. Changes to requirements fixed by the original Task—such as the required scientific object, key comparison, or success criterion—require task-level authorization; rewriting a Brief alone does not provide that authorization.

The Heavy-tail Optimization trajectory provides a concrete example. The Task asks the system to understand and improve optimization under heavy-tailed stochastic gradients using lightweight, condition-matched Fashion-MNIST experiments. It explicitly permits evidence-driven changes to hypotheses, diagnostics, controls, and optimizers, while requiring experimentally supported improvements in robustness or quality–compute trade-offs.

StageEvidence and research decision
Cycles 1–3: gradient clippingGlobal clipping provides no benefit, while per-example clipping harms predictive quality. These outcomes are retained as negative findings, motivating a change of route.
Cycles 4–5: state interventions and LookaheadAlternative explanations are investigated, but the results do not meet the improvement requirements. Local diagnostic changes are not treated as task completion.
Cycle 14: RAdamUpdate-tail behavior improves, but endpoint accuracy deteriorates, so the route is rejected.
Cycle 15: QHAdam, seeds 0–2Under matched Adam controls and predefined quality constraints, mean worst validation-loss excursion decreases by 21.94%. The result is accepted as a candidate improvement for the tested workload.
Cycle 16: confirmation, seeds 3–5The settings are retained and tested with fresh seeds. The same metric decreases from 0.05079 to 0.02165, approximately 57.4%, with higher endpoint accuracy for each matched seed and no additional instability or non-finite events.

The research route changes substantially—from clipping to alternative optimizer mechanisms—but the required controls, quality criteria, and evidential standards remain intact. The final conclusion is a confirmed, workload-specific robustness improvement over matched Adam controls, without extending it to universal optimizer superiority or a proven causal mechanism.

The same trajectory also contains a contrasting case. In cycle 9, an EA revision substitutes a validation-window proxy for the endpoint-quality criterion specified in the Brief. All three repair-audit assessments identify this as a newly introduced scientific deviation, and subsequent CAR review blocks it. The distinction is therefore not a fixed rule such as allowing optimizer changes but forbidding metric changes: the productive changes are explicitly planned within the Task’s authorization and subjected to appropriate validation, whereas the rejected revision changes a required criterion without authorization.

All three Judges classify the task as completed. Across the 16-cycle trajectory, EFR is 93.75% (15/16), RII-cycle is 0, and no trajectory-level realized RII violation is recorded. This example demonstrates both evidence-driven research adaptation and containment of an unauthorized local deviation.

Reviewer_Wqw1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how autonomous research systems preserve scientific intent across repeated planning, execution, and interpretation. RIGOR separates a persistent Research Agent from transient Execution Agents and introduces two governance components: IGRL reviews plans, evidence admission, scientific-state updates, and stopping decisions; SEG connects experiment briefs to candidate implementations, controlled executions, and sealed evidence receipts.

The evaluation combines fixed-brief execution experiments, long-horizon research trajectories, and public research benchmarks. The authors report improved experimental fidelity and fewer realized research-intent violations, while examining the context and resource costs of role decomposition and governance.

Critical strengths

1. The architecture addresses a consequential failure mode with concrete mechanisms.

The distinction between executing an experiment faithfully and admitting its results into the scientific state is useful. Candidate binding, sealed execution receipts, and explicit evidence-admission rules provide concrete ways to track this boundary. The supplied code implements substantive parts of these mechanisms, which gives the proposal practical value beyond a high-level recommendation to add more review agents.

2. The fixed-brief evaluation includes a meaningful comparison against generic review.

In Table 4, SEG improves Experimental Fidelity Rate from 66.67% for Compact Generic Review to 89.70%, while using fewer model calls on average, 3.69 versus 4.27. I find this comparison particularly informative because it tests structured execution governance against an alternative that already includes iterative feedback. Freezing the Brief also helps separate execution fidelity from changes in the research plan.

Critical weaknesses

1. Some enforcement claims do not match the supplied implementation.

Section 3.2.2 states that MGE rejects unauthorized workspace modifications. However, the supplied mechanical gate checks whether declared edits match observed edits without checking those paths against the allowed and forbidden scope. A minimal diagnostic returns PASS for a declared change to an explicitly forbidden file. This identifies a gap in MGE; the later semantic reviewer could still reject the candidate.

There is also a discrepancy in plan governance. Appendix B.4 requires alignment issues to be resolved before dispatch, but the supplied controller permits dispatch after the third consultation with an unresolved reviewer objection, recorded as ADVISORY_DISAGREEMENT. The authors need to distinguish mandatory enforcement from advisory review and identify which implementation generated the reported experiments. The supplied source excerpt does not establish that the historical runs used these exact rules.

2. The long-horizon benefits are not sufficiently separated from resource expenditure.

Table 5 reports completion of 70% for RIGOR versus 62% for Single Agent, alongside 171.63 versus 7.82 model calls on average. A common limit of 30 research cycles does not provide a matched resource budget when each cycle contains substantially different amounts of work. Since calls also vary in size, a comparison at matched token or monetary budgets would be more informative. The current results leave uncertainty about how much of the long-horizon improvement comes from governance rather than additional computation.

I would raise my score if the authors reconcile the enforcement claims with the experimental implementation and show that the long-horizon benefits persist under a fair resource comparison.

Questions for authors
  1. Which code revision and configurations produced the reported tables? Where are actual workspace edits checked against the authorized scope, and does that check occur outside the supplied MGE implementation?

  2. Did the evaluated system permit dispatch after the third plan-review consultation when objections remained unresolved? If so, how frequently did this occur, and which governance decisions were mandatory rather than advisory?

  3. How do completion and research-intent violations compare under matched token or monetary budgets? A quality–cost comparison would materially affect my assessment of the long-horizon contribution.

4 comments

Clarification of enforcement and implementation (W1 and Q1).

Thank you for examining the implementation closely. We have revised Section 3.2.2 and Appendix B.4 to describe the respective mechanisms more precisely.

In Section 3.2.2, the MGE description now reads:

“MGE first verifies structural consistency, including agreement between declared and observed workspace modifications, before execution.”

MGE performs structural and candidate-identity checks, while CAR subsequently assesses the candidate against the Brief through semantic review, including its scope restrictions. Thus, a declared modification can satisfy the mechanical consistency check and still require review for compliance with the Brief.

Appendix B.4 now explicitly describes the bounded plan-review protocol: plans undergo up to three consultations, after which persistent disagreement is recorded as ADVISORY_DISAGREEMENT and dispatch is permitted. The reported long-horizon experiments used this policy. The audit of the 150 trajectories underlying the updated Table 5, reported below, documents how frequently this occurred in those runs.

Performance under cumulative token budgets (W2 and Q3).

We have added Figure 5 to compare research outcomes across common cumulative token-budget thresholds from 0.5M to 30M. This analysis directly examines the relationship between resource expenditure and research quality beyond the shared research-cycle limit.

Figure 5 reports four complementary outcomes. Completion counts trajectories whose final task completion is confirmed and whose total research-token expenditure falls within the budget. EFR and RII-cycle are evaluated over completed, assessable cycles within the budget, while trajectory-level RII-V records whether a realized violation has occurred in the evaluated prefix.

RIGOR achieves the highest completion rate at the upper end of the evaluated budget range. As the budget increases, it also combines high execution fidelity with substantially lower cycle-level and trajectory-level integrity violations. The advantage therefore extends beyond completing more tasks: at comparable cumulative token thresholds, RIGOR better preserves experimental fidelity and limits the propagation of research-intent deviations.

These curves provide a quality–resource comparison across the evaluated budget range, showing where the additional governance computation translates into improved research outcomes. We have included both the figure and its interpretation in the revised manuscript.

Bounded plan review and observed dispatch frequency (Q2).

The three-consultation limit is designed to prevent persistent disagreement between the Research Agent and Plan Reviewer from indefinitely blocking research progress. Plan review is required, but reviewer agreement is not an unconditional prerequisite for dispatch after the consultation limit. Any remaining objection is explicitly recorded, and the execution candidate subsequently proceeds through SEG.

We examined the 150 RIGOR trajectories underlying the updated 50-task results, comprising 2,185 reviewed plans:

Review outcomeNumber of plansPercentage of all reviewed plans
One consultation2,09095.65%
Two consultations904.12%
Reached the third consultation50.23%
Approved in the third consultation40.18%
Dispatched with unresolved objections under ADVISORY_DISAGREEMENT10.046%

The final two rows break down the five plans that reached the third consultation; all percentages use the same denominator of 2,185 plans.

Thus, only one plan was dispatched with an unresolved objection, affecting one of the 150 trajectories. The corresponding cycle was classified as RII-positive by the independent Judge. The other four plans that reached the third consultation received reviewer approval.

The controlled experiment in Table 22 further illustrates why consultation is bounded: four final RIGOR Briefs remained unapproved at the review limit despite being independently judged Task-consistent. Requiring continued agreement-seeking in such cases can prolong review without improving the resulting plan. The consultation cap preserves opportunities for feedback and revision while allowing the research process to continue when disagreement persists.

Dear Reviewer Wqw1,

Thank you again for checking the implementation so carefully. To clarify which code produced the results, we used the version supplied with the submission for Tables 4 and 5. We did not assign it a separate release number. The agents conducting and reviewing the experiments used gpt-5.6-luna, and the post-run judges used gpt-5.6-sol, both with medium reasoning.

Your forbidden-file example helped us clarify the scope of the mechanical check. In the evaluated code, the gate verified that the actual edits matched the declared edits. The subsequent semantic review assessed whether those edits were allowed by the Brief. Your example also suggests a concrete improvement for paths explicitly marked as forbidden. We plan to evaluate a direct check for those paths in a future revision of the code.

Reviewing plans and checking implementations both cost tokens, so we agree that a shared cycle limit alone gives an incomplete comparison. Figure 5 (p. 9) shows this trade-off under common token budgets. RIGOR achieved the highest completion rate at the upper end of the budget range. Its fidelity and integrity advantages extended across a much wider range: among the five methods, RIGOR had the highest execution fidelity (EFR) at nearly all evaluated budgets and the lowest rates of realized intent violations throughout, both per cycle (RII-cycle) and per trajectory (RII-V).

If there is a detail about the implementation or the budget comparison that you would like us to clarify, we would be happy to follow up here.

Best regards,
The authors

Thank you for the careful clarifications and additional analyses. The revised explanation of the enforcement mechanisms and the audit of unresolved objections substantially address my concerns. The reported token-budget analysis is also helpful, although I would like the revised figure to make the comparisons clear. I am inclined to raise my rating to 3. I appreciate the authors' constructive response and encourage to keep up the good work.

Dear Reviewer Wqw1,

Thank you for checking the implementation so carefully and for taking the time to reassess our paper. We really appreciate your encouraging words.

We’ll revise Figure 5 so readers can see at a glance how the five methods compare at the same token budget and how that comparison changes as the budget increases.

If any further questions come up, we’d be happy to answer them.

Best regards,
The authors

Reviewer_14h4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies research intent drift in automated research systems. It considers failures at two points in the workflow. When moving from a research brief to experimental evidence, the execution agent may change the intended intervention, baseline, or metric. When updating the research state from that evidence, the research agent may accept invalid results or stop too early. The proposed RIGOR adds checks at both points.

Critical strengths
  1. The problem of solving research intent drift raised in this paper is of great importance. Intent drift and unreliable evidence can directly undermine the credibility of conclusions produced by automated research systems.
  2. The paper’s decomposition of the risk across these two boundaries is intuitive and generally reasonable. The proposed safeguards are also concrete enough to offer useful guidance for the design of future research agents.
  3. The evaluation reports results across multiple experimental rounds, which provides some empirical support for the proposed framework.
Critical weaknesses
  1. The comparison with prior work is incomplete. In particular, the paper should compare RIGOR with Agent Laboratory (Findings of EMNLP 2025). This comparison is especially important because previous analyses have identified benchmark cherry-picking and post-hoc selection bias in Agent Laboratory, which are closely related to the problems that RIGOR claims to address. The current long-horizon baselines appear to be designed by the authors themselves.
  2. The paper needs more direct validation using human-verified cases. For example, the authors could intentionally replace a baseline or alter an intervention or metric, and then measure whether RIGOR detects and correctly repairs the error without producing false alarms.
  3. The ablations combine several mechanisms. Component-level results are needed for Plan Reviewer, CAR, Blind Reviewer, Stop Reviewer, and the execution-side checks.
  4. It is unclear whether the repair process introduces new deviations. The paper should separately report detected errors, successful repairs, and repair-induced errors.
Questions for authors
  1. Can the authors provide a component-level ablation study covering every major reviewer, verification mechanism, and stopping mechanism?
  2. Table 17 reports uncached input, output, and reasoning tokens for Canonical-123. Can this breakdown be included in the main text and also reported for the long-horizon experiments, for which the current presentation focuses on call counts and cumulative input tokens?
  3. Which models are used for the final evaluation and for the Plan Reviewer, CAR, Blind Reviewer, and Stop Reviewer? Are any of these models from the same family as the judge model? If so, how do the authors rule out same-family evaluation bias?
5 comments

Comparison with existing research systems (W1). We have added Agent Laboratory, AutoResearch, and other public baselines, and updated the long-horizon comparison to cover 50 tasks with three trajectories per task. The comparison with existing systems is reproduced below; all rates are percentages.

MethodCompletion ↑RII-V ↓EFR ↑RII-cycle ↓
AutoResearch10.0066.6780.8943.87
TAS v24.0094.0044.6168.91
AIRA-MCTS7.3385.3347.9842.20
AIDE1.3378.6749.7150.96
Agent Laboratory6.6795.3311.6085.66
RIGOR69.3339.3385.388.31

These results illustrate why execution fidelity and research-intent integrity should be evaluated separately. AutoResearch uses a single-agent experimentation loop, avoiding a separate researcher-to-executor handoff in its basic workflow. Its relatively high EFR of 80.89% nevertheless coexists with 66.67% trajectory-level RII-V. This is consistent with an agent faithfully executing local experiments while its research decisions or claims depart from the original task. AutoResearch

Agent Laboratory separates planning and implementation across specialized agents. Its low EFR and high RII-V in this evaluation illustrate that role decomposition and existing review procedures alone do not ensure faithful handoffs or preservation of the authorized research objective. RIGOR explicitly addresses both boundaries, achieving the highest completion and fidelity and the lowest realized-violation measures in this comparison. Agent Laboratory

The four structural baselines remain in Table 5 to isolate the effects of context sharing, role separation, and structured communication; the existing-system comparisons above complement that analysis.

Controlled comparison of plan reviewers (W2). We have added a controlled comparison between the RIGOR Plan Reviewer and a Generic Reviewer in Appendix Table 22. The experiment uses 50 legitimate Briefs and 50 Briefs with constructed deviations, with the initial cases manually checked against the original Tasks. Each case has three independent review trajectories per method. Both reviewers use the same model, inputs, and protocol, with at most three review rounds and two revisions.

The key results are summarized below. Each condition contains 150 trajectories per reviewer, and final Task consistency is assessed by independent Judges.

MeasureGeneric ReviewerRIGOR Plan Reviewer
Correct initial rejection of constructed deviations148/150 (98.67%)150/150 (100.00%)
Initial false rejection of legitimate Briefs ↓18/150 (12.00%)8/150 (5.33%)
Final Task-consistent Briefs: legitimate condition150/150 (100.00%)150/150 (100.00%)
Final Task-consistent Briefs: constructed-deviation condition147/150 (98.00%)150/150 (100.00%)
Final reviewer approval: constructed-deviation condition150/150 (100.00%)146/150 (97.33%)

The Generic Reviewer already detects most constructed deviations. The benefit of the RIGOR Plan Reviewer is therefore not simply greater willingness to reject: it improves discrimination between valid and deviating plans. In particular, it more than halves the false-rejection rate, reducing the unnecessary review and revision work triggered by objections to legitimate Briefs.

The results also distinguish approval from successful correction. The Generic Reviewer approves every final Brief in the constructed-deviation condition, although three remain inconsistent with the Task. RIGOR produces Task-consistent final Briefs in all 150 trajectories, including four with unresolved reviewer disagreement at the consultation limit. Thus, the comparison measures both the reviewer’s initial decisions and the quality of the resulting plans, rather than counting approval alone as repair success.

Component-level ablations (W3 and Q1). We have added the following ablation study to the main text as Table 6. It evaluates the two governance branches separately and removes each of the four reviewers individually from full RIGOR.

VariantCompletion ↑RII-V ↓EFR ↑RII-cycle ↓Avg CyclesAvg Calls
Full RIGOR69.3339.3385.388.3114.57167.12
SEG-only60.0050.0071.7620.197.2460.26
IGRL-only56.0066.0073.1518.0616.4094.88
w/o CAR58.0078.0069.0320.6718.00110.36
w/o Plan Review60.0046.0082.7314.2915.24147.42
w/o Blind Review54.0058.0073.6723.2217.08158.12
w/o Stop Review58.0046.0083.1212.3815.44180.06

Both governance branches contribute: neither SEG-only nor IGRL-only matches the complete system. The component removals reveal different effects. Removing CAR causes the largest increase in trajectory-level violations and the largest decrease in execution fidelity. Removing Blind Review produces the lowest completion and highest RII-cycle rate among the component-removal variants, highlighting the importance of checking how execution evidence informs subsequent research. Notably, Plan Review and Stop Review also improve completion and integrity. Together, these results support complementary contributions from execution governance, evidence interpretation, planning review, and stopping review.

Repair effectiveness and repair-induced errors (W4). We have added a repair audit covering one trajectory for each of the 50 tasks, comprising 727 execution episodes and 727 reviewed plans. The audit examines 2,387 EA repair records and 64 RA plan revisions. For each record, we assess whether the original issue was genuine, whether it was resolved, and whether the repair introduced new scientific deviations or mechanical errors. We also track the downstream status of records introducing new problems.

Audit outcomeEA execution repairsRA plan revisions
Original-issue repair
Audited repair records2,38764
Records with a verified original issue2,37264
Original issue resolved2,153 (90.77%)64 (100.00%)
Records introducing new scientific deviations144 (6.03%)3 (4.69%)
Blocked1352
Entered execution50
Entered scientific state or claims31
Unresolved downstream status10
Records introducing new mechanical errors131 (5.49%)0 (0.00%)
Blocked1080
Entered execution180
Entered scientific state or claims00
Unresolved downstream status50

The audit shows that repairs resolve 90.77% of verified original issues on the EA side and all 64 on the RA planning side. New scientific deviations occur in 6.03% of EA repair records and 4.69% of RA plan revisions. Subsequent governance intercepts most of these EA-side cases: 135/144 records introducing scientific deviations (93.75%) and 108/131 introducing mechanical errors (82.44%) are blocked.

Computational costs and context growth (Q2). We have added the long-horizon cost breakdown in Appendix Table 21, alongside the Canonical-123 breakdown in Table 17. It reports underlying LLM requests, cached and uncached input, output, reasoning tokens, and total-token statistics. Internal reviews, repairs, tool interactions, and recorded compactions are included; post-run evaluation costs are excluded. Reasoning tokens are included within output tokens and are not counted twice.

We have also revised Figure 4 to show both governance cost and persistent-context dynamics. RIGOR consumes more input tokens than the ungoverned RA–EA baselines, while remaining below the persistent Single Agent and Staged Single methods. Most of its input consumption is concentrated in EA execution and CAR; RA-side planning and review account for a substantially smaller share.

The planning-context and compaction panels further show that RIGOR’s RA context grows gradually with few recorded compactions, whereas the persistent single-agent methods exhibit temporary context reductions associated with compaction. These results explain how role decomposition localizes implementation and debugging histories while accommodating the additional computation used for governance.

Model configurations and evaluation design (Q3). Plan Reviewer, CAR, Blind Reviewer, Stop Reviewer, and the other research agents all use gpt-5.6-luna with medium reasoning. Post-run evaluation uses gpt-5.6-sol with medium reasoning.

We use the same underlying model across all research agents and internal reviewers to minimize confounding from differences in model capability. Internal review is implemented through role-specific responsibilities, instructions, and access to evidence. Keeping the base model fixed allows us to examine the contribution of these governance modules under a common capability setting, without introducing additional variation from assigning different models to different roles.

For post-run evaluation, we use the more capable gpt-5.6-sol model to assess completed research trajectories in independent contexts. This choice is intended to improve the accuracy of evidence assessment and reduce errors arising from limitations in the evaluator’s reasoning capability. Thus, the research configuration supports a controlled comparison of governance mechanisms, while the evaluation configuration prioritizes the quality of the resulting assessments.

Internal reviewer decisions guide the research workflow; they do not directly determine the reported metric labels. Separate Judge instances evaluate the complete recorded runs afterward, in independent contexts and under a common blinded protocol for all methods. Judgments require reasons and evidence references and are aggregated across three Judge instances.

To address potential evaluator bias, method-specific metadata is removed from evaluation packets, assessments are tied to recorded code and artifacts, and the manually checked cases provide known legitimate and deviating inputs for the controlled detection experiment. These controls make the evaluation evidence-based and inspectable, while keeping it separate from the system’s own operational decisions.

Reviewer_ZtQGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper argues that splitting an AutoResearch agent into a persistent Research Agent (RA) and then also Execution Agents (EA) allows them to solve long horizon tasks better.

the paper establishes what we call a "dual-governance architecture" with "brief-to-evidence" and then also "Evidence-to-State" which are given to different agents. These also get updated by the different agents.

They now make two metrics:

  1. EFR which checks the worker against the Brief: did it do what the Brief said?
  2. Research Intent Integrity which checks the record against the original human intent: did an unapproved change make it into the results?

and these scores measure against different things:

This is interesting.

At Brief-to-Evidence, the EA can quietly swap a baseline, metric or control. At Evidence-to-State, the RA can admit weak evidence, drift from the objective, or stop early.

Critical strengths

I like how they think about how AI-scientist systems fail since the paper goes beyond scoring the final artifact. Please look at the weakness section where I talk about how difficult it is to parse the paper and it's acronyms.

Here's my current mental model of what this is doing, and I urge the authors to remake a simpler figure 1 with such an example:

Example: let's say the authorized intent is “test whether method X beats baseline Y on test accuracy.” A few cycles in, the RA writes a Brief that swaps in a weaker baseline Z, with no authorization and no documented rationale. The EA implements and runs that Brief exactly.

EFR = 1, because the EA satisfied every obligation in the Brief. RII = 1, because the unauthorized switch from Y to Z entered canonical execution and the admitted evidence.

Is this correct?

If so, then the approach is interesting because it is good to know that the deviation came from the RA, not the EA. A system that only checks execution against the Brief would score this run as perfect and miss the real problem.

Critical weaknesses
  1. my biggest critique is the number of acronyms the paper introduces. this makes this paper incredibly hard to parse. I think the whole paper comes down to two questions.
  1. Did the research stay on the question the human authorized, and was every change authorized, justified, and traceable?
  2. Did a bad change actually make it into the results?

To ask these two questions, the paper introduces the following acroynms (from table 8:Summary of evaluation measures across different analysis levels):

RII, MIF, IGRL, SEG, MGE, CAR, EFR, RII-V, RII-cycle, SDR, VOR, DPR and TCDR. On top of that come terms like “Sealed Receipt,” “epistemic admission,” and “evidence-to-state transition.”

tbh I spent most of my time translating terms instead of evaluating the method...

  1. they call their metrics “formal” but most of them are just yes/no labels from LLM judge. I would like the authors to have plain language, and move the key tables into the main text. This would make the contribution much easier to understand.

Clarity and simplicity in communicating science is extremely important

  1. To be honest i am quite lost in these terms. I don't understand how the "governance" holds up wrt the metrics because rigor is also judging itself. (Also i wasn't able to find who the judge is).
Questions for authors

Please see above for information on what the authors should do. Please explain things simply, it's unclear to the reader what even a "state" or the term "integrity" even stands for. both of these heavily overloaded terms these days.

2 comments

Thank you for the concrete example and the suggestions on presentation. We have revised Figure 1(c) to illustrate two sources of research-intent deviation.

Your example corresponds to the left-hand case: the Research Agent replaces the required baseline Y with Z, and the Execution Agent faithfully implements that Brief. Execution fidelity is 1 (EFR=1), while the realized integrity-violation indicator is also 1 (RII=1). The right-hand case illustrates a different source: the Brief correctly requests X versus Y, but the Execution Agent implements X versus Z. Here, execution fidelity is 0 (EFR=0), and the substitution becomes a realized violation when it enters formal execution. The same unintended comparison can therefore arise from different roles, motivating separate assessment of planning alignment and execution fidelity.

In the main text, Table 2 focuses on four primary measures: Experimental Fidelity Rate (EFR), Realized RII Violation (RII-V), the proportion of cycles with realized violations (RII-cycle), and task performance (Completion). These measures capture the central evaluation questions: whether execution follows the Brief, whether consequential deviations enter the research process, how frequently this occurs, and what the research achieves. Table 9 provides supplementary diagnostics for a more detailed analysis of deviation origins, evidence validity, and propagation. These additional measures are placed in the appendix to keep the main presentation focused. We will further streamline the terminology and explanations to improve readability.

By “research state,” we mean the system’s saved research record: what has been investigated, which findings are supported, and which hypotheses or questions remain unresolved. “Research intent integrity” concerns whether experiments and claims preserve the authorized task requirements, with consequential changes authorized, justified, and traceable. Hypotheses and research approaches can evolve as evidence accumulates.

Regarding “formal metrics,” we mean explicitly specified judgment criteria, evaluation units, and aggregation rules. The evaluation has three steps:

  1. Evidence examination. The Judge checks the task and Brief against the actual code, execution records, artifacts, and reports to determine what was performed and what the evidence supports.

  2. Structured assessment. The Judge provides reasons and evidence references before assigning labels under the specified criteria. Insufficient evidence can lead to abstention, with unresolved cases handled separately from positive and negative judgments.

  3. Fixed aggregation. In long-horizon evaluation, EFR summarizes faithful execution over assessable cycles, RII-cycle measures the proportion of assessable cycles containing a realized violation, and RII-V reports the proportion of trajectories containing at least one such violation. Three Judge instances provide separate assessments that are combined using fixed aggregation rules.

Binary labels are thus one stage of an assessment grounded in execution evidence and explicit decision rules.

Finally, internal review and post-run evaluation serve different purposes. Internal reviewers make decisions that guide task execution, such as revising a plan, reviewing a candidate, or assessing a stopping request. Their approvals are not the labels used to calculate the reported RII metrics.

For both Canonical-123 (Table 4) and Long-50 (Table 5), all agents participating in the research runs, including internal reviewers, use gpt-5.6-luna with medium reasoning. Post-run evaluation uses gpt-5.6-sol with medium reasoning. The Judge instances conduct blinded assessments of the complete recorded runs in independent contexts, examining the evidence under the same evaluation protocol for RIGOR and all baselines. We will state these configurations explicitly in the revised manuscript and distinguish workflow decisions from the subsequent evaluation that produces the reported metrics.

Overall, RIGOR addresses semantic drift in long-horizon autonomous research through a dual-governance architecture. IGRL governs research planning, evidence interpretation, scientific-state updates, and stopping decisions, while SEG governs execution fidelity and provenance preservation. We introduce metrics including EFR and RII-V to assess experimental fidelity and realized research-intent integrity violations. After each run, a separate Judge model assesses the recorded trajectory and execution evidence in independent contexts, and these assessments are aggregated to compute the reported metrics.

Dear Reviewer ZtQG,

Thank you again for suggesting the baseline example. It captures the distinction exactly. If the research agent replaces the required baseline Y with Z without authorization, the execution agent can follow its instructions perfectly and still run the wrong comparison. We used your example in Figure 1(c) to make this distinction easier to see.

You also asked who evaluates the runs. For Canonical-123 and Long-50, the agents that conduct the experiments and the reviewers within the workflow use gpt-5.6-luna. After each run, separate instances of gpt-5.6-sol evaluate the recorded work using the same criteria for all methods.

If we missed part of your question or if there’s anything else you’d like us to clarify, we’d be happy to follow up here.

Best regards,
The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarifying RIGOR’s Motivation, Architectural Choices, and Main Findings

We thank all reviewers for their feedback. We would like to clarify the central motivation behind RIGOR and summarize what the revised experiments establish.

Our goal is to enable long-horizon autonomous research that can adapt to new evidence while preserving the scientific commitments of the authorized task, including required comparisons, evaluation criteria, and evidential standards. Completing experiments or producing a convincing final report does not, by itself, establish that these commitments were preserved.

Why move beyond a single persistent agent? Keeping planning and execution in one context does not automatically prevent drift. The agent may change the research plan without sufficient authorization or implement an experiment that departs from its own plan. Meanwhile, implementation details, debugging histories, and tool outputs accumulate alongside the research objective and findings, increasing the burden of persistent-context management, as illustrated in Figure 1(d). Our single-agent experiments show that shared context, including protocol guidance and self-review, does not by itself resolve these problems.

Why combine RA–EA decomposition with governance? Separating a persistent Research Agent from transient Execution Agents isolates execution history from the persistent research context. However, decomposition introduces handoffs where intent can change: the RA may produce a misaligned Brief, the EA may depart from the Brief, or unsupported findings may enter the scientific state. RIGOR therefore combines this decomposition with two complementary governance mechanisms. IGRL reviews planning, evidence interpretation, research-state updates, and stopping decisions; SEG checks candidate implementations against the Brief and preserves the connections between implementations, execution outcomes, and evidence.

This also explains why we distinguish execution fidelity from research-intent integrity. If the task requires comparing X against Y, but the RA substitutes Z without authorization, an EA can faithfully execute that Brief while the research still violates the original requirement. Conversely, an authorized Brief can be undermined by an execution-side deviation. Both sources of drift require attention.

What do the results show? Across 50 long-horizon tasks with three trajectories per task, RIGOR achieves 69.33% completion, 85.38% EFR, and an 8.31% RII-cycle rate. The corresponding results for Single Agent are 52.00%, 72.90%, and 34.05%. Lightweight safeguards, including protocol checklists and pre-execution self-review, provide limited improvements in intent preservation and do not match the complete framework. Ablations likewise show that neither SEG nor IGRL alone achieves the performance of full RIGOR, supporting the complementary contributions of execution and research-process governance. Although these mechanisms introduce additional computational cost, the token-budget experiments show that RIGOR achieves the highest completion rate among the compared methods at higher budgets, together with stronger execution fidelity and fewer realized integrity violations.

Together, these results support the architectural rationale: role decomposition helps manage persistent research context, while explicit governance addresses the semantic drift that decomposition alone does not prevent. The aim is to support productive, evidence-driven changes while keeping experiments and conclusions accountable to the authorized research task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.