acceptodds
Under review as a conference paper at ICLR 2027

GitHarness: Git Init Your Agent Working Memory for Perpetual User Requirements

Abstract

LLM-based agents increasingly collaborate with users on long-horizon tasks, accumulating evidence, code, and drafts through extensive search, reasoning, and execution. As users inspect these results, they may supply missing information requirement completion, introduce new requirements requirement elicitation, or revise existing ones requirement shift. These changes often affect only part of the accumulated work, yet agents may carry forward obsolete information or turn local revisions into global rewrites. Existing approaches clarify current intent without determining how prior work should change, or reuse execution histories under a fixed objective. We address this gap by formulating dynamic-requirement collaboration as joint requirement tracking and local update. We introduce GitHarness, a pluggable Git-style framework that organizes requirement states and their corresponding harness work states into a branchable version history. A trainable Git Agent resolves requirement changes and selects a semantically compatible historical state. A unified version interface then restores that state and creates a new branch, enabling the underlying harness to exclude obsolete information, inherit compatible work, and focus execution on affected parts. The Git Agent is trained through interface-level black-box reinforcement learning, with downstream harnesses and task-execution models kept fixed. We also construct MTAgentBench, a verifier-preserving benchmark covering mathematical reasoning, text-to-SQL, agentic search, software engineering, and research synthesis. Experiments demonstrate strong task performance alongside effective requirement tracking, preservation of valid work, and efficient execution.

Primary area:
generative models
Keywords:
large language model agents, human-AI collaboration, agent memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yj7rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes making agents operate like Git to handle multiple requirements over long-running user sessions. It targets real-world user journeys where requirements evolve, either revealed gradually or modified along the way.
  • This is done in two main steps. First, a Git Agent performs requirement resolution to work out the full current requirement and pick the best base version to resume from. Then, in branch execution, an update agent restores that version and carries out the requested changes. If a past version already satisfies the request, it’s reused directly.
  • The Git Agent can also be trained with RL while the downstream harness and executor stay frozen. The reward combines final-task correctness with an LLM judge’s score for how well the agent tracks the user’s requirements.
Critical strengths
  • The paper is well motivated: most benchmarks don’t effectively measure how agents perform in an evolving, user-in-the-loop setting, where requirements are added, refined, or revised as the user inspects intermediate results. Framing this as deciding where to resume and what to update is intuitive. The Git-style abstraction is a natural fit, and because it plugs into existing harnesses without changing them, it’s easy to adopt across different agent setups.
  • The results are consistent across 5 domains, 2 backbones, and 3 harnesses: GitHarness beats native continuation in all 30 settings while cutting token usage substantially.
Critical weaknesses

The benchmark does a good job illustrating requirement completion and elicitation, but for requirement shift the scope feels limited to cases where a clean revert is possible. Every conversation has to end back at the original task, so shifts mostly undo earlier temporary edits, which directly rewards rolling back to a saved version (Exact Reuse dominates on Code). Two things would make the results more convincing:

  • How does GitHarness compare with widely used harnesses that already have built-in memory, compaction, or checkpointing? The current comparisons are mostly against research methods (U-Fold, GCC).
  • How does it perform under different styles of requirement change? Results are reported only in aggregate, so a breakdown by change type, plus shifts that lead to genuinely new requirements, would show where the gains actually come from.
Questions for authors
  1. The benchmark spans completion, elicitation, and shift, but results are reported only in aggregate. How does GitHarness perform on each change type, and on each operator? Is the gain concentrated in cases where the final request matches an earlier version (Exact Reuse dominates Code)? Results on conversations whose final requirement differs from every earlier state, and comparisons with widely used harnesses that have their own memory, compaction, or checkpointing, would show whether the gains generalize.
  2. How were the 7-turn extensions of the benchmarks in section 5.1 generated: which model was used, why seven turns, and how were operators and anchors chosen? What did the human inspection involve, and how many conversations were rejected? An example conversation and the distribution of operators would help to judge how realistic the requirement changes are.
5 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

W1 / Q1: Harnesses with native memory and checkpointing

hank you for this suggestion. Our experiments already cover commonly used harnesses with native state-management capabilities, beyond U-Fold and GCC. TCRAG maintains an explicit message stack: Backtrack pops the most recent Thought or Action–Observation path and pushes the feedback back onto the stack. StackPlanner's CentralAgent uses Reflect to clean its memory stack and Summarize to write compressed information back to the cross-turn MemoryStack. Claude Code automatically creates file checkpoints. To construct a stronger Claude Code baseline, we use prompt-based instructions and hooks so that Claude Code determines whether rollback is needed, while a hook selects the target checkpoint and restores the corresponding context. On top of these native or enhanced state mechanisms, GitHarness uses an independent Git Agent to explicitly reconstruct the full current requirements, maintain the correspondence between accepted requirement versions and work states, and jointly select a historical base and the ExactReuse, ReuseAndPatch, or FreshSolve execution mode. Within each harness, paired methods use the same model, tasks, tools, budget, and verifier. The DeepSeek-V4-Flash results are as follows.

MethodMathSQLSearchCodeResearch
TCRAG74.069.034.056.042.54
TCRAG + GitHarness93.080.056.062.046.65
StackPlanner75.071.037.067.047.63
StackPlanner + GitHarness94.078.058.074.049.96
OpenHands76.067.027.064.048.88
OpenHands + GitHarness92.079.057.072.051.48
Claude Code (adaptive checkpoint rollback)78.070.035.070.049.12
Claude Code + GitHarness93.079.057.077.052.06

GitHarness improves all 20 paired cells across four harnesses and five domains. The gains are 21–30 points on Search and 15–19 points on Math; even on Code and Research, where the baselines are stronger, the gains remain 6–8 and 2.33–4.11 points, respectively. These gains occur on top of different state mechanisms, including message stacks, summary-based memory, and adaptive checkpoint rollback, supporting the additional value of requirement-conditioned historical-state selection and execution beyond the existing native state-management capabilities.

AuthorsOpenReview ↗

Rebuttal for W2&Q1

W2 / Q1: Breakdown by requirement-change operator

We agree that aggregate scores can obscure which requirement changes account for the gains. To provide results by change type, we add a turn-level atomic-requirement evaluation on Research. We decompose and score all active requirements based on user requests up to the current turn, and group the results by the nine requirement-change operators defined in the paper. GitHarness and Native use exactly the same turn-level rubrics and judge. We report retained-requirement satisfaction, new/modified-requirement satisfaction, and all-current-requirement satisfaction side by side.

OperatorRetained-requirement satisfaction (GitHarness)Retained-requirement satisfaction (Native)New/modified-requirement satisfaction (GitHarness)New/modified-requirement satisfaction (Native)All-current-requirement satisfaction (GitHarness)All-current-requirement satisfaction (Native)
Reveal0.9920.9010.9670.9550.9880.908
OutputControl0.9940.9190.9740.9700.9910.926
Refine0.9900.9040.9600.9580.9860.912
Constrain0.9890.8850.9490.9460.9820.896
Comparison0.9870.8920.9440.9510.9790.904
Pivot0.9860.8710.9340.9360.9740.885
ParameterUpdate0.9840.8780.9530.9430.9780.890
Correct0.9860.8840.9580.9480.9810.896
Withdraw0.9950.8260.9700.9070.9860.855

GitHarness outperforms Native on retained-requirement satisfaction and all-current-requirement satisfaction for all nine operators. New/modified-requirement satisfaction is higher for seven operators and slightly lower than Native for Comparison and Pivot. The gains in all-current-requirement satisfaction range from 0.065 to 0.131 across operators. For the Shift operators ParameterUpdate and Correct, GitHarness achieves all-current-requirement satisfaction of 0.978 and 0.981, compared with Native's 0.890 and 0.896. For Withdraw, the scores are 0.986 and 0.855. These turn-level results show that GitHarness’s gains extend across requirement completion, elicitation, and shift, including parameter updates, premise corrections, and withdrawals.

AuthorsOpenReview ↗

Rebuttal for Q2

Q2: Seven-turn construction and human inspection

  1. Thank you for this suggestion. For dataset construction, we use DeepSeek-V4-Flash to generate candidate conversations across all five domains. The process has three steps. First, we design several seven-turn operator templates for each domain. Each template specifies the requirement changes to cover and their necessary dependencies. A fixed random seed is used to select a compatible template for each source task. Second, we extract the goals, parameters, constraints, and output format from the original task, identify the slots each operator can modify, and use DeepSeek-V4-Flash to instantiate these structured edits as natural-language conversations. For example, ParameterUpdate changes numerical or entity parameters, OutputControl changes the output form, and Comparison adds a comparison object at the same level. A candidate is rejected if the task has no valid slot for the edit. Finally, for intermediate changes without ordering dependencies, we use a fixed random seed to vary their order and increase trajectory diversity. We also check dependencies such as introducing a requirement before withdrawing it and eventually restoring the original task, filtering out trajectories with semantic contradictions or inconsistent state transitions.

In addition, the authors conduct human inspection with LLM assistance, primarily checking:

  • Whether the conversation is natural and reasonable.
  • Whether each turn fully expresses its requirement edit.
  • Whether the operator matches the text.
  • Whether withdrawal relationships are consistent.
  • Whether the terminal state restores the source requirements.
  • Whether answers or evaluator information are leaked.

The candidate-construction success rate is 65.4%, and the rejection rate due to semantic or structural problems is 34.6%. To illustrate requirement evolution in a seven-turn conversation, we will add a complete example to the revised PDF, showing the user utterance, operator, modified requirement slot, and active requirements at each turn. We also report the overall proportions of the nine operators across the five domains.

OperatorRevealOutputControlRefineConstrainComparisonPivotParameterUpdateCorrectWithdraw
Proportion20.1%11.1%7.2%10.7%8.8%6.3%7.5%11.3%17.0%
  1. We use seven turns in the main table to remain consistent with the main experimental setting of [1]. That work applies argument reveal, argument revision, and function switch twice each, producing six intent transitions and, together with the initial request, up to seven user turns. We extend these three transitions to nine finer-grained requirement operators: Completion includes Reveal, OutputControl, and Refine; Elicitation includes Constrain, Comparison, and Pivot; and Shift includes ParameterUpdate, Correct, and Withdraw. Seven turns is the standardized experimental protocol for the main table, not a length limit of the construction method or GitHarness.

In addition to the seven-turn setting, we evaluate interleaved multi-task conversations with 10, 15, 20, 25, and 30 turns. The results are reported in Table 4, where H denotes the number of user turns.

DomainMethodH=10H=15H=20H=25H=30
MathNative73.072.069.063.062.0
MathRestart71.071.067.061.059.0
MathU-Fold82.083.081.079.078.0
MathGitHarness92.090.087.085.084.0
ResearchNative42.9241.5737.8635.3934.53
ResearchRestart41.0540.0839.1435.8827.70
ResearchU-Fold37.5337.1335.4133.5234.16
ResearchGitHarness49.5349.2748.7147.5247.32

GitHarness achieves the best results at every horizon in both domains. At 30 turns, Math reaches 84.0, exceeding the strongest baseline by 6.0 points, and Research reaches 47.32, a gain of 12.79 points. These results support its effectiveness in longer interactions.

Reviewer_ca9YOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To facilitate reusing useful progress while excluding obsolete information when users’ requirements change during coding agent execution, this paper proposes GitHarness, a Git style framework for managing agent work states. Specifically, GitHarness maintains checkpoints that associate requirement states with corresponding execution states and artifacts. GitHarness also allows a trainable agent to select appropriate historical states for reuse based on the requirement changes. To evaluate GitHarness, the authors constructed MTAgentBench with synthetic multi-turn requirement changes from existing benchmarks across five domains. Experiments across multiple agent backbones and harnesses demonstrate improvements in task success rates and token efficiency over existing baselines.

Critical strengths
  1. Addressing evolving user requirements is a relevant and practically important problem for improving the reliability and efficiency of coding agents.
  2. Artifact is available for validating the reproducibility.
Critical weaknesses
  1. While GitHarness demonstrates clear performance improvements over GCC, the two frameworks share similar Git style mechanisms for managing and reusing agent context. The main difference appears to be the explicit requirement-aware historical state selection. However, GCC already supports adaptive branching and historical context retrieval, which potentially allows similar requirement-aware selection. The paper could better clarify the methodological novelty of GitHarness beyond adapting existing version-control mechanisms to evolving user requirements, and provide further analysis (e.g., failure cases of GCC or adding requirement-aware selection to GCC) of what specifically drives its improvements over GCC.
  2. The paper emphasizes preserving unaffected work and selectively updating only the parts impacted by requirement changes. However, the experiments primarily evaluate task success rates and token efficiency, without directly measuring whether unaffected artifacts are preserved or unnecessary modifications are avoided. While this is a relatively minor concern given the demonstrated performance improvements, I would appreciate it if the authors add additional artifact-level evaluations to better support this claim.
Questions for authors
  1. Could the authors further clarify the methodological novelty of GitHarness compared to GCC, and analyze which specific design choices account for its performance improvements over GCC?
  2. Could the authors provide more direct evidence that GitHarness preserves unaffected artifacts and avoids unnecessary modifications?
4 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

W1 / Q1: Methodological differences from GCC

  1. Thank you for your comments. GitHarness differs from GCC in three connected designs: explicitly binding a full requirement state to a recoverable work checkpoint; jointly reconstructing the current requirements and selecting a compatible historical state after feedback; and choosing ExactReuse, FreshSolve, or ReuseAndPatch based on the correspondence between requirements and checkpoints. GCC manages execution history and alternative reasoning paths through COMMIT, BRANCH, MERGE, and CONTEXT, and a model could potentially approximate similar behavior using these general operations. GitHarness organizes requirement equivalence, requirement-conditioned base selection, and execution-mode selection into an explicit protocol. Furthermore, the independent Git Agent can optimize these decisions through Harness RL while keeping the downstream executor unchanged.

  2. Following the reviewer's suggestion, we add a controlled GCC + Requirement-Aware Selection experiment. Under the same StackPlanner and DeepSeek-V4-Flash setting, we add GitHarness's requirement resolution and historical-state selection to GCC while retaining its original execution mechanism.

MethodMathSQLSearchCodeResearch
GCC61.056.046.066.041.45
GCC + Requirement-Aware Selection72.059.049.067.043.72
Full GitHarness94.078.058.074.049.96
w/o Historical Version Selection84.062.042.044.045.59
w/o Exact-State Reuse88.076.054.063.049.88
w/o ReuseAndPatch92.075.053.069.046.63
w/o FreshSolve89.068.047.072.042.56

Adding requirement-aware selection to GCC improves the five domains by 11.0, 3.0, 3.0, 1.0, and 2.27 points, showing the effectiveness of explicit requirement resolution and historical-state selection. The GitHarness ablations further show that improvements do not come only from this selection mechanism: disabling Exact-State Reuse reduces Code from 74.0 to 63.0; disabling ReuseAndPatch reduces Code and Research to 69.0 and 46.63; and disabling FreshSolve reduces SQL, Search, and Research to 68.0, 47.0, and 42.56. Together, these results support the contributions of requirement-aware historical selection and adaptive branch execution to GitHarness's performance. Paired case analyses in the revised appendix will further explain how these differences arise.

AuthorsOpenReview ↗

Rebuttal for W2&Q2

W2 / Q2: Preserving unaffected artifacts

Thank you for this suggestion. We will add a Research case to the revised appendix, showing the requirement change, historical checkpoint selection, and a section-level before/after comparison to directly illustrate which artifact content is retained and which content is modified in response to the new requirement.

The request adds a “non-Western modernization” dimension to a report originally organized around postcolonialism, Orientalism, and global history. GitHarness restores the accepted three-dimension report and executes ReuseAndPatch, retaining the three original theoretical sections, adding the fourth dimension, and updating the analytical framework, cross-dimension synthesis, timeline, evidence gaps, and conclusions accordingly. The section-level comparison will provide concrete evidence of artifact preservation and modification.

As complementary evidence at the requirement level, we add a turn-level evaluation on Research to measure whether unchanged requirements remain satisfied while new requirements are implemented. Report requirements can naturally be decomposed into verifiable atomic items. We construct a rubric from the requirements still active at each turn and use Qwen 3.8 Flash as the judge to score each item.

MethodUnchanged-requirement satisfactionNew/modified-requirement satisfactionAll-current-requirement satisfaction
GitHarness0.9900.9600.984
Native0.8850.9380.896
Restart0.8380.6320.797

GitHarness achieves 0.990 on unchanged-requirement satisfaction, compared with 0.885 for Native and 0.838 for Restart; its new/modified-requirement satisfaction is 0.960. This supports its ability to maintain the correctness of unchanged requirements while implementing new requirements. The turn-level aggregates reflect whether requirements remain satisfied, while the before/after comparison in the revision will show how specific content is retained and updated.

I appreciate the authors for the additional experiments and discussions, which have adequately addressed my concerns. I've updated my score accordingly.

AuthorsOpenReview ↗

Thank You !!

Thank you for your follow-up and constructive feedback throughout the discussion. We appreciate your time and careful consideration of our work.

Reviewer_ACXQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper deals with the scenario when the user's ask changes mid-conversation. Its proposition is to treat the agent's workspace like a Git repo: every successful turn commits a version pairing the requirement with a restorable snapshot, and a trainable "Git Agent" reads new feedback, reconstructs the full current requirement, and picks which past version to branch from, after which a router decides whether to return the old snapshot, start fresh, or fork and patch. The Git Agent is trained with the executor and harness frozen. Evaluation uses five single-turn benchmarks converted into seven-turn conversations across three harnesses and two backbones.

Critical strengths
  • Strong problem framing. This paper looks beyond what the user wants immediately, and handles a second decision ie. what to do with the work already produced in the stream.

  • Making branching a by-product of base selection rather than a separate action is a strong design. It removes a decision the model would get wrong and keeps the graph shape consistent with requirement history.

  • Training only the small coordination policy while freezing the executor and harness is cheap, portable, and it avoids the usual confound where a framework's gains are caused by the base model improving.

  • Broad experimental surface. They look. at five domains, three harnesses, and two backbones.

Critical weaknesses
  • The benchmark is built so the method's preferred behaviour is the winning behaviour. Appendix A keeps a trajectory only if the final requirement is semantically equivalent to the original task, and Equation 2 scores only the final artifact against the original verifier. A system that solves the source task once and returns it unchanged every turn scores perfectly. Figure 8 then shows exact reuse covers about 63 percent of Code turns. Real requirement drift ends somewhere new; this one always comes home.

  • There is weaker evidence for the RL experiment. GITHARNESS-RL is worse than the training-free version in 26 of 30 cells and better in 4. The stated explanation is the smaller 8B policy, but the Git Agent is never trained at 32B, so capacity and training are confounded and we cannot tell whether Harness RL helps at all. The only clean evidence is Figure 4a, three domains on one harness, absent from the main table. Forty updates and 1,280 episodes is also very little, with a hand-picked checkpoint and no seeds.

  • The ablation suggests the Git part is not necesarily where the gains come from. Under StackPlanner with DeepSeek, Math goes Restart 76.0, no-ReuseAndPatch 92.0, full 94.0. Both middle and left solve fresh from a reconstructed requirement, so roughly 89 percent of the Math gain is requirement reconstruction and about 2 points is the entire versioning machinery. Search is similar (reuse worth about 5 of 21). Code and Research are the reverse. This belongs in the main paper, and it means a baseline is missing: your requirement resolution feeding a plain fresh solve.

Questions for authors

Can you report results on a split where the terminal requirement is deliberately not equivalent to the source anchor, with a verifier written for the new end state? How much of the reported gain survives when the conversation does not come home?

5 comments
AuthorsOpenReview ↗

Rebuttal for W1&Q1

W1 / Q1: Final-anchor construction and intermediate requirement changes

  1. Thank you for raising this issue. We agree that terminal scores alone are insufficient to determine whether a system follows intermediate requirements. Our benchmark follows the retrospective final-anchor construction of [1], which allows reuse of established ground truth and verifiers. In most intermediate turns, the full requirement differs from the original query and requires corresponding additions, modifications, or withdrawals. We therefore add a turn-level evaluation on Research to assess whether these changing requirements are fulfilled. Each turn's rubric is constructed from the requirements active up to that turn; simply keeping the original answer unchanged does not automatically yield high scores. Research report requirements can naturally be decomposed into verifiable atomic items. We therefore use Qwen 3.8 Flash as the judge to score each item, measuring retained-requirement satisfaction, new/modified-requirement satisfaction, and all-current-requirement satisfaction.
MethodRetained-requirement satisfactionNew/modified-requirement satisfactionAll-current-requirement satisfaction
GitHarness0.9900.9600.984
Native0.8850.9380.896
Restart0.8380.6320.797

GitHarness achieves 0.990, 0.960, and 0.984 on these three dimensions, respectively, exceeding Native and Restart. These results support its ability to preserve active requirements and implement requirement changes when intermediate objectives depart from the original task, complementing evaluation using only a terminal verifier.

  1. The 62.04% ExactReuse rate in Code reflects revisits to historical requirements and reuse of existing states. When the full current requirement matches an accepted historical version, directly returning its checkpoint is correct behavior. GitHarness also achieves consistent improvements on Math, SQL, and Research, where ExactReuse is less frequent and FreshSolve or ReuseAndPatch accounts for most execution. This indicates that the gains do not depend on the high ExactReuse rate in Code.

  2. The added turn-level evaluation assesses intermediate objectives that differ from the source task; non-returning terminal requirements remain outside the scope of the current evaluation. We will clarify this distinction in the revision.

[1] Tack J, Laban P, Neville J. LLMs Get Lost in Evolving User Intent[J]. arXiv preprint arXiv:2607.20734, 2026.

AuthorsOpenReview ↗

Rebuttal for W2

W2: Controlled RL comparison and training protocol

Thank you for raising this issue. We agree that the effect of RL should be measured through a same-capacity Base/RL comparison. We therefore add controlled comparisons under StackPlanner and TCRAG: during evaluation, we fix the Qwen3-32B downstream executor, prompts, tools, execution budget, data splits, and verifier, and replace only the Git Agent policy, comparing Qwen3-8B Base, Qwen3-8B RL, and training-free Qwen3-32B. The additional RL training continues to use DeepSeek-V4-Flash as the fixed downstream executor and updates only the Qwen3-8B Git Agent. The Git Agent supports any compatible model; the original choice of an 8B policy was due to training compute constraints.

StackPlanner

Git Agent policyMathSQLSearchCodeResearch
Qwen3-32B, training-free86.0 ± 1.072.0 ± 1.015.0 ± 1.017.0 ± 1.038.84 ± 0.9
Qwen3-8B Base82.0 ± 1.769.0 ± 1.713.0 ± 1.015.0 ± 1.036.52 ± 1.0
Qwen3-8B RL88.0 ± 1.074.0 ± 1.017.0 ± 1.020.0 ± 1.041.03 ± 0.8

TCRAG

Git Agent policyMathSQLSearchCodeResearch
Qwen3-32B, training-free84.0 ± 1.770.0 ± 1.014.0 ± 1.016.0 ± 1.037.52 ± 1.0
Qwen3-8B Base80.0 ± 1.767.0 ± 1.712.0 ± 1.014.0 ± 1.035.40 ± 1.1
Qwen3-8B RL86.0 ± 1.072.0 ± 1.016.0 ± 1.018.0 ± 1.039.70 ± 0.9

Under StackPlanner, Qwen3-8B RL improves over the same-capacity Base policy by 6.0, 5.0, 4.0, 5.0, and 4.51 points across the five domains; under TCRAG, the gains are 6.0, 5.0, 4.0, 4.0, and 4.30 points. This shows consistent RL gains across the two harnesses. Unlike the original experiment with 40 updates and 1,280 episodes, each RL seed in this additional experiment is trained for 100 updates with 3,200 sampled episodes. The three independent training seeds are 42, 43, and 44.

We evaluate on a fixed validation set every 20 updates and select the checkpoint with the highest validation objective for each seed. The selected checkpoints are then evaluated on a test set that was not used for training or hyperparameter selection. Training-free Qwen3-32B and Qwen3-8B Base are also rerun with inference seeds 42, 43, and 44. All table entries report mean ± standard deviation across three independent runs. The training-free and Base results in this table come from these new three-seed runs and are reported separately from the original main-table results. We will add this controlled comparison to the main text and report per-seed results and checkpoint-selection records in the appendix.

AuthorsOpenReview ↗

Rebuttal for W3

W3: Requirement resolution versus work-state reuse

Thank you for pointing out that the component contributions have not been sufficiently separated. We agree that requirement resolution itself is an important source of improvement and should be evaluated separately from work-state reuse. To directly address the missing baseline, we add a Same Resolver + FreshSolve control. It retains the same requirement-resolution module and requirement-version history as Full GitHarness, while disabling both ExactReuse and ReuseAndPatch. At each turn, the resolved full requirement is passed to the downstream agent, which solves it in an empty workspace and a fresh execution session, without restoring historical execution states or artifacts. All three configurations use the same downstream model, tools, and execution budget. Results under StackPlanner with DeepSeek are shown below.

MethodMath Acc.SQL EXSearch EMCode Res.Research RACE
Restart76.072.045.068.046.83
Same Resolver + FreshSolve87.074.051.069.047.52
Full GitHarness94.078.058.074.049.96

Relative to Restart, Same Resolver + FreshSolve improves the five domains by 11.0, 2.0, 6.0, 1.0, and 0.69 points. With the same requirement-resolution mechanism, allowing historical work-state reuse in the full system adds 7.0, 4.0, 7.0, 5.0, and 2.44 points, respectively. Thus, requirement resolution contributes substantially, but does not fully explain the performance of the complete system.

The existing w/o ReuseAndPatch ablation disables only the execution mode that continues modifying a historical work state. It retains ExactReuse and can directly return a historical checkpoint that satisfies the current requirement. The difference between 92.0 and 94.0 on Math therefore reflects the incremental contribution of ReuseAndPatch with the other components retained, rather than the contribution of the entire work-state reuse mechanism. The new control above directly measures performance with both reuse modes disabled.

The distribution of gains differs across domains: requirement resolution provides the larger improvement on Math; the two increments are similar on Search; and the additional gains from work-state reuse are larger on SQL, Code, and Research. We will add this control and the analysis of domain differences to the main text, distinguishing the roles of requirement resolution and work-state reuse.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewer Comments

We sincerely thank the Area Chair for overseeing the review process and all reviewers for their thoughtful feedback and constructive discussion. We have strengthened the experimental design, analysis, and presentation, and incorporated the principal additions into the revised manuscript. Below, we summarize the major revisions and additional experiments addressing the reviewers' comments.

Reviewer ACXQ

We thank Reviewer ACXQ for the constructive follow-up and recognition of the added turn-level evaluation, RL experiments, and ablations. We appreciate the reviewer's reconsideration of the paper in light of this evidence.

  • W1 & Q1: Intermediate requirement changes: We added turn-level Research evaluation against the requirements active at each turn. GitHarness achieves 0.990 retained-requirement satisfaction, 0.960 new/modified-requirement satisfaction, and 0.984 all-current-requirement satisfaction, exceeding Native and Restart.
  • W2: Capacity-matched RL evaluation: We added controlled Qwen3-8B Base/RL comparisons under StackPlanner and TCRAG with the downstream executor fixed. RL improves all five domains under both harnesses, including Research gains of 4.51 and 4.30 points. We report three independent training seeds, validation-based checkpoint selection, and per-seed test results.
  • W3: Requirement resolution versus work-state reuse: We added Same Resolver + FreshSolve, retaining requirement resolution while disabling both ExactReuse and ReuseAndPatch. Full GitHarness improves on this control across all five domains, separating the contributions of requirement resolution and historical work-state reuse.

Reviewer ca9Y

We thank Reviewer ca9Y for the follow-up confirming that the additional experiments and discussions adequately addressed their concerns. The suggestions helped us clarify the differences from GCC and provide more direct evidence of artifact preservation.

  • W1 & Q1: Methodological differences from GCC: We clarified requirement–checkpoint binding, requirement-conditioned historical selection, and adaptive execution. Adding requirement-aware selection to GCC improves all five domains. Complementary ablations examine the contributions of ExactReuse, ReuseAndPatch, and FreshSolve.
  • W2 & Q2: Preserving unaffected artifacts: We added section-level before/after comparisons showing that the original three theoretical chapters are retained verbatim while the requested non-Western modernization analysis and related synthesis are added. Additional GCC and GitHarness examples illustrate requirement-preservation behavior. The turn-level evaluation provides complementary evidence of whether still-active requirements remain satisfied.

Reviewer yj7r

We thank Reviewer yj7r for the continued discussion and acknowledgment of the added Claude Code comparison. The follow-up helped us clarify how preserving still-active requirements complements implementing new requests, and how these two aspects jointly characterize successful updates under evolving user requirements.

  • W1 & Q1: Harnesses with native memory and checkpointing: We added a Claude Code baseline with adaptive checkpoint rollback. GitHarness improves performance across all five domains, complementing the existing TCRAG, StackPlanner, and OpenHands comparisons.
  • W2 & Q1: Requirement-change styles: We added a breakdown across all nine operators. GitHarness improves retained- and all-current-requirement satisfaction for every operator; new/modified-requirement satisfaction improves for seven, with small decreases for Comparison and Pivot. We clarified that the larger gains concern maintaining still-active requirements while implementing updates, rather than uniformly improving new-requirement satisfaction.
  • Q2: Dataset construction: We expanded the generation and inspection protocol, explained the seven-turn setting, and added operator proportions and a complete conversation example. We also clarified that seven turns is the main evaluation protocol, not a framework limit; the 10–30-turn experiments examine longer, interleaved interactions.

Concluding Statement

Taken together, the added controlled experiments, turn-level evaluations, and artifact comparisons provide complementary evidence for GitHarness's contribution: tracking evolving requirements and coordinating the preservation and update of accumulated work. We hope this summary assists the Area Chair and reviewers in assessing the revised manuscript. We thank them again for their time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.