- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper proposes making agents operate like Git to handle multiple requirements over long-running user sessions. It targets real-world user journeys where requirements evolve, either revealed gradually or modified along the way.
- This is done in two main steps. First, a Git Agent performs requirement resolution to work out the full current requirement and pick the best base version to resume from. Then, in branch execution, an update agent restores that version and carries out the requested changes. If a past version already satisfies the request, it’s reused directly.
- The Git Agent can also be trained with RL while the downstream harness and executor stay frozen. The reward combines final-task correctness with an LLM judge’s score for how well the agent tracks the user’s requirements.
Critical strengths
- The paper is well motivated: most benchmarks don’t effectively measure how agents perform in an evolving, user-in-the-loop setting, where requirements are added, refined, or revised as the user inspects intermediate results. Framing this as deciding where to resume and what to update is intuitive. The Git-style abstraction is a natural fit, and because it plugs into existing harnesses without changing them, it’s easy to adopt across different agent setups.
- The results are consistent across 5 domains, 2 backbones, and 3 harnesses: GitHarness beats native continuation in all 30 settings while cutting token usage substantially.
Critical weaknesses
The benchmark does a good job illustrating requirement completion and elicitation, but for requirement shift the scope feels limited to cases where a clean revert is possible. Every conversation has to end back at the original task, so shifts mostly undo earlier temporary edits, which directly rewards rolling back to a saved version (Exact Reuse dominates on Code). Two things would make the results more convincing:
- How does GitHarness compare with widely used harnesses that already have built-in memory, compaction, or checkpointing? The current comparisons are mostly against research methods (U-Fold, GCC).
- How does it perform under different styles of requirement change? Results are reported only in aggregate, so a breakdown by change type, plus shifts that lead to genuinely new requirements, would show where the gains actually come from.
Questions for authors
- The benchmark spans completion, elicitation, and shift, but results are reported only in aggregate. How does GitHarness perform on each change type, and on each operator? Is the gain concentrated in cases where the final request matches an earlier version (Exact Reuse dominates Code)? Results on conversations whose final requirement differs from every earlier state, and comparisons with widely used harnesses that have their own memory, compaction, or checkpointing, would show whether the gains generalize.
- How were the 7-turn extensions of the benchmarks in section 5.1 generated: which model was used, why seven turns, and how were operators and anchors chosen? What did the human inspection involve, and how many conversations were rejected? An example conversation and the distribution of operators would help to judge how realistic the requirement changes are.
5 comments
Rebuttal for W2&Q1
W2 / Q1: Breakdown by requirement-change operator
We agree that aggregate scores can obscure which requirement changes account for the gains. To provide results by change type, we add a turn-level atomic-requirement evaluation on Research. We decompose and score all active requirements based on user requests up to the current turn, and group the results by the nine requirement-change operators defined in the paper. GitHarness and Native use exactly the same turn-level rubrics and judge. We report retained-requirement satisfaction, new/modified-requirement satisfaction, and all-current-requirement satisfaction side by side.
| Operator | Retained-requirement satisfaction (GitHarness) | Retained-requirement satisfaction (Native) | New/modified-requirement satisfaction (GitHarness) | New/modified-requirement satisfaction (Native) | All-current-requirement satisfaction (GitHarness) | All-current-requirement satisfaction (Native) |
|---|---|---|---|---|---|---|
| Reveal | 0.992 | 0.901 | 0.967 | 0.955 | 0.988 | 0.908 |
| OutputControl | 0.994 | 0.919 | 0.974 | 0.970 | 0.991 | 0.926 |
| Refine | 0.990 | 0.904 | 0.960 | 0.958 | 0.986 | 0.912 |
| Constrain | 0.989 | 0.885 | 0.949 | 0.946 | 0.982 | 0.896 |
| Comparison | 0.987 | 0.892 | 0.944 | 0.951 | 0.979 | 0.904 |
| Pivot | 0.986 | 0.871 | 0.934 | 0.936 | 0.974 | 0.885 |
| ParameterUpdate | 0.984 | 0.878 | 0.953 | 0.943 | 0.978 | 0.890 |
| Correct | 0.986 | 0.884 | 0.958 | 0.948 | 0.981 | 0.896 |
| Withdraw | 0.995 | 0.826 | 0.970 | 0.907 | 0.986 | 0.855 |
GitHarness outperforms Native on retained-requirement satisfaction and all-current-requirement satisfaction for all nine operators. New/modified-requirement satisfaction is higher for seven operators and slightly lower than Native for Comparison and Pivot. The gains in all-current-requirement satisfaction range from 0.065 to 0.131 across operators. For the Shift operators ParameterUpdate and Correct, GitHarness achieves all-current-requirement satisfaction of 0.978 and 0.981, compared with Native's 0.890 and 0.896. For Withdraw, the scores are 0.986 and 0.855. These turn-level results show that GitHarness’s gains extend across requirement completion, elicitation, and shift, including parameter updates, premise corrections, and withdrawals.
Rebuttal for Q2
Q2: Seven-turn construction and human inspection
- Thank you for this suggestion. For dataset construction, we use DeepSeek-V4-Flash to generate candidate conversations across all five domains. The process has three steps. First, we design several seven-turn operator templates for each domain. Each template specifies the requirement changes to cover and their necessary dependencies. A fixed random seed is used to select a compatible template for each source task. Second, we extract the goals, parameters, constraints, and output format from the original task, identify the slots each operator can modify, and use DeepSeek-V4-Flash to instantiate these structured edits as natural-language conversations. For example, ParameterUpdate changes numerical or entity parameters, OutputControl changes the output form, and Comparison adds a comparison object at the same level. A candidate is rejected if the task has no valid slot for the edit. Finally, for intermediate changes without ordering dependencies, we use a fixed random seed to vary their order and increase trajectory diversity. We also check dependencies such as introducing a requirement before withdrawing it and eventually restoring the original task, filtering out trajectories with semantic contradictions or inconsistent state transitions.
In addition, the authors conduct human inspection with LLM assistance, primarily checking:
- Whether the conversation is natural and reasonable.
- Whether each turn fully expresses its requirement edit.
- Whether the operator matches the text.
- Whether withdrawal relationships are consistent.
- Whether the terminal state restores the source requirements.
- Whether answers or evaluator information are leaked.
The candidate-construction success rate is 65.4%, and the rejection rate due to semantic or structural problems is 34.6%. To illustrate requirement evolution in a seven-turn conversation, we will add a complete example to the revised PDF, showing the user utterance, operator, modified requirement slot, and active requirements at each turn. We also report the overall proportions of the nine operators across the five domains.
| Operator | Reveal | OutputControl | Refine | Constrain | Comparison | Pivot | ParameterUpdate | Correct | Withdraw |
|---|---|---|---|---|---|---|---|---|---|
| Proportion | 20.1% | 11.1% | 7.2% | 10.7% | 8.8% | 6.3% | 7.5% | 11.3% | 17.0% |
- We use seven turns in the main table to remain consistent with the main experimental setting of [1]. That work applies argument reveal, argument revision, and function switch twice each, producing six intent transitions and, together with the initial request, up to seven user turns. We extend these three transitions to nine finer-grained requirement operators: Completion includes Reveal, OutputControl, and Refine; Elicitation includes Constrain, Comparison, and Pivot; and Shift includes ParameterUpdate, Correct, and Withdraw. Seven turns is the standardized experimental protocol for the main table, not a length limit of the construction method or GitHarness.
In addition to the seven-turn setting, we evaluate interleaved multi-task conversations with 10, 15, 20, 25, and 30 turns. The results are reported in Table 4, where H denotes the number of user turns.
| Domain | Method | H=10 | H=15 | H=20 | H=25 | H=30 |
|---|---|---|---|---|---|---|
| Math | Native | 73.0 | 72.0 | 69.0 | 63.0 | 62.0 |
| Math | Restart | 71.0 | 71.0 | 67.0 | 61.0 | 59.0 |
| Math | U-Fold | 82.0 | 83.0 | 81.0 | 79.0 | 78.0 |
| Math | GitHarness | 92.0 | 90.0 | 87.0 | 85.0 | 84.0 |
| Research | Native | 42.92 | 41.57 | 37.86 | 35.39 | 34.53 |
| Research | Restart | 41.05 | 40.08 | 39.14 | 35.88 | 27.70 |
| Research | U-Fold | 37.53 | 37.13 | 35.41 | 33.52 | 34.16 |
| Research | GitHarness | 49.53 | 49.27 | 48.71 | 47.52 | 47.32 |
GitHarness achieves the best results at every horizon in both domains. At 30 turns, Math reaches 84.0, exceeding the strongest baseline by 6.0 points, and Research reaches 47.32, a gain of 12.79 points. These results support its effectiveness in longer interactions.
Rebuttal for W1&Q1
hank you for this suggestion. Our experiments already cover commonly used harnesses with native state-management capabilities, beyond U-Fold and GCC. TCRAG maintains an explicit message stack:
Backtrackpops the most recentThoughtorAction–Observationpath and pushes the feedback back onto the stack. StackPlanner's CentralAgent usesReflectto clean its memory stack andSummarizeto write compressed information back to the cross-turnMemoryStack. Claude Code automatically creates file checkpoints. To construct a stronger Claude Code baseline, we use prompt-based instructions and hooks so that Claude Code determines whether rollback is needed, while a hook selects the target checkpoint and restores the corresponding context. On top of these native or enhanced state mechanisms, GitHarness uses an independent Git Agent to explicitly reconstruct the full current requirements, maintain the correspondence between accepted requirement versions and work states, and jointly select a historical base and the ExactReuse, ReuseAndPatch, or FreshSolve execution mode. Within each harness, paired methods use the same model, tasks, tools, budget, and verifier. The DeepSeek-V4-Flash results are as follows.GitHarness improves all 20 paired cells across four harnesses and five domains. The gains are 21–30 points on Search and 15–19 points on Math; even on Code and Research, where the baselines are stronger, the gains remain 6–8 and 2.33–4.11 points, respectively. These gains occur on top of different state mechanisms, including message stacks, summary-based memory, and adaptive checkpoint rollback, supporting the additional value of requirement-conditioned historical-state selection and execution beyond the existing native state-management capabilities.