- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ArbiGraph, a framework for generating benchmark instances that compose arbitrary verifiable tasks into DAG-structured workflows for agent evaluation. The core mechanism is answer-gated input access: to obtain a task's input, the agent must report the answers of its parent tasks via a get_input tool; only if all reported answers are correct does the tool return the intended input, otherwise it returns an alternative valid input whose verified answer differs. This design allows heterogeneous tasks to be composed without I/O type matching or hand-written adapters, while keeping every node's intermediate answer independently verifiable, preventing conversions (e.g., rounding) from masking errors, and preserving difficulty control. The framework also exposes an RLVR interface (no training is performed).
Critical strengths
It decouples task composability from I/O type compatibility, replacing adapters with report checking plus alternative inputs whose verified answers differ from the intended one. This enables tasks from different domains to be composed into arbitrary DAG topologies while retaining per-node verifiability and error traceability. It also avoids the error-masking problem of conversion-based composition, where rounding, for example, can map a wrong parent answer to the same child input; the paper shows this to be a real flaw in prior approaches such as RACES and Composition-RL. This point is consequential to my assessment because the community currently lacks a way to vary dependency structure while holding the difficulty of individual problems fixed. Combined with per-node verification records and an RLVR interface, the mechanism has the potential to become shared infrastructure rather than a one-off benchmark.
The paired designs (the same problems are reused across topologies, and the shuffled braid changes only task numbering, not problems or edges), the bootstrap intervals for the independent-error baseline, the Wilson intervals for measured accuracy, and the per-node failure localization together reflect above-average statistical and experimental discipline. In particular, the observation that most failures begin at a node that received its intended input, and that even root-node accuracy degrades once a problem is placed inside a larger workflow, extends the self-conditioning and long-horizon degradation results of Sinha et al. into the compositional-task setting. This constitutes substantive evidence supporting acceptance, rather than merely constituting yet another benchmark.
Critical weaknesses
Root-node accuracy degrades even at nodes with no dependencies whatsoever, and topology (edge structure) has no consistent effect on target accuracy. Together these strongly suggest that a large fraction of the headline loss may stem from generic interference caused by presenting many problems in one prompt within a long context, rather than from dependency composition itself. The paper lacks a REST-style control, in which the same tasks are presented in a single prompt, solvable independently, and without get_input gating, that would decompose these two effects. This is consequential: if most of the loss does not come from dependency structure, the paper's central empirical claim reduces to a reconfirmation of known long-context and multi-task degradation phenomena in the line of REST and RULER, leaving the engineering framework as the main contribution. The Limitations section acknowledges this gap but provides no ablation.
(a) Only three mid-size open-weight models are tested, and the strongest model loses the least by a wide margin, which suggests the phenomenon may fade rapidly with model capability; the absence of frontier models leaves the implicit claim that this benchmark will remain discriminative for the next generation of models entirely untested. (b) Tasks are filtered to those each model solves perfectly in isolation, so the results characterize only the easy-task regime; the mixed-category experiment reuses a handful of RLVE and HotpotQA tasks across all instances, offering little diversity. (c) The RLVR interface is listed as a contribution but is never validated by any training experiment. (d) The alternative-input mechanism silently swaps inputs when a report is wrong, so the agent is never told its report was incorrect and continues solving on the wrong branch; unlike real workflows where failed tool calls return error feedback, this design may artificially amplify error propagation, and the paper does not analyze sensitivity to this choice.
Questions for authors
Q1 Could you add a no-dependency control, in which the same tasks from each instance are presented jointly in the prompt, solvable independently, and without get_input gating, to decompose how much of the additional loss comes from dependency structure versus from the multi-task context itself? This result directly determines whether the paper's central finding is incremental over prior work such as REST or a replication of known phenomena.
Q2 When a report is wrong, get_input silently returns an alternative input, giving the agent no signal that its report was incorrect. How sensitive are the results to this design choice? If the tool instead returned explicit error feedback, allowing corrections within the existing per-node request limit, would the additional loss remain significant? This bears on whether the framework measures compositional ability under realistic feedback or error propagation in a no-feedback setting.
Q3 The task-selection filter retains only tasks that each model solves on all screened inputs"which systematically excludes tasks at the boundary of each model's ability. Does the direction and magnitude of the additional loss hold on an unfiltered or relaxed task pool? If the conclusions hold only for tasks that are near-ceiling in isolation, the claims in the main text should be explicitly scoped to that regime.