- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CARTA, a framework for synthesizing structurally controlled deep-search tasks using relational contracts. CARTA introduces three graph construction operations, EXTEND, BRANCH, and COUPLE, to compose relational structures involving depth, branching, and cross-path coupling. The resulting graphs are compiled into explicit relational contracts that specify relation obligations and entity identity constraints. These contracts guide question realization, structural verification, targeted revision, and teacher trajectory selection.
The authors evaluate CARTA using supervised fine-tuning on Qwen3.5-4B and Qwen3.5-9B across three deep-search benchmarks: BrowseComp, BrowseComp-ZH, and xbench-DeepSearch-2510. Under matched training conditions, CARTA consistently outperforms REDSearcher and OpenSeeker-v1. Additional experiments investigate structural composition, realization fidelity, trajectory selection, and the reliability of generated tasks.
Critical strengths
S1. A coherent framework for structure-aware task synthesis. The paper addresses an important limitation of existing deep-search data synthesis approaches: generating longer reasoning chains does not necessarily ensure relational diversity or consistency. The proposed EXTEND, BRANCH, and COUPLE operations, together with explicit relational contracts, provide a systematic way to construct and verify compositional search tasks. The integration of construction, realization, and trajectory selection is a meaningful contribution.
S2. Strong empirical results with useful controlled analyses. CARTA consistently outperforms the evaluated baselines across three benchmarks and two model scales. The authors also provide stage-specific ablations, including matched-source-graph realization comparisons and trajectory selection from answer-correct rollouts. These experiments go beyond aggregate benchmark improvements and provide evidence for the utility of preserving relational structure during task synthesis.
Critical weaknesses
W1. The independent contribution of relational composition remains insufficiently isolated. Although the paper introduces three graph construction operations, the structural enrichment experiments vary depth, branching, and coupling jointly. Consequently, it is difficult to determine whether the improvements primarily arise from deeper reasoning chains, branching constraints, coupling structures, or their interactions. The current results support the overall framework but provide limited evidence for the individual necessity of each construction operation. This also makes it harder to assess the contribution beyond existing graph-based synthesis approaches.
W2. The reliability and efficiency of contract-based verification require further justification. The structural conformance evaluation relies substantially on LLM-based judges, and the authors acknowledge that the operational audit can miss certain unlicensed relations. While human calibration is provided, the calibration subsets are relatively small. Moreover, CARTA requires multiple model calls, retrieval operations, and iterative verification for each synthesized task. A more explicit analysis of generation cost, acceptance rate, and performance gains under matched computational budgets would strengthen the practical significance of the framework.
Questions for authors
Q1. Can the authors provide a more fine-grained ablation that independently varies depth, branching, and coupling while controlling the remaining structural properties? This would help clarify which relational patterns contribute most to downstream improvements.
Q2. How robust are the reported structural conformance improvements to the choice of LLM-based evaluator? Could the authors provide additional human validation or cross-judge evaluations, particularly for violations involving additional relations between already licensed entity pairs?
Q3. What is the end-to-end cost of generating 5,000 accepted CARTA tasks, including rejected candidates, verification, and revision? How does CARTA compare with the graph-to-query baseline when total generation cost is matched, rather than only the planning-and-revision budget?
Q4. How do the authors assess potential benchmark contamination or near-duplicate overlap between synthesized tasks and evaluation benchmarks? Since the current prevention mechanism primarily excludes benchmark-related search terms and domains, would an additional post-generation overlap audit affect the reported results?
7 comments
Response to W2, Q2 & Q3: Reliability and Efficiency of Contract-Based Verification
We provide additional cross-judge evaluations, independent human validation, targeted error-detection experiments, and synthesis cost analyses to address the reviewer's concerns.
1. Reliability of Structural Conformance Evaluation (W2 & Q2)
We appreciate the concern regarding LLM-based evaluation, particularly the possibility of missing additional relations between already licensed entity pairs. We conducted three complementary evaluations to directly address this issue.
(1) Strict relation-level evaluation with cross-family judges.
We re-evaluated the same 480 outputs from 240 paired source graphs, without regeneration or filtering, using a stricter relation-level protocol. Unlike the original pair-level addition check, the new protocol examines every expressed relation and flags unlicensed predicates even between already licensed entity pairs, while allowing equivalent paraphrases and valid inverse relations.
| Judge | Graph-to-Query | CARTA | Improvement |
|---|---|---|---|
| GPT-4o | 70.8% | 83.3% | +12.5 pp |
| Gemini-3-Flash-Preview | 69.6% | 82.9% | +13.3 pp |
Although the stricter protocol lowers absolute conformance relative to the original 72.9%/86.3%, CARTA retains a substantial 12.5–13.3 pp advantage across both evaluator families. This directly addresses the identified blind spot and shows that the structural-fidelity improvement is not an artifact of the original pair-level check.
(2) Independent human validation.
We additionally randomly sampled 60 source graphs and both methods' outputs (120 questions) for independent blind annotation by two annotators. Their inter-annotator agreement reached Cohen's κ = 0.87 before adjudication.
Against the adjudicated human labels, GPT-4o and Gemini achieved 94.2% and 95.0% overall agreement, respectively, with violation-detection rates of 88.9% and 92.6%. We also report Wilson 95% confidence intervals to reflect uncertainty, particularly for the 27 human-confirmed violations.
(3) Targeted minimal-edit evaluation.
We constructed 180 examples from 60 human-confirmed conforming questions, including original questions, versions with one added unlicensed predicate, and valid paraphrases. GPT-4o and Gemini detected added predicates at rates of 91.7% and 93.3%, respectively, while maintaining low false-rejection rates (1.7–5.0% across the original and paraphrase conditions).
Together, these results demonstrate that CARTA's structural-fidelity advantage remains robust under stricter relation-level verification, cross-family evaluation, and independent human calibration. We will include the full protocols and uncertainty estimates in the revision, while continuing to characterize automated conformance as an estimate rather than a formal guarantee.
2. End-to-End Generation Cost and Efficiency (W2 & Q3)
We provide additional accounting covering rejected candidates, model calls, tool calls, and token consumption.
(1) Full-scale generation cost.
Producing 5,000 accepted CARTA tasks requires approximately 5,556 candidate attempts (90% acceptance rate), including 556 rejected or failed attempts.
| Metric | Total |
|---|---|
| Candidate attempts | 5,556 |
| LLM calls | 157,222 |
| Tool calls | 100,789 |
Including unsuccessful attempts, this corresponds to approximately 31.4 LLM calls and 20.2 tool calls per accepted task. Mean synthesis runtime is approximately four minutes per task. Teacher trajectory generation and SFT are excluded from these synthesis totals.
(2) Generation cost and structural-fidelity comparison.
We additionally re-tallied the original 240-graph realization experiment using total LLM calls and token consumption, rather than only planning-and-revision budgets.
| Method | LLM calls | Tokens | Conformance |
|---|---|---|---|
| Graph-to-Query | 6,432 | 46.8M | 72.9% |
| CARTA | 6,792 | 52.1M | 86.3% |
CARTA improves whole-query structural conformance by 13.3 pp with only 5.6% more LLM calls and 11.3% more tokens.
Separately, the matched-source-graph SFT experiment (Table 2) shows downstream improvements of +6.1/+4.1/+6.6 pp with supervised tokens matched within 1%.
These results support the practical training value of contract-guided realization, delivering substantial structural-fidelity and downstream performance improvements with modest additional generation overhead.
We will incorporate these analyses into the revised paper. We hope the additional evidence addresses the reviewer's concerns and supports a more favorable assessment of our work.
Response to Q4: Benchmark Contamination and Near-Duplicate Overlap
Beyond the benchmark-related keyword and domain exclusions described in the paper, we conducted an additional post-generation overlap audit covering all 5,000 accepted CARTA tasks against the three evaluation benchmarks.
The audit combines normalized exact matching, cross-lingual semantic retrieval using BGE-M3 (top-5 candidates, cosine similarity above 0.80), and human verification. The overlap criteria consider the target answer, core relational conditions, and their composition rather than surface wording alone.
| Benchmark | Exact matches | Near-duplicate candidates | Confirmed overlap |
|---|---|---|---|
| BrowseComp | 0 | 6 | 0 |
| BrowseComp-ZH | 0 | 3 | 0 |
| xbench-2510 | 0 | 5 | 0 |
| Total | 0 | 14 | 0 |
Among the 14 candidates, 11 share only entities or topics but have different target answers. The remaining three share a target answer but differ in their essential relational conditions and composition.
No question-level overlap was confirmed, and no training examples required removal, leaving the reported results unchanged.
Response to W1 & Q1: Contributions of Relational Composition
Thank you for your constructive feedback on the contributions of depth, branching, and coupling. We provide additional experimental evidence to clarify the practical value of these relational structures.
1. Component-level Relation-Deletion Analysis
Removing BRANCH and COUPLE constraints increases alternative-answer discovery by 11.3 and 6.7 percentage points, respectively, demonstrating that these relations contribute meaningful answer-disambiguating constraints.
We conduct a within-task leave-one-relation-out analysis on accepted CARTA tasks. For each task, we remove one BRANCH or COUPLE clause while preserving the remaining applicable relations and identity constraints, with minimal grammatical adjustments. The same independent evidence-backed solver evaluates the original and modified questions under identical retrieval budgets.
Both removals significantly increase the discovery of alternative satisfying answers. These results demonstrate that BRANCH and COUPLE impose meaningful constraints on the answer space rather than merely increasing graph complexity.
2. Structural Composition and Downstream Training
The original Low/High/Mixed experiment provides complementary training-level evidence. Under matched training examples, source counts, trajectory lengths, tool calls, and supervised tokens, Mixed consistently outperforms both Low and High across all three benchmarks.
Mixed improves over Low by 5.8/6.4/7.4 percentage points and over High by 2.1/2.3/2.0 points, respectively. These results show that combining different structural composition levels provides more effective training supervision than either level alone.
3. Interpretation
Together, these experiments demonstrate the practical value of relational composition at both the task and training levels: BRANCH and COUPLE provide meaningful answer-disambiguating constraints, while diverse structural compositions improve downstream SFT performance.
Combined with the structural guarantees established by Lemma 1 and Proposition 1, these findings provide complementary theoretical and empirical support for CARTA's controllable relational composition framework.