- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ITG which jointly generates an answer and auxiliary facts then applies human-authored rules to correct inconsistencies without updating model weights. Experiments across contracts, charts, arithmetic, and driving scenes report substantial gains in selected settings, particularly chart medians, but mixed performance against CoT and occasional harmful corrections.
Critical strengths
-
Substantial chart gains: On FigureQA, Llama-4-Scout’s upper-median accuracy increases from 51.8% with Base+A to 96.2% with ITG (Table 8). This is useful evidence that deterministic computation can recover information the model extracts but fails to use correctly.
-
Inspectable enforcement: Fixed schemas and correction functions make the decision process explicit and reusable without retraining. This provides practical value for narrowly specified tasks where the rules are valid and can be independently audited.
Critical weaknesses
-
Limited established contribution: For chart tasks, the corrected answer is effectively a deterministic function of extracted values; the benefit of additionally predicting and reconciling an answer is unestablished. GSM8K still executes model-generated expressions, weakening the claimed distinction from PAL. The experiments do not establish a substantial advance over extraction followed by conventional computation.
-
Unreliable enforcement and validation: Some “invariants” are unjustified implications, such as third-party permission implying employee permission (Eq. 4). Separately, GPT-4o’s NDA-5 accuracy falls from 91.3% with Base+A to 78.7% with ITG (Table 5), but FCR is only 7.97% (Figure 3). A 12.6-point decline cannot arise from that harmful-correction rate on the same paired predictions.
Questions for authors
-
Evaluation: Can you reconcile the accuracy/FCR contradiction above and provide the underlying paired counts? Additionally, Table 9 contains 325 decisions per run, whereas §4.1 describes 123 contracts × five hypotheses, implying 615 decisions.
-
Rule validity: What justifies third-party permission implying employee permission, or absent brake lights implying a vehicle is not stopped? Are additional conditions omitted from the equations, and how are conflicting rules ordered? These assumptions can fail even with correct extraction.
-
Baseline failures: What fractions of PAL/SatLM failures arise from formatting, execution, or incorrect reasoning? SatLM’s roughly 1–5% contract accuracies and PAL’s roughly 10% chart accuracies require explanation. Where are the GSM8K PAL/SatLM results promised in §4.3?
-
Incremental benefit : Does predicting and reconciling H improve over extracting only A and applying the same deterministic functions? An equal-budget comparison, including input/output tokens and retries, would materially affect whether ITG offers an advantage beyond a conventional extraction-and-computation pipeline.