- Rating
- 3(+1)
- Confidence
- 2(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RECON evaluates compositional reasoning over long investigative case files, with 1,604 questions from 24 cases in criminal, medical, and financial domains. Its six task families cover evidence-chain reconstruction, cascading invalidations, source conflicts, counterfactuals, temporal constraints, and temporal fact retrieval. A deterministic generator constructs the underlying evidence dependencies and answer keys before an LLM renders the narrative. The paper compares long-context models, RAG pipelines, and memory systems, together with a structured-graph Oracle condition. On the 1,414-question filtered evaluation set, the best non-Oracle accuracy is 22.4%, compared with 54.6% for the Oracle. Validation includes structural checks, a 200-question human audit, and a 100-response audit of free-form grading.
Critical strengths
-
The focus on how revisions to evidence affect downstream conclusions is useful. Tracking invalidation and independent support goes beyond retrieving isolated facts, and provides a concrete target for evaluating memory systems.
-
Constructing evidence graphs and answer keys before generating prose is a sensible design choice. It makes the intended dependencies inspectable and supports targeted diagnostics. The comparison across long-context, retrieval, and memory systems, together with human and grading audits, provides a useful initial characterization.
Critical weaknesses
-
I am not convinced that the dataset's broader usefulness is sufficiently established. Why were only criminal, medical, and financial investigations selected? These domains share a similar investigation structure. Low model accuracy shows difficulty, it may be better to measure memory capabilities relevant beyond this synthetic setting.
-
I remain concerned about label reliability. In the 200-question audit, human majority judgments contradicted gold on 8 items and were unsure on 19. How these disagreements and reasonable alternative answers were independently adjudicated is unclear to me. I also find the human–model comparison difficult to interpret: humans received selected evidence packets, whereas long-context models received full case files. A matched-input comparison would be more informative.
-
Limited variability across cases. The 1,604 questions come from only 24 shared case contexts, so aggregate accuracy may obscure substantial case-level differences. Reporting per-case results would help establish how consistent the findings are.
As an additional presentation clarification, I would like the authors to explain whether Figures 2, 3, and 5 used generative AI and how their text, arrows, and correspondence to the actual pipeline were checked.
Questions for authors
See weaknesses.
We have uploaded a revised PDF that incorporates the changes described in our response, marked in blue. We would be grateful to hear whether our response addresses your concerns, and we are happy to clarify anything further.