- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
StateLedger selects historical code needed to rerun a given data-analysis computation. It tracks versions of variables and columns, follows their dependencies backwards, and replays the selected cells in their original order. The main evaluation measures whether this reduced history reproduces the output obtained using the full history.
Critical strengths
- Testing actual execution gives a concrete way to evaluate whether the selected history is sufficient.
- The authors include stronger slicing baselines and openly report negative results, including the lack of a clear gain in the fully online setting.
Critical weaknesses
-
The practical motivation is not established. The introduction describes accumulating analytical state and lists related benchmarks, but never clearly explains the workflow in which selective replay is needed. What makes keeping the existing execution state or rerunning the full history impractical? The connection between this problem and LLM memory is also weak: the main experiment assumes that the target code is already known. I understand the evaluation task, but I am not convinced why it is an important research problem in the setting studied.
-
The writing makes a fairly straightforward idea difficult to follow. Terms such as “versioned analytical state” “producer resolution” and “executable projection” appear before a concrete example establishes what is going wrong. The example in Section 4.4 should have appeared much earlier. The abstract is also fragmented into blocks and reads like a compressed experimental report, with too many numbers and qualifications before the reader understands the practical goal.
-
The methodological contribution over program slicing is unclear. Tracking variable versions and recursively recovering dependencies are familiar ideas. The paper itself acknowledges its connection to cell-level reaching-definition analysis, but does not clearly identify what the ledger representation adds. The closure ablation shows that indirect dependencies matter; it does not establish that the proposed representation is needed. Likewise, the replay guarantee assumes complete and faithful provenance, leaving the reliability of the actual extraction procedure unresolved.
-
The recent-three guard is poorly justified. It adds cells without necessarily recovering their dependencies. In Table 2, removing it improves replay accuracy from 74.5% to 80.0% while also reducing history size. This raises a basic question about the design of the default method.
-
The evaluation leaves the claimed practical benefit uncertain. Only 455 of 1050 CoCoNote candidates are retained, and their histories average about 1781 source characters. The filtering is method-independent, but it favors shorter histories and limits the evidence for long-history use. Reducing source characters alone also does not establish a worthwhile end-to-end saving once selection, extraction, and replay failures are considered.
Questions for authors
- What specific deployment scenario requires selective replay, and why are retaining execution state or replaying the full history inadequate there?
- What capability does StateLedger add beyond a version-aware, column-aware backward slicer under the same information constraints?
- Why is recent-three the default when it hurts both accuracy and history size on the main benchmark? How do the stronger baselines compare without this guard?
- Can you demonstrate a practical benefit on longer histories, including total overhead and downstream task success, rather than primarily selected source length?