- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies factual errors and path-dependence in LLM-generated long-form descriptions of the world. Using a city-walk benchmark of pedestrian corridors, it shows that models often make incorrect landmark-level factual commitments during free-form narration even when they can often identify the same errors when later asked to audit them. The paper’s main contributions are a large-scale claim-level benchmark with a grounded evaluation pipeline and a three-parameter model of critique-and-revise loops, (p,q,r), used to analyze repair, corruption, and the resulting error floor under repeated revision. Validation relies on external map data, geometric relationships, a grounded automated evaluator with evaluator checks and a human audit, as well as forward/reverse traversal consistency analyses. These experiments provide substantial evidence for the main claims, though in my view some central revision-dynamics conclusions would benefit from stronger robustness analysis across prompting variants.
Critical strengths
The paper introduces a valuable city-walk benchmark that supports large-scale, claim-level, map-grounded analysis. Combined with the grounded evaluation pipeline and the breadth of the city and model comparisons, this enables multidimensional analysis of factual errors across models, cities, and revision rounds, making the empirical phenomena statistically convincing. The paper also provides a clear analytic framework for studying revision dynamics. In particular, the three-parameter model of critique-and-revise loops, (p,q,r), offers an interpretable way to separate repair from corruption and to explain why repeated self-revision exhibits a nonzero error floor. More broadly, the paper meaningfully distinguishes between possessing geographic knowledge and maintaining spatial consistency during long-form narration.
Critical weaknesses
The generation-side procedure has limited methodological novelty. The self-critique-and-revise loop is close to an existing critique-and-revise paradigm, and the paper’s main novelty seems to lie more in the benchmark construction, grounded claim-level evaluation pipeline, and the (p,q,r) analysis than in the revision mechanism itself. This matters for my overall recommendation because a substantial part of the paper’s headline empirical story is organized around this loop, yet the paper does not sufficiently establish what is methodologically new beyond applying a fairly standard paradigm in this domain. The main quantitative conclusions about revision dynamics and the estimated error-rate floor are established under a fairly specific prompt family, especially a full-rewrite revision setting. Different prompt formulations or rewrite instructions could plausibly change both corruption and repair behavior, and full rewrites may themselves introduce additional stochasticity by corrupting previously correct claims. Therefore, I would like to see stronger stability evidence across prompt variants, revision formulations (e.g., full rewrite versus more local editing), decoding settings, or repeated runs. Without such robustness analysis, it remains unclear how much of the reported floor is a general phenomenon of long-form spatial narration and how much is specific to the chosen prompting protocol.
Questions for authors
How sensitive are the fitted revision dynamics and the resulting error-rate floor to the prompting protocol? In particular, if you vary the critique/revision prompt, compare full-rewrite revision with more local editing, or rerun the same routes under multiple prompt variants or decoding settings, do the key qualitative conclusions remain stable—especially the presence of a nonzero corruption rate q and a persistent error floor? This would materially affect my assessment, because it would clarify whether the reported floor is a robust property of long-form spatial narration or is partly induced by the specific critique-and-rewrite setup used in the paper.
2 comments
Thank you for the additional clarifications. I appreciate the discussion of prompt variation and the theoretical analysis. However, my concern about novelty remains. I continue to view the primary contribution as the benchmark, empirical findings, and accompanying analysis, rather than a substantially new revision framework. I will therefore keep my original score unchanged.
Author Comment
We thank you for the review. We agree with you that having a model revise its own output is an existing family of frameworks, and that our contributions (as noted in the paper) are in the construction of the task and the evaluation, the empirical findings, and the theoretical analysis of the revision framework, which is new relative to prior work. This new analysis reveals things that were not a priori known and are fundamental, including the fact that deletion is the primary source of improvements in these revisions, and that performance on a given city can be predicted from properties of the city. This analysis also provides an optimal algorithm for scheduling revisions across a family of multiple revisers; this formal problem specification and the resulting algorithm are both new.
The evidence in the paper suggests that the phenomena are robust, in a number of respects. First, the prompt for the revision explicitly asks the model to maintain correct landmarks; we note that in the standard paradigm for generation, we can’t replace a token in situ; rather, the generative process must take into account all context. So alternative approaches to this will still have the same constraint of token generation. Second, we did in fact try multiple prompting strategies; some of these are in the paper, where for example Appendix D shows that when we add the true heading after each turn, it does not significantly change side-of-street errors. Others were preliminary runs that are not reported in the current version: in an earlier run with a different prompt, on all 6034 routes, every one of the models again had corruption above zero and a floor in the same range. We can show this variation across prompts in a revision.
If these points have addressed your concerns in part, we would appreciate it if you considered raising your score.