- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Invarialoop argues that verifying candidates or intermediates doesn't certify the object finally released after mapping, screening, and aggregation. Contributions: a scope rule, matched candidate evaluation, and three simple propositions. Validation: synthetic tests, Bike Sharing, and SEC filings, where averaging passed accounting checks but was less accurate.
Critical strengths
This paper clarifies an easily overlooked problem: which object a verification conclusion applies to, and offers a concrete distinction. Table 1 and Section 2 separate routes, mapping, screening, aggregation or selection, final output into successive stages, and they state a scope rule under which a verification result supports only the object that was actually checked. The single-sample SEC study is a demonstration: Table 5 and Figure 2 show all 99 filings released by the arithmetic average satisfy the accounting identity, yet only 56 have both totals correct, so passing verification and being correct on the task are two independent endpoints. Section 4.1 therefore requires reporting the property used to decide release separately from the withheld task correctness. Even though I find its theoretical content shallow, this norm would help systems that use multi-route aggregation avoid a common evaluation error, namely treating the verification of an intermediate object as verification of the final output.
Critical weaknesses
My concern is that the paper's core empirical finding is largely determined by the experimental design and offers little novel insight. The most practically relevant evidence in the paper comes from the single-sample SEC study. Taking Section 4.5, Table 5, and Figure 2 together, the arithmetic average released 99 filings, all of which satisfy the accounting identity, but only 56 of them have both totals correct, whereas the fixed Qwen3.5-9B route is correct on 94 of these 99 filings (and on 95 of all 100). The paired outcomes are 56 filings where both are correct, 38 where only Qwen is correct, 0 where only the average is correct, and 5 where both are wrong. The paired difference is −38.4 percentage points, with a 95% interval of [−48.5, −29.3]. I recomputed these figures from the authors' per-filing table using their analyze_sec.py, and they match the paper, which is to the authors' credit. However, my problem is that this gap is likely produced by the choice to average one strong route with one weak route. In single_sample_filings.csv, Ministral has both totals correct on only 55 filings, versus 95 for Qwen. Among the 99 released filings, there are 40 in which Qwen is correct and Ministral is wrong, and the average is correct on only 2 of them. In addition, 97 filings have an admitted candidate count of 2, meaning both Qwen and Ministral were admitted. Paper doesn't report Ministral's accuracy, nor does it give this breakdown of the 40 filings. In Section 4.5, screening uses only an invalid-output-rate threshold of 0.10 on development filings: Qwen and Ministral are both 0/10 invalid, while Gemma is 4/10 and is excluded. The authors state screening is not assumed to improve task accuracy. I therefore think the 38 filings where only Qwen is correct result from averaging a route that is roughly 95% accurate with one that is roughly 55% accurate, and that this gap is not surprising. Proposition 3 states, for an affine check, the residual of a weighted average equals the weighted sum of the residuals, so zero-residual candidates remain zero-residual after averaging. I regard this experiment as using an artificially chosen operator to demonstrate a fact that an existing proposition already establishes.
Distributional study is weaker still. In Section 4.6, only 1 of 100 filings shows a release disagreement between the two verifiers (1 of 16 candidates violates the identity, so the violation mass is 1/16 = 6.25% ≤ 10%, and the residual of the average is −$103,878.94), and describes this only as an existence witness of the failure mode. I confirmed from the CSV, this is the only disagreement case (Nuvve Holding Corp.) and only 7 of the 100 filings have a violation mass greater than 0. Regarding the controlled tests, Table 3 and Appendix A report unverified false-acceptance rates of 44% in the sensor-walls setting and 24% in the obstacles setting.
My second concern is the lack of a testable methodology, together with the absence of any substantive comparison to existing work. The paper states explicitly that it offers an evaluation method, but its three propositions are standard facts: a zero mean doesn't imply a small violation mass; the violation mass of a weighted mixture is bounded below by the minimum over its components; and an affine residual is linear under weighted averaging. Matched candidate evaluation is a paired controlled design in which the upstream stage is frozen, and only one downstream factor is varied, which is routine experimental methodology. Paper's central distinction, namely which object a verification result applies to, already appears implicitly in selective prediction, conformal risk control, and ensemble learning. Section 5 merely declares, one family at a time, that Invarialoop doesn't replace these guarantees. Paper also doesn't compare against any of them on a shared task, offers no example where an existing method fails and the proposed approach succeeds. The experimental scope is equally narrow: the SEC study uses only 100 eligibility-filtered filings and three routes, the Bike Sharing experiment inherits part of its validity, in the paper's own words, from generators consistent with the identity, and the screening is transductive, computed on the evaluation batch itself. The authors report these limitations honestly, but they show their paper can support only the claim this phenomenon exists in these constructed settings. For a paper proposing a new evaluation framework, there is no scenario in which the framework itself is put to the test, for example, one where it predicts a specific failure and the prediction is confirmed on an independent task.
Questions for authors
-
In the SEC single-sample study, Ministral has both totals correct on far fewer filings than Qwen, but the screen admits it because it only checks the invalid-output rate. How would the 38 Qwen-only filings change if the average were taken over routes of comparable accuracy, or if the screen also used a task-relevant signal?
-
Where would an existing method give a different release decision from Invarialoop on the same task? Can you show concrete cases where the scope rule predicts a failure that those methods miss?
3 comments
Thank you for your response. I appreciate your transparency in addressing the breakdown of the SEC filings and the conditions behind the Nuvve disagreement case. However, my main concerns remain unresolved. Your response shows that the single-sample accuracy gap is driven by averaging routes of very different quality to illustrate a narrow point, namely that passing an accounting check doesn't imply task correctness. Furthermore, as noted in your rebuttal, your revision still lacks a head-to-head comparison with calibrated selective prediction on a shared task and a confirmatory test on new filings, leaving the evaluation limited to the same 100 filings and three routes. Therefore, I am maintaining my score.
Thank you for engaging with our response. We agree the claim is narrow: passing the release check does not establish task correctness. We think the narrow claim is the useful one, because the release check is the evidence these pipelines act on, and your review describes treating verification of an intermediate object as verification of the final output as a common evaluation error. Averaging was chosen to show that the check stays satisfied while accuracy falls, not as a recommended operator. We agree that a comparison with calibrated selective prediction on a shared task and a confirmatory test on new filings are the right next steps.
Response to Reviewer 2bZF
Thank you for rerunning analyze_sec.py on our per-filing table and confirming the numbers. Our general response restates the purpose: the studies test whether evidence carries across an operation, not which operator is best. Within that frame, your analysis sharpens the paper.
W1/Q1, averaging a strong and a weak route. Your breakdown is right. On all 38 filings where both routes were admitted and only Qwen was correct, the average was wrong; the 2 correct averages among your 40 are the filings where only Qwen was admitted. The average is correct only when it coincides with a correct constituent: none of the 39 averages that differ from every admitted candidate is correct. We agree the direction of the gap follows from constituent quality, and Ministral's accuracy (55 of 99) should have been reported; the revision reports it. The claim the study supports is narrower than "averaging hurts": the identity holds on all 99 averages whatever their accuracy, so passing it cannot stand in for correctness. That stays true for routes of comparable accuracy. Any weighted average of identity-satisfying candidates satisfies the identity (Proposition 3), so a comparable-accuracy average would shrink the accuracy gap without making the check any more informative about it. A task-relevant signal is the right remedy, and it is what Section 4.1 asks for: report task correctness separately from the release check. With two admitted routes, majority vote is one such signal. Computed from the same table, it releases 60 filings (58 where both routes agree, plus the 2 Qwen-only filings), 56 of them correct, and abstains on the other 39. It is not sufficient either: on 4 filings both routes agree on the same wrong totals, and both records satisfy the identity.
W1, the distributional study. Agreed: one disagreement, and seven filings with positive violation mass. The disagreement has a stated condition. With at most two equally weighted candidates, a mass threshold of 0.10 admits no violating candidate, so in the single-sample study every average the mass rule releases also passes the mean check. A mass-rule release that fails the mean check needs a violator under the threshold, which at 0.10 needs at least ten candidates. When one does, the mean's residual is bounded only by the violators' share times their residual, which is how one candidate in sixteen produced a residual of -$103,878.94. We present this as a mechanism with a condition, not as a rate.
W2/Q2, existing methods. The scope rule does not compete with selective prediction or conformal risk control; it states which object their guarantees cover. A risk-control threshold calibrated on the intact route's outputs bounds risk for those outputs, not for an average built from them. On our own data, the candidate-mass rule is a candidate-level gate that decides differently from the output check on Nuvve, and it is the one that is wrong there. We agree that a head-to-head with calibrated selective prediction on a shared task would strengthen the paper; we have not run it for this revision.
Testable predictions. The conditions above say, before looking at any filing, when the two checks must agree (no admitted violator) and when they can differ (a violator under the threshold), and the observed decisions fall into both cases as stated. We agree a confirmatory test on new filings is the stronger form.
Scope. We agree that 100 filings and three routes limit generality, and the paper says so. The per-filing table and script you used are in the supplement.
If these answers address your concerns, we would be grateful if you would reconsider your score.