- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce Scrip, a system where a team of agents answers questions together. Each agent has a budget it can spend on extra information, and the money it has left decides how much its answer counts. Agents that predict well earn more over time, so reliable agents gain influence. Most of the improvement comes from letting agents buy evidence and trusting those with a good track record.
Critical strengths
- In general I like the setting and the question the authors are asking with this work. Most prior LLM market related work uses wealth as an aggregated signal or to allocate actions. This paper asks whether the same budget should pay for information and also set influence. I think that is a real design choice for multi agent llm systems. This is the main reason the work could interest me and the community, even though the evidence here is too thin to establish it.
- The controlled experiment is well-designed to evaluate what the paper set out to evaluate.
Critical weaknesses
- The biggest concern I have is that the central effect is basically null. The paper’s own preregistered primary outcomes, shared vs. separate budgets on 2Wiki and Brier, are basically null. The headline 0.8 accuracy gain is reported with conflicting intervals, which i'll circle back later, and one of them crosses zero. The larger >3 gain comes from a sweep that looks a bit post hoc. The shared budget is the paper's main contribution, which I like and as I said, can help move the conversation forward, but the results supporting it is basically null.
- I think the presentations are very hard to read and follow, plus there are conflicts here and there. To be honest, I sense heavy AI, likely codex, generated traits in the paper (e.g. frozen, sealed, and the figure style). I'm not saying it's an issue using AI-assistance for writing, but it's really hard to read and follow the general progression, plus the visualizations are also hard to read with text overlaps. At the same time, there are some inconsistency and conflicts, to name a few: (a) Gigure 2b shows identical shared and separate prices, but E.1 computes different ones. (b) Figure 18 ranks Scrip best on ECE, while Figures 17 and 20 say it is worse. I suggest the authors to give the writing and presentations a genuine clean up.
- The theory part in the paper really adds little. This is a minor point, but worth noting as it's related to the general hard to read issue.
Questions for authors
- Can you clarify which intervals is the real one? The +0.8 shared-vs-separate effect is reported as [+0.25, +1.35], [−0.2, +1.8], and [+0.2, +1.4] in different places. Please say which evaluation each comes from / or clarify which is the actual one.
- What actually drives the coupling effect? I'm wondering is the gain due to training-time selection (wasteful agents lose purchasing power) or to the debit at evaluation time? It would be great to see a 2×2 that crosses shared-vs-separate budgets in training with shared-vs-separate budgets in evaluation.
- Does the advantage hold against your strongest baseline under realistic retrieval? InfoDelphi is missing from the 1,000 question evaluation, yet it basically tied scrip in the original retrieval run. Please add it, with paired intervals.
I wonder considering raising my score when these questions are answered and experiments added with meaningful quality, but actually my biggest concern is still the hard-to-follow writing style and the null main result.
5 comments
Response to Reviewer 3Zjz (2/n)
W2: The paper is hard to follow, and two figures conflict
The reviewer is right on both specific checks, and we located the cause of each.
(a) Fig. 2b versus Appendix E.1. Panel (b) printed the same price 0.6833 in both rows because the panel hard-coded the separate-budget pool value. The shared value in the same episode is from Eq. (2), as E.1 computes. The panel has been rebuilt to show 0.6799 for shared and 0.6833 for separate, and its caption now names the E.1 episode as its source so the two cannot drift apart again.
(b) Fig. 18 versus Figs. 17 and 20. This was a sorting-direction bug in the calibration plot, not a data disagreement. The plotting routine sorted each metric panel ascending and then drew the value as bar length, so the longest bar — Scrip at ECE 0.047 — sat at the visually "best" end of the axis while the printed numbers said otherwise. The underlying data agree with Figs. 17 and 20: Scrip has worse ECE than the calibrated stack (0.047 versus 0.020 in Fig. 18) and the paired difference is +0.031 [+0.006, +0.041]. Scrip is better calibrated than the uncalibrated pools and worse than the calibrated stack, and the paper should say so plainly; the revision does. The panel now sorts so that the best row is at the top, and every panel carries a "lower is better" direction marker.
On readability we accept the criticism as substantive rather than stylistic, since two of three reviewers raise it. The revision: restates the shared-budget hypothesis in one plain sentence in the abstract and the first paragraph of Sec. 1, ahead of any market vocabulary; condenses the LMSR variant and the population-renewal material in Sec. 3 to a short pointer that defers the derivations to Appendices B and C, so Sec. 3 is organized around the sealed rule; adds a one-paragraph glossary of credited terms (balance, stake, sealed, compact, settlement) at the end of Sec. 1; and gives the multi-panel comparison figures a single shared row order plus an explicit better-direction marker, so a reader can no longer take the longest bar to be the best result.
W3: The theory adds little
We accept that the theory as written is not a proof that coupling helps, and the revision states that limitation in Sec. 3 rather than implying otherwise. We disagree that the theory is decorative, and W1 gives the concrete reason: the cost term in Proposition 2 generates a falsifiable prediction about where the effect can appear — large in the interior of the affordability range, absent at both ends — and Sec. 4.4 measures that shape across five levels. A section that predicts the location, the sign at each end, and the non-monotonicity of the main empirical result is doing work even though it is not an efficiency proof. The revision makes this link explicit by relocating the one-sentence prediction from Sec. 3.1 to sit immediately before the scarcity table.
Q1: Which interval is the real one?
The first row of the table in W1: +0.8 with [+0.25, +1.35], from the 1,000-question paired stream/item bootstrap over 5,000 paired observations. The other two are the same contrast in the 500-question evaluation and in the scarcity protocol's k≥6 arm.
Q2: Is the coupling effect from training-time selection or evaluation-time debit?
We ran the requested 2×2. Both training arms use the same five 300-draw streams and both evaluation arms use 1,000 controlled 2Wiki questions with three decoding seeds; the two cross cells are new training runs, and everything else follows Sec. 4.1.
| Train-time budget | Eval-time budget | Accuracy |
|---|---|---|
| Separate | Separate | 64.0 |
| Separate | Shared | 64.6 |
| Shared | Separate | 64.2 |
| Shared | Shared | 64.8 |
The evaluation-time debit carries most of the effect (+0.6), training-time selection carries +0.2, and the interaction is 0.0. The interpretation is that the coupling acts mainly as an incentive at decision time rather than as a filter that removes wasteful agents over training. This is consistent with the failure of the renewal arm to add more than 1.0 point (Fig. 8) and with the static-initial-wealth ablation at 62.8, which shows that most of the training-side gain is already captured by reliability weighting alone.
Response to Reviewer 3Zjz (3/3)
Q3: Does the advantage hold against the strongest baseline under realistic retrieval?
InfoDelphi was missing from the 1,000-question evaluation because its router was trained on the original 500-question split. We re-ran it on the expanded split with the same five training streams, the same BM25 pool, and the same reranker that never sees supporting-fact labels; only the evaluation set changed.
| Method | 1,000-question non-oracle accuracy |
|---|---|
| Separate budgets | 60.9 |
| Spend-matched relevance router | 60.6 |
| InfoDelphi | 62.2 |
| Scrip, shared account | 62.6 |
| Scrip − InfoDelphi | +0.4 [−0.5, +1.3] |
InfoDelphi and Scrip are statistically indistinguishable on the expanded retrieval set, with a difference of 0.4 points. We report this as a tie rather than a win, and the revision changes the claim accordingly: the shared-account advantage over the same acquisition policy with a separate budget is +1.7 points [+0.8, +2.6], and this is the contribution the retrieval evaluation supports. On the controlled task the comparison with InfoDelphi was already +0.8 [−0.3, +2.0] (Appendix D), so the paper never claimed a decisive margin over InfoDelphi; it claimed a benefit over separate budgets, and that survives retrieval.
Response to Reviewer 3Zjz (1/n)
We thank the reviewer for the detailed reading. We address the three weaknesses and three questions in order. Two of the presentation criticisms are correct and we have already fixed the underlying plotting code; one part of the main-result criticism rests on a reading of the primary endpoints that the paper does not support. We also add four experiments: a 2×2 crossing of training-time and evaluation-time budget mode, an InfoDelphi run on the 1,000-question retrieval evaluation, a cost-proportional pricing arm, and a fresh-seed replication of the scarcity peak.
W1: The main effect is null, and the +0.8 interval conflicts across the paper
We take the statistical concern seriously, so we start with what is and is not null. The paper declares two primary endpoints in Sec. 4.1: shared versus separate budgets on 2Wiki NLL, and on forecasting Brier. Their paired results are:
So one of the two declared primaries is null and one is not. We agree that the 2Wiki NLL interval reaches zero at its upper limit, which means the correct characterization is a small but consistently signed improvement, not a decisive one. We should have written that sentence in the paper rather than leaving the reader to infer it from a table in the appendix; the revision states it directly in Sec. 4.1.
On the three intervals: they are the same +0.8 contrast computed in three analyses, not three estimates of one quantity. We never placed them side by side, which is the source of the conflict. Here is the reconciliation, which we will add as a table in the revision.
The interval that crosses zero is the 500-question evaluation, and it is the widest of the three precisely because it uses the fewest questions. The 1,000-question main analysis is the tightest. The primary estimate in the revision is the first row; the other two are retained for transparency and will be labeled by analysis wherever they appear.
On the +3.6 being post hoc: the scarcity scan is a predeclared five-level scan (Sec. 4.1), and its shape is predicted by the theory before the scan is run. Proposition 2 gives the cost term as log[(1−ait)/(1−ϕt)], which vanishes when spending is small relative to wealth and also vanishes when evidence is unaffordable; Sec. 3.1 states that the effect must therefore be largest in between. All five affordability levels are positive and all five intervals exclude zero ([+0.2,+1.4], [+1.2,+2.4], [+2.8,+4.4], [+1.7,+3.3], [+0.5,+2.1]). This is a dose-response curve, not a selected cell. We additionally re-run the peak on five fresh streams:
The replication is 0.2 points below the original estimate with an overlapping interval, so the peak is not an artifact of the particular five training streams. Setting: Qwen3-8B, controlled 2Wiki, sealed aggregation in both arms, the same five 300-draw training streams and three decoding seeds as Sec. 4.1, with only the affordability level k≈1.5 retained.