acceptodds
Under review as a conference paper at ICLR 2027

Scrip: When Does a Shared Budget Help Collective Reasoning?

Abstract

How should a collective solver allocate a limited budget between acquiring evidence and weighting predictions? We study this question through Scrip, where frozen language agents purchase documents, tools, or peer rationales from the same account that determines their predictive influence. Verified outcomes update account balances during training, and evaluation uses a frozen trained state. Across controlled multi-hop and forecasting tasks, selective acquisition and outcome-based weighting explain most of the improvement over a judge. Coupling acquisition and influence adds a smaller gain at the original budget and a larger gain under scarcity. The advantage is also present under non-oracle retrieval, where retrieval noise narrows it. We characterize the mechanism with two analyses. Matched-report replay identifies sealed Scrip exactly with a cost-adjusted online pool, while a convex-hull bound locates the limit of probability aggregation when reports lack complementary information. These results identify when a shared budget helps collective reasoning and when the task calls for richer communication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Collective Reasoning, Mechanism Design, Predictive Market

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3ZjzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce Scrip, a system where a team of agents answers questions together. Each agent has a budget it can spend on extra information, and the money it has left decides how much its answer counts. Agents that predict well earn more over time, so reliable agents gain influence. Most of the improvement comes from letting agents buy evidence and trusting those with a good track record.

Critical strengths
  • In general I like the setting and the question the authors are asking with this work. Most prior LLM market related work uses wealth as an aggregated signal or to allocate actions. This paper asks whether the same budget should pay for information and also set influence. I think that is a real design choice for multi agent llm systems. This is the main reason the work could interest me and the community, even though the evidence here is too thin to establish it.
  • The controlled experiment is well-designed to evaluate what the paper set out to evaluate.
Critical weaknesses
  • The biggest concern I have is that the central effect is basically null. The paper’s own preregistered primary outcomes, shared vs. separate budgets on 2Wiki and Brier, are basically null. The headline 0.8 accuracy gain is reported with conflicting intervals, which i'll circle back later, and one of them crosses zero. The larger >3 gain comes from a sweep that looks a bit post hoc. The shared budget is the paper's main contribution, which I like and as I said, can help move the conversation forward, but the results supporting it is basically null.
  • I think the presentations are very hard to read and follow, plus there are conflicts here and there. To be honest, I sense heavy AI, likely codex, generated traits in the paper (e.g. frozen, sealed, and the figure style). I'm not saying it's an issue using AI-assistance for writing, but it's really hard to read and follow the general progression, plus the visualizations are also hard to read with text overlaps. At the same time, there are some inconsistency and conflicts, to name a few: (a) Gigure 2b shows identical shared and separate prices, but E.1 computes different ones. (b) Figure 18 ranks Scrip best on ECE, while Figures 17 and 20 say it is worse. I suggest the authors to give the writing and presentations a genuine clean up.
  • The theory part in the paper really adds little. This is a minor point, but worth noting as it's related to the general hard to read issue.
Questions for authors
  • Can you clarify which intervals is the real one? The +0.8 shared-vs-separate effect is reported as [+0.25, +1.35], [−0.2, +1.8], and [+0.2, +1.4] in different places. Please say which evaluation each comes from / or clarify which is the actual one.
  • What actually drives the coupling effect? I'm wondering is the gain due to training-time selection (wasteful agents lose purchasing power) or to the debit at evaluation time? It would be great to see a 2×2 that crosses shared-vs-separate budgets in training with shared-vs-separate budgets in evaluation.
  • Does the advantage hold against your strongest baseline under realistic retrieval? InfoDelphi is missing from the 1,000 question evaluation, yet it basically tied scrip in the original retrieval run. Please add it, with paired intervals.

I wonder considering raising my score when these questions are answered and experiments added with meaningful quality, but actually my biggest concern is still the hard-to-follow writing style and the null main result.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3Zjz (1/n)

We thank the reviewer for the detailed reading. We address the three weaknesses and three questions in order. Two of the presentation criticisms are correct and we have already fixed the underlying plotting code; one part of the main-result criticism rests on a reading of the primary endpoints that the paper does not support. We also add four experiments: a 2×2 crossing of training-time and evaluation-time budget mode, an InfoDelphi run on the 1,000-question retrieval evaluation, a cost-proportional pricing arm, and a fresh-seed replication of the scarcity peak.

W1: The main effect is null, and the +0.8 interval conflicts across the paper

We take the statistical concern seriously, so we start with what is and is not null. The paper declares two primary endpoints in Sec. 4.1: shared versus separate budgets on 2Wiki NLL, and on forecasting Brier. Their paired results are:

Primary endpointPaired difference95% intervalReading
2Wiki NLL (lower better)−0.026[−0.049, −0.000]does not cross zero
Forecasting Brier (lower better)−0.002[−0.007, +0.003]crosses zero

So one of the two declared primaries is null and one is not. We agree that the 2Wiki NLL interval reaches zero at its upper limit, which means the correct characterization is a small but consistently signed improvement, not a decisive one. We should have written that sentence in the paper rather than leaving the reader to infer it from a table in the appendix; the revision states it directly in Sec. 4.1.

On the three intervals: they are the same +0.8 contrast computed in three analyses, not three estimates of one quantity. We never placed them side by side, which is the source of the conflict. Here is the reconciliation, which we will add as a table in the revision.

IntervalEvaluationAnalysis
[+0.25, +1.35]1,000 questions, five streamsMain analysis: paired stream/item bootstrap over 5,000 paired observations, b=118 shared-only versus c=78 separate-only, exact binomial p=0.0052
[−0.2, +1.8]500 questionsOriginal sealed-only evaluation, Appendix D
[+0.2, +1.4]1,000 questionsScarcity protocol, k≥6 arm (Fig. 12)

The interval that crosses zero is the 500-question evaluation, and it is the widest of the three precisely because it uses the fewest questions. The 1,000-question main analysis is the tightest. The primary estimate in the revision is the first row; the other two are retained for transparency and will be labeled by analysis wherever they appear.

On the +3.6 being post hoc: the scarcity scan is a predeclared five-level scan (Sec. 4.1), and its shape is predicted by the theory before the scan is run. Proposition 2 gives the cost term as log⁡[(1−ait)/(1−ϕt)]\log[(1-a_i^t)/(1-\phi_t)], which vanishes when spending is small relative to wealth and also vanishes when evidence is unaffordable; Sec. 3.1 states that the effect must therefore be largest in between. All five affordability levels are positive and all five intervals exclude zero ([+0.2,+1.4], [+1.2,+2.4], [+2.8,+4.4], [+1.7,+3.3], [+0.5,+2.1]). This is a dose-response curve, not a selected cell. We additionally re-run the peak on five fresh streams:

Streamsk≈1.5, shared − separate
Original five+3.6 [+2.8, +4.4]
Five fresh streams+3.4 [+2.6, +4.2]

The replication is 0.2 points below the original estimate with an overlapping interval, so the peak is not an artifact of the particular five training streams. Setting: Qwen3-8B, controlled 2Wiki, sealed aggregation in both arms, the same five 300-draw training streams and three decoding seeds as Sec. 4.1, with only the affordability level k≈1.5 retained.

AuthorsOpenReview ↗

Response to Reviewer 3Zjz (2/n)

W2: The paper is hard to follow, and two figures conflict

The reviewer is right on both specific checks, and we located the cause of each.

(a) Fig. 2b versus Appendix E.1. Panel (b) printed the same price 0.6833 in both rows because the panel hard-coded the separate-budget pool value. The shared value in the same episode is p(A)=97.9/144=0.6799p(A)=97.9/144=0.6799 from Eq. (2), as E.1 computes. The panel has been rebuilt to show 0.6799 for shared and 0.6833 for separate, and its caption now names the E.1 episode as its source so the two cannot drift apart again.

(b) Fig. 18 versus Figs. 17 and 20. This was a sorting-direction bug in the calibration plot, not a data disagreement. The plotting routine sorted each metric panel ascending and then drew the value as bar length, so the longest bar — Scrip at ECE 0.047 — sat at the visually "best" end of the axis while the printed numbers said otherwise. The underlying data agree with Figs. 17 and 20: Scrip has worse ECE than the calibrated stack (0.047 versus 0.020 in Fig. 18) and the paired difference is +0.031 [+0.006, +0.041]. Scrip is better calibrated than the uncalibrated pools and worse than the calibrated stack, and the paper should say so plainly; the revision does. The panel now sorts so that the best row is at the top, and every panel carries a "lower is better" direction marker.

On readability we accept the criticism as substantive rather than stylistic, since two of three reviewers raise it. The revision: restates the shared-budget hypothesis in one plain sentence in the abstract and the first paragraph of Sec. 1, ahead of any market vocabulary; condenses the LMSR variant and the population-renewal material in Sec. 3 to a short pointer that defers the derivations to Appendices B and C, so Sec. 3 is organized around the sealed rule; adds a one-paragraph glossary of credited terms (balance, stake, sealed, compact, settlement) at the end of Sec. 1; and gives the multi-panel comparison figures a single shared row order plus an explicit better-direction marker, so a reader can no longer take the longest bar to be the best result.

W3: The theory adds little

We accept that the theory as written is not a proof that coupling helps, and the revision states that limitation in Sec. 3 rather than implying otherwise. We disagree that the theory is decorative, and W1 gives the concrete reason: the cost term in Proposition 2 generates a falsifiable prediction about where the effect can appear — large in the interior of the affordability range, absent at both ends — and Sec. 4.4 measures that shape across five levels. A section that predicts the location, the sign at each end, and the non-monotonicity of the main empirical result is doing work even though it is not an efficiency proof. The revision makes this link explicit by relocating the one-sentence prediction from Sec. 3.1 to sit immediately before the scarcity table.

Q1: Which interval is the real one?

The first row of the table in W1: +0.8 with [+0.25, +1.35], from the 1,000-question paired stream/item bootstrap over 5,000 paired observations. The other two are the same contrast in the 500-question evaluation and in the scarcity protocol's k≥6 arm.

Q2: Is the coupling effect from training-time selection or evaluation-time debit?

We ran the requested 2×2. Both training arms use the same five 300-draw streams and both evaluation arms use 1,000 controlled 2Wiki questions with three decoding seeds; the two cross cells are new training runs, and everything else follows Sec. 4.1.

Train-time budgetEval-time budgetAccuracy
SeparateSeparate64.0
SeparateShared64.6
SharedSeparate64.2
SharedShared64.8

The evaluation-time debit carries most of the effect (+0.6), training-time selection carries +0.2, and the interaction is 0.0. The interpretation is that the coupling acts mainly as an incentive at decision time rather than as a filter that removes wasteful agents over training. This is consistent with the failure of the renewal arm to add more than 1.0 point (Fig. 8) and with the static-initial-wealth ablation at 62.8, which shows that most of the training-side gain is already captured by reliability weighting alone.

AuthorsOpenReview ↗

Response to Reviewer 3Zjz (3/3)

Q3: Does the advantage hold against the strongest baseline under realistic retrieval?

InfoDelphi was missing from the 1,000-question evaluation because its router was trained on the original 500-question split. We re-ran it on the expanded split with the same five training streams, the same BM25 pool, and the same reranker that never sees supporting-fact labels; only the evaluation set changed.

Method1,000-question non-oracle accuracy
Separate budgets60.9
Spend-matched relevance router60.6
InfoDelphi62.2
Scrip, shared account62.6
Scrip − InfoDelphi+0.4 [−0.5, +1.3]

InfoDelphi and Scrip are statistically indistinguishable on the expanded retrieval set, with a difference of 0.4 points. We report this as a tie rather than a win, and the revision changes the claim accordingly: the shared-account advantage over the same acquisition policy with a separate budget is +1.7 points [+0.8, +2.6], and this is the contribution the retrieval evaluation supports. On the controlled task the comparison with InfoDelphi was already +0.8 [−0.3, +2.0] (Appendix D), so the paper never claimed a decisive margin over InfoDelphi; it claimed a benefit over separate budgets, and that survives retrieval.

Reviewer_pGWDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Scrip, a collective solver that uses a shared account to fund evidence acquisition and determine predictive influence. Each LLM agent can spend virtual credits to purchase documents, tool results, or other agents’ reasoning, then update its prediction using the acquired information. In the sealed implementation, each agent’s remaining balance determines its weight in the collective prediction. The paper evaluates Scrip using Qwen3-8B and GPT-5-mini on mathematical reasoning, scientific question answering, multi-hop question answering, and binary forecasting tasks. The experiments compare Scrip with single-agent, deliberation, learned aggregation, and internal-market baselines, and examine the effects of budget scarcity, acquisition policies, error correlation, and non-oracle retrieval. Ablations separate the contributions of evidence acquisition, outcome-based weighting, and budget coupling, while matched-report replay and controlled information-structure tests examine the aggregation mechanism and its limitations.

Critical strengths

The experimental design carefully separates the contributions of evidence acquisition, outcome-based weighting, and budget coupling. Comparisons with separate budgets and learned aggregation baselines help identify the additional value of the shared account, while experiments varying budget scarcity and using non-oracle retrieval examine when that value persists.

Critical weaknesses

Weakness: The theoretical justification for coupling evidence acquisition with predictive influence remains limited. The analysis derives weight updates and loss bounds under the prescribed mechanism, but these results do not establish that the coupling improves resource allocation relative to separate budgets. In particular, an agent that purchases expensive but crucial evidence is immediately downweighted, even though its prediction may become more reliable. It remains unclear when subsequent performance-based rewards adequately compensate for this penalty, or whether the resulting interaction admits a stable and efficient equilibrium.

Questions for authors
  1. Under what conditions does coupling evidence costs with predictive influence improve resource allocation over separate budgets, particularly when an agent purchases expensive but crucial evidence?

  2. Does the joint acquisition and prediction mechanism admit an equilibrium, and if so, can its stability and efficiency be characterized?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pGWD (1/n)

We thank the reviewer for the careful reading and for the positive assessment of the experimental design. The single weakness is the central theoretical question, and it is the right question to ask. Our answer has two parts: a short argument showing that the penalty imposed by an expensive purchase is exactly commensurable with the information gain it buys, so the two are not ad hoc trade-offs; and a statement of what the mechanism does and does not guarantee as an allocation rule, with a proof in Appendix B.

W1: Why does coupling not systematically punish agents who buy expensive but crucial evidence?

It does not, and the reason is a single inequality that the paper already contains in two separate places but never joined. We should have joined them, and the revision adds the corollary below.

Under sealed settlement, an agent that spends fraction ai=ci/Wia_i=c_i/W_i of its wealth has expected log wealth after one task, conditional on its own information, of

Elog⁡UiWi=log⁡(1−ai)+KL(qipost∥p),\mathbb{E}\log\frac{U_i}{W_i}=\log(1-a_i)+\mathrm{KL}(q_i^{\text{post}}\|p),

where qipostq_i^{\text{post}} is the post-purchase report and pp is the sealed price. This is Eq. (2) and Eq. (6) rearranged: the first term is the debit from the purchase, and the second is the expected return from reporting better than the pool. The agent buys exactly when the second term grows by more than the first falls. The criterion is

KL(qipre∥p)−KL(qipost∥p)>log⁡11−ai≈ai.\mathrm{KL}(q_i^{\text{pre}}\|p)-\mathrm{KL}(q_i^{\text{post}}\|p)>\log\frac{1}{1-a_i}\approx a_i.

This is the same quantity as Proposition 3's value-of-information criterion, Δκ>log⁡[W/(W−c)]\Delta_\kappa>\log[W/(W-c)], with the pool price in place of the external price. So "expensive but crucial" is not a case the mechanism mishandles: crucial evidence is evidence that reduces KL(qi∥p)\mathrm{KL}(q_i\|p) by a lot, and the inequality is easiest to satisfy exactly then. Expensive-but-redundant evidence is the case the mechanism penalizes, and that is the intended behavior.

The magnitude is worth stating because it answers the "immediately downweighted" concern directly. With the paper's defaults, W0=100W_0=100 and service prices (2,4,8,16)(2,4,8,16), the largest single-tier debit is ai=0.16a_i=0.16, so the penalty term is log⁡(1/0.84)=0.174\log(1/0.84)=0.174 nats. The information gain from a purchased gold-supporting passage in the controlled task averages well above that; the forced buy/no-buy diagnostic in Fig. 27 estimates 73% correct purchase decisions and mean regret 0.031 nats for the bundle controller. For the purchases the controller actually makes, the expected KL gain exceeds the debit; where it does not, the mechanism should and does discourage the purchase.

What the coupling buys over separate budgets, stated precisely. Separate budgets sever the link, so the agent's private purchase threshold is zero and it buys any evidence with positive expected KL gain regardless of cost. Coupling makes the private threshold equal the social cost of the debit. When evidence quality is heterogeneous — some purchases move the report a lot, others barely move it — the coupled threshold raises collective accuracy; when all available evidence is highly informative, the two arms coincide, which is what the k≥6 arm of the scarcity scan shows (+0.8, and small).

AuthorsOpenReview ↗

Response to Reviewer pGWD (2/2)

Q2: Is there an equilibrium, and can stability and efficiency be characterized?

Yes for existence, and with an explicit bound on the externality.

Existence. Fix the training state. A within-task game has finitely many acquisition actions per agent (at most four menu decisions, per Sec. 3.2) and reports on the smoothed simplex, so the action set is compact and the payoff in Eq. (2) is continuous. A mixed-strategy Nash equilibrium therefore exists. We do not claim it is unique.

Externality bound. The influence of one agent's purchase on the collective answer is bounded by its own weight. From the capital-limited-influence lemma in Appendix B, a coalition adding stake ss to a pool of stake MM moves the pool by at most s/(M+s)s/(M+s) in total variation. With 16 equal agents and a single-tier purchase at ai=0.16a_i=0.16, this is approximately 0.16/16≈0.010.16/16\approx0.01; the externality of one purchase on other agents' realized payouts is first-order small. A purchase is therefore close to a private decision with a private cost, which is why the coupled threshold and the social optimum nearly coincide in the intermediate-scarcity regime.

Stability over training. Proposition 1 gives the no-purchase regret bound and Proposition 2 the cost-adjusted version, so the bound on Ri(T)\mathcal{R}_i(T) does not grow with TT, and the per-task average regret therefore tends to zero as 1/T1/T; this holds without requiring independence between agents. This is an online guarantee on realized wealth, not a claim that the population reaches an efficient equilibrium; the paper is explicit about that distinction in Sec. 3.1, and we have added a sentence to the same effect in the revision.

Where the characterization stops. The guarantees cover capital and maker loss under the fixed-seat protocol and the regret bounds above. They do not cover collusion, unreliable labels, or strategic evidence content, and Appendix G lists these as open problems. We have also added the corollary above to Appendix B with its proof, and restated the relevant prediction ("the effect is largest in the interior of the affordability range and negligible at both ends") in Sec. 4.4 directly alongside the scarcity scan it explains, rather than leaving it only in the theory section.

Reviewer_QN7tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies resource allocation in multi-agent LLM systems. Most existing systems give several agents the same context and combine their answers through voting, confidence weighting, a judge, or debate. Scrip instead lets agents purchase different evidence and uses the same account balance to determine their weight in the final prediction. The main contribution is a careful decomposition showing that selective evidence acquisition and learned reliability weights provide most of the improvement, while coupling evidence spending to prediction weight gives a smaller additional gain that is largest under intermediate budget scarcity.

Critical strengths
  • The paper studies an underexplored aspect of multi-agent reasoning: how agents should allocate limited resources to acquiring different information before their answers are aggregated.
  • The separate-budget control cleanly isolates the effect of charging evidence purchases against prediction weight.
  • The evaluation includes strong aggregation and deliberation baselines, matched resource access, uncertainty estimates, non-oracle retrieval, and useful failure analyses.
  • The paper is transparent that acquisition and reliability weighting, rather than the shared account, explain most of the improvement.
Critical weaknesses
  1. The paper is difficult to place within the standard multi-agent reasoning setting. The abstract and introduction introduce accounts, evidence purchases, predictive influence, and multiple market mechanisms before clearly explaining the baseline setting and the specific problem that existing voting, judging, or debate methods cannot solve. The paper would be easier to follow if it first stated the shared-budget hypothesis plainly and separated the core method from its many variants and diagnostics.

  2. My main technical concern is that the main contribution has a narrow empirical effect. Sharing the account improves over separate budgets by 0.8 points in the primary controlled 2Wiki comparison and by 3.6 points at one intermediate scarcity level. Differences on mathematics, science, and forecasting are small and often statistically inconclusive. The paper therefore shows that coupling can help, but not that it is broadly useful.

  3. The strongest gain depends on an internal credit system and chosen service prices, which are distinct from actual token and tool costs. It is unclear how these prices should be set in practice or whether the gain remains when scarcity is imposed through real compute or monetary budgets.

  4. Controlled 2Wiki guarantees that gold-supporting passages are available for purchase. Under non-oracle retrieval, the gain over separate budgets falls to 1.7 points, and no other realistic retrieval or tool-use setting is studied. This limits the evidence that Scrip helps when finding useful information is itself difficult.

  5. The replay analysis shows that sealed Scrip is exactly a cost-adjusted fractional online pool on a fixed trace. The novelty is therefore the closed-loop application of this rule to evidence acquisition, rather than a new aggregation method or theoretical result.

Questions for authors
  • How should evidence prices be chosen in practice, and do the gains persist when they reflect actual inference or tool costs?
  • Is there a task beyond controlled 2Wiki where sharing the account significantly improves over separate budgets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer QN7t (1/n)

We thank the reviewer for the careful summary and for recognizing that the decomposition is clean and the baselines strong. The weaknesses split into one about presentation, three about how far the results generalize, and one about novelty. We address the generalizability ones with three new experiments — a second task under non-oracle retrieval, a cost-proportional pricing arm, and a fresh-seed replication — and we accept the novelty limitation as stated rather than argue it.

W1: It is hard to place the paper in the standard multi-agent setting

We accept this. The abstract introduces accounts, purchases, and influence before it says what problem existing voting, judging, or debate cannot solve. The revision leads with the hypothesis in plain terms — is it better for one budget to both pay for evidence and decide whose answer counts, or should those two decisions be separate from each other? — then introduces Scrip as one implementation, then the market vocabulary. The LMSR variant and population renewal move entirely into the appendix; Secs. 1–3 contain only the sealed rule. The revision also adds a four-line glossary of balance, stake, sealed, and settlement at the end of Sec. 1, because "sealed" was doing too much unstated work.

W2: The main effect is narrow and often statistically inconclusive

We agree with the characterization and have changed the paper's claim to match it. The paper does not show that sharing an account is broadly better; it shows that the benefit is conditional, and that the condition is identifiable in advance. Three things support the narrower claim.

The effect is positive at every affordability level tested. The scarcity scan is a predeclared five-level scan (Sec. 4.1); the level at which the effect peaks was not pre-specified, so we do not treat the +3.6 cell as a confirmatory test. What we do claim is the shape: all five paired differences are positive with intervals excluding zero, and the shape is non-monotonic in the way Proposition 2 predicts.

The effect appears on a second task. Controlled 2Wiki places gold passages in the menu; that is a favorable setting, and the reviewer's W4 says so. We therefore ran the same protocol on HotpotQA with the same acquisition interface and no oracle information in the retrieval path.

TaskSeparateScrip, sharedDifference
2Wiki, k≥664.064.8+0.8 [+0.2, +1.4]
2Wiki, k≈1.558.462.0+3.6 [+2.8, +4.4]
HotpotQA, k≈1.553.156.0+2.9 [+1.8, +4.0]

On HotpotQA the shared account retains a 2.9-point advantage at intermediate scarcity, smaller than the 3.6-point 2Wiki result, with an interval excluding zero. Setting: Qwen3-8B, 1,000 HotpotQA questions, five 300-draw training streams, three decoding seeds, sealed aggregation in both arms, BM25 retrieval over the full pool and a query-based reranker that never sees supporting-fact labels, and the same service menu with affordability held at k≈1.5 in both arms. Gold support is used only for scoring.

The effect replicates on fresh streams. We re-ran the k≈1.5 peak on five new training streams: +3.4 [2.6, 4.2] versus +3.6 [+2.8, +4.4] originally. The intervals overlap and the point estimate is 0.2 lower, so the peak is not an artifact of one set of streams.

W3: The prices are internal credits, not real token or tool costs

This is the sharpest of the four, and the concern is fair: our tier prices (2,4,8,16)(2,4,8,16) are chosen, not measured. We ran an arm in which each service is priced in proportion to its measured cost — generation plus reading tokens at a fixed exchange rate — with the rate set so that total expenditure at k≈1.5 matches the original schedule. Everything else is held fixed: same agents, same menu contents, same five streams, sealed aggregation in both arms.

PricingkSeparateSharedDifference
Fixed tiers, defaultk≥664.064.8+0.8
Cost-proportionalk≈1.557.259.6+2.4 [+1.6, +3.2]
Cost-proportionalk≥663.864.2+0.4 [-0.3, +1.1]

Under cost-proportional pricing the intermediate-scarcity advantage shrinks from +3.6 to +2.4 and still excludes zero; at the default budget it falls to +0.4 and is not distinguishable from zero. This is consistent with our reading rather than a rescue: the mechanism's value is concentrated where a budget forces a choice, and real costs are less heterogeneous than our tiers, so the effect is smaller. We report it as a smaller effect, not an equivalent one.

AuthorsOpenReview ↗

Response to Reviewer QN7t (2/2)

W4: Controlled 2Wiki makes gold passages available for purchase

Correct, and we say so in Sec. 4.1. The controlled construction exists to isolate allocation from retrieval quality, and the non-oracle results are the generalization check. The expanded 1,000-question retrieval evaluation gives 62.6 for the shared account against 60.9 for separate budgets and 60.6 for a spend-matched relevance router, a paired +1.7 [+0.8, +2.6]. Retrieval noise narrows the advantage, and the HotpotQA result in W2 is a further step down in passage quality with the advantage still present at scarcity. What we do not show, and no longer claim, is that the advantage is as large when finding useful information is itself hard; the revision states the 1.7-point retrieval figure as the headline for that regime.

W5: Replay reduces sealed Scrip to a cost-adjusted pool, so novelty is limited

We agree with the observation and with the conclusion the reviewer draws from it. The paper's own replay shows an exact identity on a fixed trace: conditional on the same purchases, fees, reports, and labels, sealed Scrip and a cost-adjusted fractional pool make identical predictions (Fig. 9). We reported this because it is the honest description of the mechanism, and because identifying the aggregation operator is what lets the ablation separate acquisition from weighting. The contribution is not a new aggregator; it is the closed-loop application of a known rule to evidence acquisition, where the purchase itself moves the weight through the same account. The revision says this in the abstract's last sentence rather than leaving it to Sec. 4.5.

Q1: How should prices be chosen in practice, and do gains persist at real costs?

W3 answers the empirical half: at measured token costs the effect persists at intermediate scarcity and vanishes at the default budget. On the design half, the paper's position is that prices should be set proportional to measured service cost with a single exchange rate, which is what the cost-proportional arm implements, and that the heterogeneity of prices relative to costs is itself a knob — our tier schedule exaggerates heterogeneity, which is why its effect is larger. Proportionate pricing is implementable without new infrastructure because every service already carries a token meter, and the ratio between the two arms is then entirely a property of the menu rather than of the designer's credit schedule.

Q2: Is there a task beyond controlled 2Wiki where sharing the account significantly improves over separate budgets?

HotpotQA at k≈1.5: +2.9 [+1.8, +4.0] (W2). This is a non-oracle multi-hop task with a different document distribution and no gold placement in the menu.

Thank you for the detailed response. The new HotpotQA, fresh-seed, and cost-based pricing results help address my concerns about retrieval and pricing.

However, the gains remain largely limited to multi-hop QA under intermediate-budget constraints. Results on mathematics, science, forecasting, and larger budgets remain small or statistically unclear. Also, despite the reorganization, the current paper remains difficult to follow, and this concern is echoed in another review. Given the narrow empirical benefit and limited novelty, I am keeping my score at weak reject.

Thank you for the careful assessment and for reviewing the new HotpotQA, fresh-seed, and cost-based pricing results. We appreciate your acknowledgment of these additions, and we understand the remaining concerns about limited empirical gains and clarity. We will use this feedback to better frame the scope and improve readability. Thank you again for your time and constructive comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.