acceptodds
Under review as a conference paper at ICLR 2027

APIX: Sustaining an Open LLM API Market Without Centralized Provider Admission

Abstract

Autonomous agents increasingly rely on LLM API services. Yet existing API platforms adopt centralized admission, limiting two-sided openness by restricting provider participation and curating user choice. A decentralized market is a potential solution, but opaque API services introduce a lemons problem: a plausible response alone does not reveal whether the provider actually delivered the advertised service. The key challenge is therefore how to generate credible evidence of provider behavior and sustain market viability in such an open setting. In response, we present APIX, a decentralized, permissionless market for LLM API services. APIX uses auction-based matching to coordinate heterogeneous user demands with provider-defined services, a multidimensional reputation system to support informed user selection, and peer-relative consistency evaluation to assess whether delivered services are consistent with providers' advertised capabilities. We evaluate APIX through an LLM-agent-based market simulation that captures the co-evolution of participant strategies and market conditions: APIX reduces opportunistic-profit share from 26.4% to 0.5% and improves service satisfaction from 90.0% to 99.4%. Our blockchain-based prototype further incurs less than 300ms of additional latency per call over direct API access, corresponding to only 3% overhead for the 8B model. Overall, these results show that APIX can sustain a reliable and two-sided-open LLM API market without centralized admission.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Decentralized Markets, LLM API Services, Multi-Agent Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Skn2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

APIX is a decentralized market system for LLM API services. It lets users submit service requests detailing their LLM service requirements and budget, and providers respond by bidding on the request (both on the blockchain). After that, users can review bids and reputation scores, select a provider, and perform the request off-chain, before submitting the outcome of the service on-chain. Importantly, a consistency score is assessed for different providers, where scores demonstrate how a model delivers relative to similarly advertised capabilities of other models (to prevent provider under-provisioning).

Critical strengths

Did a good job motivating the problem and why similar solutions are inadequate - showed why computing a reputation score is non-trivial for LLM output, as compared with other computing services sold in marketplaces. The peer-relative evaluation system is fair in the sense that it only evaluates models with respect to their stated capabilities, and not their absolute capabilities with respect to the mean. This makes for a scoring system that is more relevant to users, as they are only concerned with whether or not a provider is delivering what they promise. Market workflow is simple from the perspective of both the user and the provider, and works well as a service broker. The user only has to submit their requests to the blockchain and look for bids referencing that request, then submit their outcomes back to the chain. Providers must only look for requests and submit their bids. APIX handles the service agreement between the user and the provider.

Critical weaknesses

Users might not want to opt in to submitting outcomes to the system if it involves sensitive or proprietary data, especially since they might be unfamiliar with the different providers that will analyze the output and their privacy policy. Moreover, is it possible to prove to users that their data will be anonymized or not trained on? For peer-relative consistency evaluation, there is evidence to suggest that LLMs are able to tell which LLM models generated which output, based on the distribution of that output. This is especially crucial when considering that the judge selection isn’t uniform, and prefers providers with high consistency scores. Many of the high-reputation providers (a majority) may have a specific bias against a certain other provider (e.g., to remove them from the market to increase their own share), and intentionally push their scores down to alienate them from the users. Certain exploit patterns weren’t necessarily covered by either the single- or two- sided test with agents. While I can’t be sure that these weren’t tested, I didn’t see any reasoning as to why certain advanced strategies wouldn’t allow providers to take control of the market. For example, a set of providers could over-provision to gain consistency score, and push other providers’ consistency scores down, then take the majority of judge positions for consistency scoring, and begin under-provisioning, while still detecting their model’s output (as mentioned above), and rating it highly (to keep their consistency score high), even if it isn’t a better response. This would allow them to profit heavily in the long run by underprovisioning, while other models are accurately provisioning their systems. Either running more experiments to demonstrate that specific exploits like these don’t work, or using a game theory model to show that an honest provider strategy is unexploitable would be more convincing. It seems like through competing with each other, providers might establish a universal equilibrium point where everyone either over- or under-provisions their resources. This equilibrium will be found by the nature of this dynamic market, but in this market, it seems like the free variable is performance instead of cost, which makes it less clear for the consumers what they’re getting (Providers might be tweaking performance to compete, which isn’t transparent, instead of cost, which is).

Questions for authors

How do the reputation updates work? For 2 outputs, are the positive and negative updates proportional to the difference in quality between the 2 results? If so, then how are they proportional? If not (i.e., it is a constant positive and negative update regardless of the difference between 2 results), is it because assigning numerical scores with LLMs is difficult and error-prone? For peer-relative evaluation, what if exact matches of a provider’s capabilities cannot be found, and a provider is systematically compared to better models, thus losing consistency scores? Figure 5 and the text reference to Figure 5 seem to disagree. It is said, “As shown in Figure 5, once a lease serves more than 100 calls, APIX adds less than 300 ms of latency per request compared with direct API invocation”, but the Figure makes it look like 3 seconds of latency per call.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer Skn2

We thank the reviewer for the detailed analysis and the concrete attack scenarios. We implemented each scenario as an adversary, and address the comments point by point below.

W1. Privacy of user data.

Users no longer need to share sensitive calls. We add a privacy flag: users can mark calls as private; private calls are never sampled, so they are never shared with peers or judges, and the flag stays on the user side. Private users still benefit from the evaluation of other users' calls (detail in Appendix D, Appendix F{\color{#2563EB}\text{Appendix D, Appendix F}}).

[Table R1]{\color{#C05A18}\text{[Table R1]}} One host substitutes while a share of the users marks all of its calls private (10 seeds, base sampling rate 5%). A host is "filtered" when it is confidently below the threshold that users set, so they stop selecting it. A request is "satisfied" when the served model meets its requirement. Detection is unaffected when up to half of the users are private and is still 92% with three quarters, and private users stay protected by the evaluation of other users' calls (92–100% of their requests are satisfied).

Share of users that mark all calls private25%50%75%
Cheating epochs in which the host is filtered100%99%92%
Requests of private users that are satisfied99.9%97.0%91.5%

Because the flag stays on the user side, a host that tries to cheat only on private calls cannot tell them apart, and is filtered in 85% of its cheating epochs.

On proving anonymization or no training: the protocol cannot prove what a provider does with data it has received. APIX limits exposure instead, since private calls reach only the provider that the user chose, and TEE attestation can be added as an optional reputation dimension.

W2. Fingerprinting and biased high-reputation majorities.

We agree with both observations: LLMs can recognize which model produced an output, and reputation-weighted selection lets a high-reputation group amplify itself. We therefore do not rely on judge anonymity, no longer select judges by reputation, and bound what a biased group can do (detail in Sec. 3.4, Appendix D{\color{#2563EB}\text{Sec. 3.4, Appendix D}}):

  • Seats follow stake. Judges post stake, and committee seats are drawn in proportion to stake rather than reputation, so a high-reputation group cannot amplify itself. The fee does not depend on agreement.
  • No reward above the declared model. A provider is filtered only when it is confidently below users' threshold, and scoring above the declared model earns nothing. Favorable votes therefore cannot lift a partner in the ranking.
  • Contested verdicts are re-judged. A decisive verdict between two hosts of the same model is re-judged by a larger committee, and judges whose votes are repeatedly overturned lose their stake. To push an honest provider down, a coalition must hold both committees.

[Table R2]{\color{#C05A18}\text{[Table R2]}} Colluding judges that recognize their members' outputs perfectly, favor them, and vote against honest hosts (10 seeds). Four hosts of the upper tiers, where few judges are eligible, hold 5% to 30% of all stake. The coalition gains nothing while it holds up to a third of the judge seats of any tier; it gains only once it holds half or more, which is outside the assumption of an honest stake majority among the judges of each tier. With 100 providers, the same four hosts hold 3% of all judge seats and gain nothing.

Coalition's share of all stake5%10%20%30%
Largest share of a tier's judge seats held by the coalition19%32%50%62%
Coalition's share of service profit (14% when it votes honestly)14%14%21%48%
Honest hosts: epochs filtered ↓\downarrow0%0%0.4%3.8%
AuthorsOpenReview ↗

Reply to Reviewer Skn2

W3. Over-provision, take over the judge seats, then under-provision.

We implemented exactly this strategy, with 6 of 20 providers holding 30% of stake. The two steps that are specific to it no longer work, so the build-up is a pure cost: the cartel ends at −223k, against +26k when it behaves honestly.

[Table R3]{\color{#C05A18}\text{[Table R3]}} The three-phase cartel, phase by phase (10 seeds). Over-provisioning neither pushes honest providers down nor wins additional demand, and reputation does not convert into judge seats, so the build-up buys nothing and costs 295k. What remains in the last phase is under-provisioning by a bloc that holds half of the judge seats of a tier (3 of the 6 hosts of the 8B tier), which is the case outside the honest-majority assumption in W2. A bloc with a minority of the seats cannot shield its members: a substituting host with two accomplice judges is filtered in 97% of its cheating epochs.

PhaseWhat it is meant to buyResult
1. Over-provision to gain score and push peers downa higher rank and filtered competitorshonest providers filtered: 0%; the cartel's share of requests is the same as when it behaves honestly
2. Convert reputation into judge seatscontrol of the committeesthe cartel's seat share stays at 23%, the same as when it behaves honestly
3. Under-provision while voting as a blocrecovery of the cost of phases 1 and 2not recovered within the run: −223k overall, against +26k when honest

On the game-theoretic side, as the reviewer suggests, we add a formal analysis (detail in Appendix E{\color{#2563EB}\text{Appendix E}}): propositions on truthful declaration, deterrence of under-delivery by a bond, and the influence of a judge coalition as a function of its stake.

W4. Performance as a hidden competitive variable.

We agree that competing on a hidden variable would be opaque to users. We updated the mechanism so that performance is visible and bounded, from above by the declared model and from below by the tolerance that users set: each provider's estimate is published with a confidence interval, over-provisioning earns nothing, and substitution is filtered. Competition therefore returns to price and observable service dimensions.

[Table R4]{\color{#C05A18}\text{[Table R4]}} The three ways in which providers could drift in performance, and what happens to each (10 seeds). Over-provisioning earns nothing, under-provisioning is filtered (99% of the epochs of a substituting host), and a tier that drifts down together is still detected (70% of its epochs, 94% of its hosts).

DriftMechanismResult
Over-provisioningscoring above the declared model earns nothing, so a provider with a better model does best by declaring itno honest host is filtered when a peer over-provisions
Under-provisioning by one providerit is filtered once it is confidently below a user's threshold99% of its cheating epochs filtered; profit 1.1k against 5.9k when honest
Under-provisioning by a whole tiercross-tier checks: a quarter of the comparisons pair a provider with an adjacent tier and are scored against a public calibration table, which anyone can reproduce from the public weightsall 8 hosts substitute: 70% of their epochs filtered, 94% of the hosts at least once

Q1. How do reputation updates work?

The reviewer's reading is correct: the update was a constant ±1, precisely because numeric scores from LLMs are unreliable. We now make it proportional without asking judges for a number. Judges still give pairwise verdicts (A, B or tie, with correctness first), and we aggregate them with a Bradley–Terry model, so each verdict counts in proportion to how surprising it is given both providers. The estimate has a confidence interval, and a provider is filtered only when its upper bound is below the user's threshold. As a result, no honest provider is filtered when nobody cheats (detail in Appendix D{\color{#2563EB}\text{Appendix D}}).

AuthorsOpenReview ↗

Reply to Reviewer Skn2

Q2. What if no exact peer exists?

Such a provider does not lose score for meeting a stronger model. In a cross-tier comparison (W4), the verdict is scored against the outcome expected between the two declared models, not against a win.

[Table R5]{\color{#C05A18}\text{[Table R5]}} Comparison partners by type of declared model; the label is shown to users. A public model is compared within its tier and, through the calibration table, with adjacent tiers, so even a public model with a single host receives a score. Only a private model with a single host receives none, and it is labeled as unverifiable.

Declared modelLabelCompared with
Public, any number of hostsCalibratedhosts of the same model; adjacent tiers in a quarter of the comparisons, scored against the calibration table
Private, several hostsPeer-onlyhosts of the same model only
Private, single hostUnverifiablenone; no score is published

Q3. Fig. 5 vs. the text.

Thank you for catching this. Fig. 5 shows end-to-end latency, about 3 s of which is model inference, not APIX's own overhead. We re-measured the overhead separately, corrected the text, and added the decomposition as a figure (detail in Sec. 5.1, Appendix J{\color{#2563EB}\text{Sec. 5.1, Appendix J}}).

[Table R6]{\color{#C05A18}\text{[Table R6]}} APIX's own overhead. APIX adds 0.31 s to every call plus a one-time lease setup of 20.5 s under our setting, i.e., 20.5/N + 0.31 s per call for a lease of N calls; with a lease kept open, only 0.31 s remains.

Overhead added by APIXWhen it is paidMeasured
Lease setup timeonce per lease20.5 s (4 blocks)
Added latency per callevery call0.31 s (p90 0.35 s)
Reviewer_jLKzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes APIX, a decentralized LLM API marketplace combining auction theory, reputation theory, and peer-relative evaluation. Experiments show that APIX reduces provider misbehavior and improves market reliability.

Critical strengths
  1. Well-written: The paper is clearly written and well-organized, making it easy to follow the motivation, methodology, and experimental results.
  2. Interesting Topic: The paper explores an interesting and practically relevant problem of building a decentralized LLM API market while addressing provider trust and service reliability.
Critical weaknesses
  1. Further theoretical analysis should be included such as incentive compatible
  2. The evaluation primarily focuses on market viability rather than overall social welfare. While APIX improves service satisfaction and provider participation, it remains unclear whether these improvements translate into higher social welfare, especially considering the additional evaluation costs borne by users.
  3. Individual Rationality: Although APIX allows users and providers to participate voluntarily, the paper does not formally establish individual rationality.
  4. Minor: The proposed auction mechanism is relatively straightforward and lacks a design tailored to the unique characteristics of LLM API markets; while existing mechanisms such as second-price and reverse auctions are designed for specific economic settings, the authors could explore a novel auction mechanism that explicitly considers hidden service quality and strategic provider behavior.
Questions for authors

see in weakness

4 comments
AuthorsOpenReview ↗

Reply to Reviewer jLKz

We thank the reviewer for the positive assessment and the suggestions on theory and welfare. We address the comments point by point below.

W1. Theoretical analysis, e.g., incentive compatibility.

We added a formal analysis (detail in Appendix E{\color{#2563EB}\text{Appendix E}}). What users cannot observe in APIX is the model that a provider actually serves, so incentive compatibility means that declaring this model and serving it is the provider's best strategy (prices are bid competitively and paid as bid). We establish it from both sides.

The analysis builds on an updated evaluation mechanism, in which every claim is backed by a deposit and checked on a sample (detail in Sec. 3.4, Appendix D{\color{#2563EB}\text{Sec. 3.4, Appendix D}}). Providers post a bond, which is forfeited when they are "filtered", i.e., found confidently below the threshold that users set.

  • Truthful declaration (Proposition 1). A provider is tested only for being below its declared model, and scoring above it earns nothing. Serving a stronger model than declared, or declaring a weaker model than served, is therefore dominated.
  • Honest delivery (Proposition 2). Under-delivery is unprofitable when the bond plus the future profit lost on being filtered exceeds the saving made before detection. This gives a minimum deterrent sampling rate, which decreases with the bond. A deviation too small to be filtered stays within the tolerance that users set.

Together, the two propositions bound delivered quality between the declared level and the tolerance that users set. The appendix further bounds the influence of a judge coalition by its stake (Proposition 3) and shows that a tier whose hosts all deviate together is still identified (Proposition 4). We check honest delivery empirically by letting one host deviate.

[Table R1]{\color{#C05A18}\text{[Table R1]}} Profit of one host over the run (48 epochs, in the market's currency units) under each behavior, with the share of its cheating epochs in which it is filtered (10 seeds, base sampling rate 5%). Each of these deviations earns 3–5× less than honest provisioning (1.1k–2.1k against 5.9k), even before any bond is forfeited (detail in Appendix F{\color{#2563EB}\text{Appendix F}}).

Behavior of the hostIts profit over the runCheating epochs in which it is filtered
Honest5.9k–
Substitutes on every call1.1k99%
On-off cheating2.1k96%
Re-registers when filtered1.4k88%

W2. Social welfare including evaluation cost.

We now report social welfare net of evaluation cost, defined as user value minus serving cost minus the real cost of evaluation (detail in Appendix I{\color{#2563EB}\text{Appendix I}}). In short, at the default rate evaluation costs 6% of welfare when nobody deviates and raises welfare by 38% when one host substitutes.

Users pay for evaluation, so it must be cheap. We therefore make sampling adaptive: calls are sampled at a low base rate, and densely only while a provider's status is undecided, i.e., it is new or looks suspicious.

[Table R2]{\color{#C05A18}\text{[Table R2]}} Social welfare relative to the same scenario without evaluation (= 100%), for four base sampling rates (10 seeds). Values above 100% mean that evaluation raises welfare net of its cost. At the default rate, the gain when a host substitutes (+38%) is several times the cost when nobody deviates (−6%). Adaptive sampling keeps fees at 6.7% of user spend, and welfare falls as the base rate grows.

Base sampling rateEvaluation fees (% of user spend)Welfare: no attackWelfare: one host substitutes
0.023.2%97%141%
0.056.7%94%138%
0.111.8%89%137%
0.221.0%78%122%
  • Where the gain comes from. Without evaluation, users who are served a substitute leave the market: requests over the run fall from 3072 to 1948, against 2506 with APIX.
  • Choice of the default. We set the base rate to 0.05. A lower rate saves little and weakens detection: at 0.02, a fully substituting tier is filtered in 36% of its epochs instead of 70%. The committee size matters little: fees are 4.8% with one judge and 7.1% with five.
AuthorsOpenReview ↗

Reply to Reviewer jLKz

W3. Individual rationality.

We now establish individual rationality formally, as Proposition 5, for each role (detail in Appendix E{\color{#2563EB}\text{Appendix E}}).

[Table R3]{\color{#C05A18}\text{[Table R3]}} Individual rationality by role (Proposition 5). Every role is at least as well off participating honestly as staying out.

RoleWhy participation is rationalEvidence
Providerbids at a price that covers its cost and is paid as bid; an honest provider is not filtered, so its bond is returnedhonest providers filtered: 0%
Userparticipates when the value of a call exceeds its price plus the evaluation premiumevaluation fees: 6.7% of user spend
Judgereceives a posted fee of 1.2 times its judging cost, independent of how others votethe fee covers the cost by construction

Minor. Auction design.

We agree that hidden quality should shape the auction, and the updates above give APIX such a design: matching is now a procurement auction with prequalification and performance bonds. Consistency acts as an eligibility test and providers post a bond. Eligible providers compete on price and observable service dimensions under each user's scoring rule, and hidden quality is handled by bonded ex-post checks rather than by the allocation rule. This is the standard remedy for hidden action in procurement, and it keeps each user in control of its own price–quality trade-off.

Dear all,

After carefully reading the response from authors and the paper again, I decide to increase my score, although some points are not very solid. Thanks!

AuthorsOpenReview ↗

Reply to Reviewer jLKz

Dear Reviewer,

Thank you for your careful reconsideration and constructive feedback. We sincerely appreciate your time and effort, and will further improve the paper based on your comments.

Reviewer_xNDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APIX, a decentralized marketplace for LLM API services that allows providers to participate without centralized admission. It combines auction-based service matching, multidimensional reputation, and peer-relative evaluation to help users select providers and discourage dishonest service delivery. The authors evaluate the system through LLM-agent-based market simulations and a blockchain prototype, showing improvements in service satisfaction, provider participation, and resistance to opportunistic behavior.

Critical strengths
  1. The proposed framework considers both user preferences and provider incentives, useful for understanding how an open LLM API marketplace might operate.

  2. Generally, this paper is well-structured and easy to follow.

  3. The research problem in this paper is interesting and practical, as users may have difficulty verifying whether an LLM API provider actually delivers the advertised service.

Critical weaknesses
  1. To my understanding, the proposed framework mainly combines existing ideas, including auction-based matching, reputation systems, and peer-based evaluation, and the authors do not clearly explain what is technically new beyond their integration into an LLM API marketplace.

  2. APIX relies on comparing responses from providers advertising similar capabilities, but lower-quality responses do not necessarily indicate model substitution, and the mechanism may fail when multiple providers misrepresent their capabilities or judges behave dishonestly.

  3. The experiments mainly compare APIX with versions that disable evaluation or change judge selection, without sufficiently comparing it against alternative reputation, auditing, or provider verification mechanisms.

Questions for authors
  1. How would the framework handle Sybil attacks, reputation manipulation, or dishonest evaluation results in a permissionless setting?

  2. How does APIX perform when the number of users and providers becomes much larger, particularly in terms of transaction costs, evaluation overhead, and reputation update latency?

  3. What is the main technical novelty of APIX compared with existing decentralized marketplaces that already use auctions, reputation systems, and peer-based evaluations?

  4. Could the authors compare APIX against existing reputation-based selection methods, rather than mainly comparing it with configurations that disable parts of APIX?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer xNDW

We thank the reviewer for the thoughtful comments. We address them point by point below.

W1/Q3. Technical novelty.

We agree that auctions, reputation and peer evaluation exist individually. What is new is the problem they are made to solve here: a market in which the delivered service cannot be verified. Our contributions are:

  1. A new verification target. Existing decentralized marketplaces trade resources whose attributes are readily verifiable (e.g., GPU capacity on Akash or Render), and validator networks such as Bittensor reward agreement with the majority. Verifiable-inference approaches check that a computation ran or that outputs are identical, which requires open weights, deterministic outputs or a trusted auditor (comparison in Sec. 2{\color{#2563EB}\text{Sec. 2}}). An LLM API offers none of these: a provider may resell a closed model, serve a private fine-tune, or run a different serving stack. APIX instead verifies delivered quality relative to the declared model, without a trusted participant.
  2. One mechanism for strategic providers and strategic judges. Every claim is bonded, checked on a sample of real work that cannot be told apart from unchecked work, tested one-sidedly with a confidence level, and has an economic consequence (detail in Sec. 3.4, Appendix D{\color{#2563EB}\text{Sec. 3.4, Appendix D}}). A provider's claim is its declared model, checked by pairwise comparison within its tier and, across tiers, against a public calibration table that anyone can reproduce from the public weights. A judge's claim is its vote, checked by a larger committee. This also exposes a tier that substitutes collectively, which same-tier comparison cannot.
  3. A market-level finding. We study the market with co-evolving LLM agents rather than a static benchmark. Without behavioral evidence, users retreat to a few trusted providers, which capture 86–95% of profit. With APIX, honest unverified providers can compete, and that share falls to 16–26% (detail in Appendix H{\color{#2563EB}\text{Appendix H}}).
AuthorsOpenReview ↗

Reply to Reviewer xNDW

W2. Lower quality does not necessarily indicate substitution; multiple misreporting providers; dishonest judges.

We agree on all three points. For each, we updated the mechanism (detail in Sec. 3.4, Appendix D{\color{#2563EB}\text{Sec. 3.4, Appendix D}}) and tested it.

Lower quality is not substitution. APIX does not infer model identity. It only tests whether delivered quality is confidently below what the declared model delivers. We replace the ±1\pm 1 score with a one-sided confidence test: judges answer A, B or tie with correctness first, and a provider is "filtered" by users only when it is confidently below the threshold they set. We verified this on real responses: six hosts declare MiniMax-M3.1-flash and answer 150 MATH-500 problems; two secretly serve a weaker model and one a stronger model; three real LLM judges compare them (detail in Appendix G{\color{#2563EB}\text{Appendix G}}).

[Table R1]{\color{#C05A18}\text{[Table R1]}} Reputation rank computed from real responses and real LLM judges. The ranking follows answer accuracy: the stronger substitute ranks first and is not penalized, the three honest hosts come next, and the two weaker substitutes rank last. Honest pairs tie 91% of the time and no honest host is filtered.

HostAnswer accuracyRank by APIX reputation (1 = best)
Stronger substitute ×10.991
Honest ×30.96–0.972–4
Weaker substitutes ×20.89–0.935–6

Multiple misreporting providers. We add cross-tier checks: a quarter of the comparisons pair a provider with an adjacent tier and are scored against the public calibration table, so providers of one tier cannot cover for each other.

[Table R2]{\color{#C05A18}\text{[Table R2]}} Detection when hosts of one tier misreport together: of the 8 hosts that declare the 4B model, some secretly serve 0.6B (10 seeds, base sampling rate 5%). "Trusted auditor" is an ideal centralized auditor with a trusted reference (baseline B2 in Table R4), as an upper bound. APIX stays within 5 points of the trusted auditor, still detects a tier that has no honest host left (70% of its epochs, 94% of its hosts), and filters no honest host when nobody cheats.

Cheating hostsAPIX: cheating epochs in which the cheater is filteredAPIX: cheaters filtered at least onceTrusted auditor: cheating epochs in which the cheater is filtered
1 of 899%100%99%
4 of 879%95%84%
7 of 874%94%76%
8 of 870%94%n/a (no honest reference)
No cheating host: honest hosts filtered ↓\downarrow0%0%0%

Dishonest judges. We make judges stake-based and accountable. Seats are drawn in proportion to stake rather than reputation; scoring above the declared model earns nothing, so favorable votes cannot buy rank; and a verdict that contradicts the declarations is re-judged by a larger committee, with judges whose votes are repeatedly overturned losing their stake.

[Table R3]{\color{#C05A18}\text{[Table R3]}} Colluding judges that recognize their members' outputs perfectly, favor them, and vote against honest hosts (10 seeds). Four hosts of the upper tiers, where few judges are eligible, hold 5% to 30% of all stake. Dishonest judging gains nothing while the coalition holds up to a third of the judge seats of any tier; it gains only once the coalition holds half or more, which is outside the assumption of an honest stake majority among the judges of each tier. With 100 providers, the same four hosts hold 3% of all judge seats and gain nothing.

Coalition's share of all stake5%10%20%30%
Largest share of a tier's judge seats held by the coalition19%32%50%62%
Coalition's share of service profit (14% when it votes honestly)14%14%21%48%
Honest hosts: epochs filtered ↓\downarrow0%0%0.4%3.8%
AuthorsOpenReview ↗

Reply to Reviewer xNDW

W3/Q4. Comparison with existing reputation and auditing mechanisms.

We added two external baselines (detail in Sec. 4.3, Appendix F{\color{#2563EB}\text{Sec. 4.3, Appendix F}}):

  • B1, user-feedback reputation. A Beta reputation over user ratings, the standard form of reputation-based selection, with and without fake ratings.
  • B2, an ideal centralized auditor. Every sampled call is compared with a trusted reference by a trusted judge. It is an upper bound that requires a trusted party.

[Table R4]{\color{#C05A18}\text{[Table R4]}} Share of cheating epochs in which the cheater is filtered, for APIX and the two baselines (10 seeds, base sampling rate 5%); the last two rows give the false positives and the cost. APIX filters 70–99% of cheating epochs where user-feedback reputation filters 0–54%, and nothing once ratings are faked. It is on par with the trusted auditor for full and tier-wide substitution at less than half of the auditor's fees, and still works when no honest reference exists.

ScenarioAPIXB1: user-feedback reputationB1 with fake ratingsB2: trusted auditor
One host substitutes99%54%0%99%
It substitutes on half of its calls81%0%0%94%
7 of 8 hosts of a tier substitute74%11%0%76%
All 8 hosts of a tier substitute70%10%0%n/a (no honest reference)
No attack: honest hosts filtered ↓\downarrow0%0%–0%
Evaluation fees (% of user spend) ↓\downarrow6.7%0%0%14.3%

Q1. Sybil attacks, reputation manipulation and dishonest evaluation.

In a permissionless setting identities are free, so no defense in APIX depends on identity: each is tied to stake or to evidence. Judges post stake, and providers post a bond that is forfeited when they are filtered. We implemented the attacks below as adversaries.

[Table R5]{\color{#C05A18}\text{[Table R5]}} The three attack classes in the question, with the defense and the measured result (10 seeds; detail in Appendix F{\color{#2563EB}\text{Appendix F}}). Sybil identities gain nothing when splitting stake and are filtered again after re-registering (88% of epochs); reputation manipulation is filtered in 96–97% of its epochs or has no effect; fake judges are kept out by the qualification exam; and colluding judges gain nothing with up to a third of a tier's seats.

ClassAttackDefenseResult
SybilSplitting stake across identitieseach judge seat is an independent draw in proportion to stakeno gain by construction: the seats a coalition wins depend only on its total stake
SybilRe-registering when filtereda new identity needs a new bond and is checked denselyfiltered in 88% of its epochs; earns 4× less than when honest
Reputation manipulationOn-off cheating (every 4 epochs / every epoch)estimates are published four times per epoch96% / 97% of cheating epochs filtered
Reputation manipulationOver-provisioning to push peers downscoring above the declared model earns nothingno honest host filtered
Reputation manipulationFake user ratingsonly completed evaluations countno effect
Dishonest evaluationFake judges that claim a large modeljudges must pass a qualification examno target filtered (83% of target epochs without the exam)
Dishonest evaluationColluding judgesone-sided test; larger committee on contested verdictsno gain with up to a third of a tier's seats (Table R3)
Dishonest evaluationCopying the majority votethe judge fee does not depend on agreementnot rewarded

Q2. Scalability.

Costs grow with the number of leases and the sampling rate, not with the number of participants. Sampling is adaptive: providers are checked at a 5% base rate, and densely only while their status is undecided. We verified this in a market five times larger (100 providers, 80 users; 5 seeds).

[Table R6]{\color{#C05A18}\text{[Table R6]}} How each cost scales, measured with 20 and with 100 providers. Evaluation fees stay at the same level in a market five times larger (6.4% of user spend, against 6.7% in the default market). On-chain cost is paid per lease rather than per call, and reputation updates need no per-call transaction. Detection holds at this scale: no honest host is filtered, a single substituting host is filtered in 100% of its epochs, and a fully substituting tier in 97%.

CostGrows withMeasured
On-chain transactionsnumber of leases, not callsone setup per lease; calls need no transaction
Evaluation fees (% of user spend)sampling rate, not market size6.7% with 20 providers, 6.4% with 100
Comparisons per provider per epochsampling rate, not market size0.24 with 20 providers, 0.30 with 100
Reputation update latencynothing (fixed schedule)estimates published four times per epoch; no per-call transaction

Thank you for the detailed rebuttal and additional experiments. However, I remain unconvinced that the contribution meets the bar for ICLR and will maintain my score.

Reviewer_N9kZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces APIX, a decentralized marketplace where users can find and choose LLM API providers. To prevent providers from offering low-quality services, the system uses auctioning, reputations and comparisons between model outputs. Tests using simulations and a blockchain prototype showed improved service satisfaction and reduced profits for dishonest providers.

Critical strengths

Interesting problem: Creating an open marketplace for LLM API services without a centralized provider is the problem investigated in this paper. This is important because providers can claim to have better models.

Fully designed system: Auctioning, reputations, and peer comparisons are combined into one system. The authors also built a blockchain prototype and tested its overhead, helping to show that the method could be practical.

Critical weaknesses

Low verification for dishonest providers: By comparing outputs from providers claiming to have similar models, the system does not verify which model was actually used. It may fail to detect cases where several providers use the same low quality model or collude. Given the lack of centralized admission, this is a major issue.

Simple simulations: The main results come from a simulated marketplace with only 16 users and 20 providers. Pre-defined performance profiles were used to generate service results, and the satisfaction metric does not directly relate to answer quality. The results may therefore be unrepresentative of a real marketplace.

Not fully tested in real life: Testing of the prototype took place on a local blockchain, with low latency mainly seen when a lease included more than 100 API calls. Smaller workloads, real network conditions and attacks on the reputation system are not fully explored. This makes the practical claims less convincing.

Questions for authors

How would this system deal with providers colluding or multiple providers in the same group using low quality models? Testing these cases would help establish whether peer comparison is reliable.

Could the system be tested using real API responses and the quality of the answers assessed? Results from multiple simulation runs would also help show whether the improvements are consistent.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer N9kZ

We thank the reviewer for the valuable feedback. We address the comments point by point below.

W1/Q1. Collective substitution and collusion.

Identifying the exact served model is challenging, so APIX is designed around a complementary goal: whether the delivered quality is confidently below what the declared model delivers. We agree that same-tier comparison alone cannot see a tier that substitutes together, and that it relies on honest judges. We updated the evaluation mechanism for these two cases and tested both (detail in Sec. 3.4, Appendix D{\color{#2563EB}\text{Sec. 3.4, Appendix D}}). In short, a tier is still detected when all of its hosts substitute, and colluding judges gain nothing while they hold up to a third of the judge seats of any tier.

Collective substitution. We add cross-tier checks. A quarter of the sampled comparisons now pair a provider with a provider of an adjacent tier, and the verdict is scored against a public calibration table of how genuine models of the two tiers compare. Anyone can reproduce the table from the public weights, so no trusted participant is needed. A tier that substitutes together deviates from the table, even when it has no honest host left.

[Table R1]{\color{#C05A18}\text{[Table R1]}} Detection when hosts of one tier substitute together: of the 8 hosts that declare the 4B model, some secretly serve 0.6B (10 seeds, base sampling rate 5%). A host is "filtered" when it is confidently below the threshold that users set, so they stop selecting it. "Trusted auditor" is an ideal centralized auditor with a trusted reference, as an upper bound. APIX stays within 5 points of the trusted auditor, still detects a tier that has no honest host left (70% of its epochs, 94% of its hosts), and filters no honest host when nobody cheats.

Cheating hostsAPIX: cheating epochs in which the cheater is filteredAPIX: cheaters filtered at least onceTrusted auditor: cheating epochs in which the cheater is filtered
1 of 899%100%99%
4 of 879%95%84%
7 of 874%94%76%
8 of 870%94%n/a (no honest reference)
No cheating host: honest hosts filtered ↓\downarrow0%0%0%

Collusion. We make judges stake-based and accountable, and make the score one-sided:

  • Judges post stake, and committee seats are drawn in proportion to stake rather than reputation, so a group cannot amplify itself by rating its members highly.
  • A provider is filtered only when it is confidently below users' threshold, and scoring above the declared model earns nothing, so favorable votes cannot lift a partner in the ranking.
  • A decisive verdict between two hosts of the same model is re-judged by a larger committee, and judges whose votes are repeatedly overturned lose their stake.

[Table R2]{\color{#C05A18}\text{[Table R2]}} Colluding judges that recognize their members' outputs perfectly, favor them, and vote against honest hosts (10 seeds). Four hosts of the upper tiers, where few judges are eligible, hold 5% to 30% of all stake. Collusion gains nothing while the coalition holds up to a third of the judge seats of any tier; it gains only once the coalition holds half or more, which is outside the assumption of an honest stake majority among the judges of each tier. With 100 providers, the same four hosts hold 3% of all judge seats and gain nothing.

Coalition's share of all stake5%10%20%30%
Largest share of a tier's judge seats held by the coalition19%32%50%62%
Coalition's share of service profit (14% when it votes honestly)14%14%21%48%
Honest hosts: epochs filtered ↓\downarrow0%0%0.4%3.8%
AuthorsOpenReview ↗

Reply to Reviewer N9kZ

W2/Q2. Simulation realism, answer quality, and repeated runs.

We added repeated runs, a metric of answer quality, an evaluation on real API responses with real LLM judges, and a market five times larger.

Repeated runs. Scripted experiments use 10 seeds, and we reran the LLM-agent settings with 3 seeds (detail in Appendix H{\color{#2563EB}\text{Appendix H}}).

Answer quality. We also score each call by the probability that its answer beats one from the declared model, using strengths fitted to judgments of real responses. A call is "clearly worse" if this probability is below 0.4.

[Table R3]{\color{#C05A18}\text{[Table R3]}} LLM-agent market with APIX at its default 5% base sampling rate (mean ±\pm std over 3 seeds). "Under-provisioning episodes" counts the provider-episodes in which a provider serves a weaker model than it declares. APIX reduces under-provisioning by 3–6× and clearly worse answers in all three settings.

SettingUnder-provisioning episodes ↓\downarrow: no evaluation →\rightarrow APIXCalls with a clearly worse answer ↓\downarrow: no evaluation →\rightarrow APIX
Case 1: single-sided adaptation40.0 ±\pm 2.8 →\rightarrow 11.3 ±\pm 6.820.9 ±\pm 1.2% →\rightarrow 6.4 ±\pm 4.5%
Case 2: two-sided adaptation48.7 ±\pm 15.4 →\rightarrow 15.3 ±\pm 2.59.9 ±\pm 2.7% →\rightarrow 6.6 ±\pm 3.0%
Case 2 with GLM agents15.3 ±\pm 8.4 →\rightarrow 2.7 ±\pm 0.97.2 ±\pm 5.0% →\rightarrow 0.0 ±\pm 0.0%

Real API responses. To go beyond pre-defined performance profiles, we ran the evaluation on real responses with real judges. Six hosts that declare MiniMax-M3.1-flash answer 150 MATH-500 problems through hosted APIs: three are honest, two secretly serve the weaker Nemotron-3.5, and one secretly serves the stronger DeepSeek-V4.1. Three LLM judges (Kimi-K3, GLM-5.3, gpt-oss-120b) answer A, B or tie, with correctness first (detail in Appendix G{\color{#2563EB}\text{Appendix G}}).

[Table R4]{\color{#C05A18}\text{[Table R4]}} Reputation rank computed from real responses and real LLM judges. The ranking follows answer accuracy: the stronger substitute ranks first and is not penalized, the three honest hosts come next, and the two weaker substitutes rank last. Honest pairs tie 91% of the time and no honest host is filtered.

HostAnswer accuracyRank by APIX reputation (1 = best)
Stronger substitute ×10.991
Honest ×30.96–0.972–4
Weaker substitutes ×20.89–0.935–6

Scale. In a market five times larger (100 providers, 80 users; 5 seeds), no honest host is filtered, a single substituting host is filtered in 100% of its epochs, and a fully substituting tier in 97% (every host at least once). Evaluation fees are 6.4% of user spend, the same level as in the default market (6.7%).

AuthorsOpenReview ↗

Reply to Reviewer N9kZ

W3. Real-world testing.

We re-measured the overhead, including for small workloads, and tested six further attacks on the reputation system.

Overhead, small workloads and network conditions. We re-measured the prototype with a hosted LLM API as the backend (detail in Sec. 5.1, Appendix J{\color{#2563EB}\text{Sec. 5.1, Appendix J}}).

[Table R5]{\color{#C05A18}\text{[Table R5]}} Prototype overhead. Under our setting, APIX adds 0.31 s to every call plus a one-time lease setup of 20.5 s, i.e., 20.5/N + 0.31 s per call for a lease of N calls. A lease can be kept open and reused across small workloads, which leaves 0.31 s per call. On a public network, the setup time follows the chain's block time, and calls stay off-chain.

Overhead added by APIXWhen it is paidMeasured
Lease setup timeonce per lease20.5 s (4 blocks)
Added latency per callevery call0.31 s (p90 0.35 s)
Total added latency per call, for a lease of 10 / 50 / 100 / 200 callssetup amortized over the lease2.68 / 0.74 / 0.59 / 0.47 s

Attacks on the reputation system. Besides the judge-side updates in W1, we add two provider-side ones. Providers post a bond, which is forfeited when they are filtered, so a new identity costs a new bond. Sampling is adaptive: new or suspicious providers are checked densely, and the others at the 5% base rate.

[Table R6]{\color{#C05A18}\text{[Table R6]}} Six further attacks on the reputation system, each with its defense and the measured result (10 seeds; detail in Appendix F{\color{#2563EB}\text{Appendix F}}). Each attack is either filtered in 81–97% of its cheating epochs or has no effect on honest hosts.

AttackDefenseResult
Re-registering when filtereda new identity needs a new bond and is checked denselyfiltered in 88% of its epochs; earns 4× less than when honest
On-off cheating (every 4 epochs / every epoch)estimates are published four times per epoch96% / 97% of cheating epochs filtered
Substituting on half of the callsadaptive sampling81% of cheating epochs filtered
Over-provisioning to push peers downscoring above the declared model earns nothingno honest host filtered
Fake judges that claim a large modeljudges must pass a qualification examno target filtered (83% of target epochs without the exam)
Fake user ratingsonly completed evaluations countno effect

Comments on the paper

1 comment

Dear Authors and Reviewers,

Thank you for supporting ICLR'27!

We are now entering the discussion phase. Please make the most of this time to exchange your views. I encourage each Reviewer, if you have not already done so, to read the other reviews and the Authors' responses. Please respond to the Authors and actively participate in the discussion to help clarify any remaining issues.

Thank you,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.