- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
APIX is a decentralized market system for LLM API services. It lets users submit service requests detailing their LLM service requirements and budget, and providers respond by bidding on the request (both on the blockchain). After that, users can review bids and reputation scores, select a provider, and perform the request off-chain, before submitting the outcome of the service on-chain. Importantly, a consistency score is assessed for different providers, where scores demonstrate how a model delivers relative to similarly advertised capabilities of other models (to prevent provider under-provisioning).
Critical strengths
Did a good job motivating the problem and why similar solutions are inadequate - showed why computing a reputation score is non-trivial for LLM output, as compared with other computing services sold in marketplaces. The peer-relative evaluation system is fair in the sense that it only evaluates models with respect to their stated capabilities, and not their absolute capabilities with respect to the mean. This makes for a scoring system that is more relevant to users, as they are only concerned with whether or not a provider is delivering what they promise. Market workflow is simple from the perspective of both the user and the provider, and works well as a service broker. The user only has to submit their requests to the blockchain and look for bids referencing that request, then submit their outcomes back to the chain. Providers must only look for requests and submit their bids. APIX handles the service agreement between the user and the provider.
Critical weaknesses
Users might not want to opt in to submitting outcomes to the system if it involves sensitive or proprietary data, especially since they might be unfamiliar with the different providers that will analyze the output and their privacy policy. Moreover, is it possible to prove to users that their data will be anonymized or not trained on? For peer-relative consistency evaluation, there is evidence to suggest that LLMs are able to tell which LLM models generated which output, based on the distribution of that output. This is especially crucial when considering that the judge selection isn’t uniform, and prefers providers with high consistency scores. Many of the high-reputation providers (a majority) may have a specific bias against a certain other provider (e.g., to remove them from the market to increase their own share), and intentionally push their scores down to alienate them from the users. Certain exploit patterns weren’t necessarily covered by either the single- or two- sided test with agents. While I can’t be sure that these weren’t tested, I didn’t see any reasoning as to why certain advanced strategies wouldn’t allow providers to take control of the market. For example, a set of providers could over-provision to gain consistency score, and push other providers’ consistency scores down, then take the majority of judge positions for consistency scoring, and begin under-provisioning, while still detecting their model’s output (as mentioned above), and rating it highly (to keep their consistency score high), even if it isn’t a better response. This would allow them to profit heavily in the long run by underprovisioning, while other models are accurately provisioning their systems. Either running more experiments to demonstrate that specific exploits like these don’t work, or using a game theory model to show that an honest provider strategy is unexploitable would be more convincing. It seems like through competing with each other, providers might establish a universal equilibrium point where everyone either over- or under-provisions their resources. This equilibrium will be found by the nature of this dynamic market, but in this market, it seems like the free variable is performance instead of cost, which makes it less clear for the consumers what they’re getting (Providers might be tweaking performance to compete, which isn’t transparent, instead of cost, which is).
Questions for authors
How do the reputation updates work? For 2 outputs, are the positive and negative updates proportional to the difference in quality between the 2 results? If so, then how are they proportional? If not (i.e., it is a constant positive and negative update regardless of the difference between 2 results), is it because assigning numerical scores with LLMs is difficult and error-prone? For peer-relative evaluation, what if exact matches of a provider’s capabilities cannot be found, and a provider is systematically compared to better models, thus losing consistency scores? Figure 5 and the text reference to Figure 5 seem to disagree. It is said, “As shown in Figure 5, once a lease serves more than 100 calls, APIX adds less than 300 ms of latency per request compared with direct API invocation”, but the Figure makes it look like 3 seconds of latency per call.
3 comments
Reply to Reviewer Skn2
W3. Over-provision, take over the judge seats, then under-provision.
We implemented exactly this strategy, with 6 of 20 providers holding 30% of stake. The two steps that are specific to it no longer work, so the build-up is a pure cost: the cartel ends at −223k, against +26k when it behaves honestly.
The three-phase cartel, phase by phase (10 seeds). Over-provisioning neither pushes honest providers down nor wins additional demand, and reputation does not convert into judge seats, so the build-up buys nothing and costs 295k. What remains in the last phase is under-provisioning by a bloc that holds half of the judge seats of a tier (3 of the 6 hosts of the 8B tier), which is the case outside the honest-majority assumption in W2. A bloc with a minority of the seats cannot shield its members: a substituting host with two accomplice judges is filtered in 97% of its cheating epochs.
| Phase | What it is meant to buy | Result |
|---|---|---|
| 1. Over-provision to gain score and push peers down | a higher rank and filtered competitors | honest providers filtered: 0%; the cartel's share of requests is the same as when it behaves honestly |
| 2. Convert reputation into judge seats | control of the committees | the cartel's seat share stays at 23%, the same as when it behaves honestly |
| 3. Under-provision while voting as a bloc | recovery of the cost of phases 1 and 2 | not recovered within the run: −223k overall, against +26k when honest |
On the game-theoretic side, as the reviewer suggests, we add a formal analysis (detail in ): propositions on truthful declaration, deterrence of under-delivery by a bond, and the influence of a judge coalition as a function of its stake.
W4. Performance as a hidden competitive variable.
We agree that competing on a hidden variable would be opaque to users. We updated the mechanism so that performance is visible and bounded, from above by the declared model and from below by the tolerance that users set: each provider's estimate is published with a confidence interval, over-provisioning earns nothing, and substitution is filtered. Competition therefore returns to price and observable service dimensions.
The three ways in which providers could drift in performance, and what happens to each (10 seeds). Over-provisioning earns nothing, under-provisioning is filtered (99% of the epochs of a substituting host), and a tier that drifts down together is still detected (70% of its epochs, 94% of its hosts).
| Drift | Mechanism | Result |
|---|---|---|
| Over-provisioning | scoring above the declared model earns nothing, so a provider with a better model does best by declaring it | no honest host is filtered when a peer over-provisions |
| Under-provisioning by one provider | it is filtered once it is confidently below a user's threshold | 99% of its cheating epochs filtered; profit 1.1k against 5.9k when honest |
| Under-provisioning by a whole tier | cross-tier checks: a quarter of the comparisons pair a provider with an adjacent tier and are scored against a public calibration table, which anyone can reproduce from the public weights | all 8 hosts substitute: 70% of their epochs filtered, 94% of the hosts at least once |
Q1. How do reputation updates work?
The reviewer's reading is correct: the update was a constant ±1, precisely because numeric scores from LLMs are unreliable. We now make it proportional without asking judges for a number. Judges still give pairwise verdicts (A, B or tie, with correctness first), and we aggregate them with a Bradley–Terry model, so each verdict counts in proportion to how surprising it is given both providers. The estimate has a confidence interval, and a provider is filtered only when its upper bound is below the user's threshold. As a result, no honest provider is filtered when nobody cheats (detail in ).
Reply to Reviewer Skn2
Q2. What if no exact peer exists?
Such a provider does not lose score for meeting a stronger model. In a cross-tier comparison (W4), the verdict is scored against the outcome expected between the two declared models, not against a win.
Comparison partners by type of declared model; the label is shown to users. A public model is compared within its tier and, through the calibration table, with adjacent tiers, so even a public model with a single host receives a score. Only a private model with a single host receives none, and it is labeled as unverifiable.
| Declared model | Label | Compared with |
|---|---|---|
| Public, any number of hosts | Calibrated | hosts of the same model; adjacent tiers in a quarter of the comparisons, scored against the calibration table |
| Private, several hosts | Peer-only | hosts of the same model only |
| Private, single host | Unverifiable | none; no score is published |
Q3. Fig. 5 vs. the text.
Thank you for catching this. Fig. 5 shows end-to-end latency, about 3 s of which is model inference, not APIX's own overhead. We re-measured the overhead separately, corrected the text, and added the decomposition as a figure (detail in ).
APIX's own overhead. APIX adds 0.31 s to every call plus a one-time lease setup of 20.5 s under our setting, i.e., 20.5/N + 0.31 s per call for a lease of N calls; with a lease kept open, only 0.31 s remains.
| Overhead added by APIX | When it is paid | Measured |
|---|---|---|
| Lease setup time | once per lease | 20.5 s (4 blocks) |
| Added latency per call | every call | 0.31 s (p90 0.35 s) |
Reply to Reviewer Skn2
We thank the reviewer for the detailed analysis and the concrete attack scenarios. We implemented each scenario as an adversary, and address the comments point by point below.
W1. Privacy of user data.
Users no longer need to share sensitive calls. We add a privacy flag: users can mark calls as private; private calls are never sampled, so they are never shared with peers or judges, and the flag stays on the user side. Private users still benefit from the evaluation of other users' calls (detail in Appendix D, Appendix F).
[Table R1] One host substitutes while a share of the users marks all of its calls private (10 seeds, base sampling rate 5%). A host is "filtered" when it is confidently below the threshold that users set, so they stop selecting it. A request is "satisfied" when the served model meets its requirement. Detection is unaffected when up to half of the users are private and is still 92% with three quarters, and private users stay protected by the evaluation of other users' calls (92–100% of their requests are satisfied).
Because the flag stays on the user side, a host that tries to cheat only on private calls cannot tell them apart, and is filtered in 85% of its cheating epochs.
On proving anonymization or no training: the protocol cannot prove what a provider does with data it has received. APIX limits exposure instead, since private calls reach only the provider that the user chose, and TEE attestation can be added as an optional reputation dimension.
W2. Fingerprinting and biased high-reputation majorities.
We agree with both observations: LLMs can recognize which model produced an output, and reputation-weighted selection lets a high-reputation group amplify itself. We therefore do not rely on judge anonymity, no longer select judges by reputation, and bound what a biased group can do (detail in Sec. 3.4, Appendix D):
[Table R2] Colluding judges that recognize their members' outputs perfectly, favor them, and vote against honest hosts (10 seeds). Four hosts of the upper tiers, where few judges are eligible, hold 5% to 30% of all stake. The coalition gains nothing while it holds up to a third of the judge seats of any tier; it gains only once it holds half or more, which is outside the assumption of an honest stake majority among the judges of each tier. With 100 providers, the same four hosts hold 3% of all judge seats and gain nothing.