- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This work studies which stable matchings LLMs select when multiple stable matchings exist, and how prompt framing influences that selection. Using small two-sided markets where the full stable set can be enumerated, this work pairs a generic request for a stable matching with a request for a stable matching that is neither side-optimal.
At last, they find several frontier modes that deal almost exclusively with one side's optimum under generic prompts yet can reliably produce non-endpoint stable matchings on the same instances when asked; and that the favoured side can be flipped by changing neutral-seeming prompt formats.
Critical strengths
The work focuses on selection among admissible stable matchings, rather than only feasibiity/stability or proposer-optimality. Filling a gap in current LLMs for matching evaluations.
Other than that, it introduces a paired evaluation on shared data, allowing strong within-instance comparisons that isolate default choice vs. capability to comply with a selection rule.
At last, clever use of distance-matched negative markets to probe non-existence reporting beyond trivial cues.
Critical weaknesses
There seems to be some issue with work; experiments are restricted to N=5 with strictly random preferences, where generalisation to larger markets or structured/correlated preferences is untested for the core selection findings.
Single-shot generations per market/request with no measurement of within-instance variance or sensitivity to stochasticity.
Use of different clients/system prompts and undisclosed sampling/compute defaults across providers introduces confounds that prevent attributing effects solely to model weights.
Other than the above, the assured vs unassured request changes two degrees of freedom at once, making it hard to isolate which element drives behaviour changes.
The JSON schema for the unassured case conflicts with instructions to withhold a matching, affecting withholding compliance; while noted and partially controlled, it remains a confound.
At last, limited exploration of alternative, normative selection objectives across all models; only a few supplementary workings were tested on a subset of models.
Questions for authors
The things mentioned in the weaknesses raise some questions, such as:
-
Can you disentangle the effects of existence assurance vs permission to abstain by adding an ablation where you assure existence but also allow abstention or vice versa? This would clarify which change drives model behaviour.
-
How stable are the framing effects under repeated sampling for the same market and prompt? Do model selections vary or remain locked to the same side's optimum?
-
Did you try a DA step-by-step prompt that forces the model to choose the proposing side, and then vary which side is designated as proposer? Would that directly confirm the hypothesised mechanisms behind the framing-induced endpoints?
-
For the instruction to withhold a match on nonexistence, if you provide a null matches schema and eliminate the conflict, do models still differ in withholding compliance? Any preliminary evidence?
-
You show that several models can produce non-endpoints reliably. Do they concentrate on specific endpoints within a market or diversify across the non-endpoints?
I would or could raise the rating if the justification seems convincing enough.
4 comments
Response to the Authors
Thanks for the rebuttal. In Table 31, an independent written prompt (set 1/ set 2) breaks the "favoured side meets two of three conditions" rule. Five out of six models favour set 1 even when set 2 is listed first and keys the answer. So the existence of one-sided default looks robust, but the specific three-condition rule seems to depend on the template. The abstract states that the rule has general findings, so the authors should tone it down. In any case, I have increased my score of soundness from 1 to 3 and informativeness from 2 to 3.
Thank you for taking the time to review this!
We agree with your reading of Table 31: The one-sided default holds under both prompts, and the two-of-three rule is specific to the main prompt. In the revised abstract, the rule is replaced by "Which side the default favors depends on the framing and wording of the prompt." In the introduction, the Set 1 result now follows the rule directly.
If this resolves your remaining concern, we would be grateful if you would reconsider your overall rating!
Response to the Authors
As said previously, and in your new response, I acknowledge, but I will maintain my overall ratings.
Thank you for the questions!
Q1 and Q4 (assurance versus permission; a null schema): We crossed the assurance with the permission sentence under a schema that allows a null matching on the positive markets, and ran the permitted request under that schema on the 50 negative markets, with the populated schema rerun as a control (Table 29). The assurance drives false nonexistence claims: with it, Sol, Terra, and Luna make 0 to 1, 2 to 3, and 11 to 13 such claims on the 50 positive markets; without it, 7, 5 to 11, and 30 to 32. The permission sentence changes little, except for Terra without the assurance (11 against 5). Opus, Fable, and Astra succeed in all four cells (47 to 50). On the negative markets, the null schema leaves withholding within two markets of the populated control for every model. Opus 5.5, which withheld the matching on 40 of the 50 negatives in Table 6, withholds on 49 in the control and on 50 under the null schema, so its earlier lapses vary between runs of the same prompt as Section 5 reports in the updated draft.
Q2 (repeated sampling): Astra gives the same outcome in all five draws on every market, under both requests and both opposing framings. One more draw from Opus 5.5 repeats its original outcome on every market except one where the original assured answer was unstable. Sol's generic outcome repeats in all five draws on 48 of 50 markets and each opposing framing on 19 of 20. Terra and Luna vary more, mostly between stable and unstable answers: Across 200 generic draws they return Group 1's optimum 141 and 155 times, and Group 2's 5 and 13 times (Table 28).
Q3 (mechanism): With a named proposing side, Opus, Fable, and Astra return that side's optimum on 50 of 50 for each side, Sol on 48 and 49, and Luna on 44 and 44; Terra's answers are often unstable (25 and 29 successes; Table 30). Without the instruction, Opus 5.5 and Fable 5.1 name the favored side as proposer in 346 and 325 of 350 responses (Table 5). So the endpoint follows the stated proposer.
Q5 (which non-endpoint): They concentrate near Group 1's optimum: 126 of 135 at N = 5 (86.5 expected under uniform choice; Table 17), and at N = 10, on the 27 markets with two to eight non-endpoints, 122 of 132 for all models but Luna (69.5 expected; Table 27). Opus and Astra return the same alternative in every repeated draw on 12 and 10 of the 12 markets with several.
The central comparisons are all within one model and client but Appendix A.2 lists every setting in detail.