- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines answers from several language models with judgments from an answer checker and a reasoning checker. It learns how much to trust each source on a question, then uses an approximate posterior over those trust parameters to represent uncertainty about the fit. A separate Learn-Then-Test rule decides when to return an answer. The main experiments suggest that learned trust helps answer selection, while the extra Bayesian step has mixed benefits. Its clearest positive result is answering more ARC questions after one generator is replaced with Gemma.
Critical strengths
- The comparisons help separate what is doing the work. The paper tests constant source trust, question-dependent trust, and Bayesian averaging using the same evidence (Table 2). This makes it easier to see whether a gain comes from learning source weights or from representing uncertainty about them.
- The mixed-family ARC result is promising. With one Qwen generator replaced by Gemma, RA-BSC q10 answers about 92 out of 100 questions, compared with 86 for fixed trust, under the same calibrated error-control procedure (Table 16). That could be useful even though overall accuracy is slightly lower.
Critical weaknesses
- I’m not yet convinced the Bayesian step adds enough beyond simpler weighting. Simpler trust models choose answers at least as accurately in the main experiments, while the coverage gains are mixed. Since the claimed benefit is deciding when to answer, the strongest simple scores should be tested under the same abstention procedure. Verifier-weighted voting is missing from that main comparison despite its strong ARC ranking result (Tables 1 and 7).
- It is unclear how much the benefit depends on the chosen sources. Copying a generator raises confidence without adding information (Table 14), and the strongest positive coverage result comes from one changed generator mixture. This makes it hard to tell how well the reported benefit carries over to another source setup. More broadly, it seems the result depends quite a bit on how sources are selected - some recommendations and/or acknowledgement of this limitation would be helpful.
Questions for authors
- How does it compare with the strongest simple scores under the same answer-or-abstain rule? Consider adding verifier-weighted voting and the scalar-calibrated voting scores to the main LTT comparison, with paired uncertainty for coverage differences.