- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
My understanding is that the paper studies whether experiments on simpler combinations of components of an LLM system can justify a specified calling decision in an untested combination. Under a restriction on interaction order, it reconstructs the relevant reward contrasts and studies how sampling error is amplified. It also proposes additional measurements that reduce this amplification and a procedure that can approve, refute, or leave the decision unresolved. I am not fully sure how this maps onto practical LLM component reuse, given my limited familiarity with this subfield.
Critical strengths
As a statistician, I appreciate the distinction between insufficient information and insufficient statistical precision. The paper is very clear in its thesis that more repetitions cannot resolve every limitation of an experimental design. Focusing on the contrasts needed for a particular decision also seems sensible. The paper explicitly acknowledges that its guarantees depend on structural assumptions and illustrates failure when these assumptions are violated, and I find that very transparent and honest.
Critical weaknesses
My expertise is in causal inference, and I am not familiar with the literature on LLM systems and component reuse. Some difficulties I found while reading the paper may reflect this background, but I still found the motivation, objectives, and constructions hard to follow. In particular:
- The motivation and practical setting are insufficiently developed for a broader audience. For somebody outside the literature on LLM system, it is very hard to understand what the paper is about in its fullest, and what are the selected focused contributions.
- The abstract and introduction introduce technical concepts, notation, and mathematical objects before making the underlying task concrete. Several central objects are formally introduced but insufficiently explained operationally, particularly functional roles, backgrounds, and the intervention protocol, until partially much later.
- The presentation feels rushed. A running example and a more step-by-step connection between the decision problem, reconstruction, and certification would substantially improve readability.
- The intervention protocol needs a more concrete explanation. Section 3 defines interventions through role availability and states that their task meaning and protocol remain fixed across backgrounds. However, I found it difficult to understand how this is implemented when the roles have different concrete realizations across tasks. A worked example would help.
- The practical justification for the degree restriction remains limited. Section 6.2 addresses bounded misspecification, but obtaining a credible bound on untested contexts remains an unresolved application question.
- The experiments use a synthetic generator with controlled interaction degree. They illustrate the theory, but provide limited evidence about whether its assumptions and measurement options are realistic for LLM systems.
Questions for authors
- Could you provide one LLM-system example defining , and the reward concretely instead of at the abstract level, and carry it through the proposed procedure?
- What exactly is manipulated by a “causal intervention”: component availability, actual execution, or another system property? What remains fixed across intervention conditions?
- How would a practitioner choose and justify , or the misspecification bound , without observing the target background?
- The calibration method tests combinations that can be almost as large as the target combination. Could you give a practical example where these tests are feasible but directly testing the target is not? Does counting reward observations adequately reflect their computational cost?
- What practical reuse scenario motivates using , rejecting alone, and using when is available? Is this pattern an illustrative choice, or does the proposed method specifically depend on it?
Dear Reviewer 7mj3,
We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.
We would also like to share important updates that may change your view of the paper. Following your suggestions, we have clarified the intervention protocol, certification assumptions, and practical setting, and conducted additional experiments with Qwen3-8B on FinQA.
We have substantially revised the presentation to connect the practical decision problem, reconstruction theory, measurement design, and finite-sample certification.
Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.