- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether an LLM’s successful solution to a problem necessarily reflects that the LLM had accurately diagnosed what the problem was in the first place. The authors analyze this in a simulated environment for quantum error correction, where an LLM needs to estimate three noise parameters from observed error counts. A pre-specified program then uses these estimates to generate additional training data to train a neural decoder. This allows the authors to measure two variables separately: how accurate is an LLM in estimating parameter accuracy, and what’s the improvement in the decoder’s performance? The authors compare decoder performance when its train data is generated under three different conditions: (1) using the LLM’s estimates, (2) deliberately corrupting LLM’s estimates, (3) fixed default values. They find that the LLM’s original estimates produce greater improvements compared to corrupted estimates or default values, but corrupted estimates also improve the decoder. The authors argue that this provides evidence that improved performance alone does not establish that the proposed noise model was correct. Lastly, the authors find that a simple statistical estimator performs similarly to the LLM in the parameter estimation task.
Critical strengths
The question is relevant to the broader field of AI-driven scientific discovery The authors include baselines, and also report negative results / unsuccessful experiments.
Critical weaknesses
- The authors use a single model (gpt-sol-5.6) on a single (custom-designed) environment to motivate their findings. It is unclear if these results would generalize to other tasks or models.
- It is unclear whether the task design helps answer the main underlying question, because it is the pre-specified program that determines how the estimated noise parameters affect the decoder’s training, and therefore its performance. This limits what the results actually establish about the LLM’s ability to diagnose decoder’s limitations and design succssful interventions.
- The paper is very difficult to read. The authors should dedicate a considerable amount of effort to clarify the writing in most sections of the paper. This is an interesting case where using a frontier LLM to improve the writing, section-by-section, would likely improve the paper significantly.
- Many important details about the experiments either missing or insufficiently explained. This includes, for example, the prompts that were used, maximum output tokens for the model, and decoder architecture.
Questions for authors
- The task analyzes how accurately an LLM can estimate environmental noise (the three parameters). Can you clarify how this connects to the limitations of the decoder, rather than simply describing its environment? It seems like the central argument of the paper is based on the connection between the two.
- How sensitive is the relationship between parameter accuracy and the accuracy of the decoder?
- Can you provide more details about the experiment, such as the prompts that were used, output token limits, decoder architecture, etc?
- Could you add results for at least one additional model and an at least one additional task / configuration?