- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Agent-NASCD presents a single-agent, LLM-controlled neural architecture search framework for cognitive diagnosis. A ReAct-style controller uses seven domain-specific tools to construct or mutate operator programs, retrieve candidate lineage and evaluation history from a model optimization graph (MOG), and select between proxy and full-fidelity evaluation, with legality checks and optional final confirmation.
Critical strengths
- Structured historical information and diagnostic feedback are supported by experimental evidence. The component studies demonstrate improvements over Direct LLM Search and Fixed-Policy search. Furthermore, the paired repair experiment shows that detailed diagnostic feedback successfully repairs more invalid programs with fewer calls (96/100 versus 89/100 successful repairs; 153 versus 174 calls). These results provide meaningful evidence for the practical value of structured search observations and actionable diagnostic feedback.
- The experimental evaluation was fairly thorough. The authors conduct a broad predictive evaluation against psychometric, neural, and NAS-based approaches across three datasets and four training ratios. The use of ten matched random seeds, held-out test evaluation, variance reporting, and paired statistical tests improves the credibility of the experimental findings.
Critical weaknesses
- The independent contribution of LLM-based sequential decision-making, beyond a strong adaptive graph-aware heuristic using the same tools and budget, remains insufficiently isolated. The proposed framework combines ReAct-style tool selection, candidate-lineage tracking, program validation, and proxy/full evaluation, which are closely related to established techniques in agentic optimization and NAS. The Fixed-Policy comparison is informative, but it is a single, relatively simple heuristic rather than a tuned or stronger graph-aware baseline. This limits the strength of the claimed methodological novelty.
- The generality of the reported performance and efficiency gains is not fully established. Table 8 indicates that the AUC improvements are not statistically significant in all dataset-ratio settings, and Table 3 reports component-level means without uncertainty estimates. The primary end-to-end runtime comparison (Table 2) is limited to ASSIST09 at the 70% training ratio. Table 12 further shows that complete search-cost records are available for only 12/40 ASSIST09 runs, 12/40 XES-2k runs, and 11/40 MOOC-2k runs, and monetary LLM costs are unavailable. These limitations weaken the evidence for broadly generalizable performance and efficiency improvements.
Questions for authors
Q1: Could the authors compare Agent-NASCD against a stronger, tuned graph-aware adaptive non-LLM controller using the same candidate registry, diagnostic checks, proxy/full evaluation options, and matched computational budgets? This would help clarify the contribution of LLM-based sequential decisions beyond existing adaptive search heuristics. Q2: What explains the incomplete search-cost coverage in Appendix Table 12? Could the authors provide per-seed end-to-end wall time, GPU training time, proxy/full evaluation counts, and LLM usage across all settings, or analyze whether the missing records introduce bias into the reported efficiency results? Q3: Could the authors provide paired uncertainty estimates for the component comparisons in Table 3 and multi-seed, budget-matched anytime performance across additional representative dataset-ratio settings against the strongest NAS baselines? Given that Table 8 reports statistically significant improvements in only 5/12 and 4/12 settings against EMO-A7 and OSCD-AA1, respectively, such evidence could materially affect the assessment of the reported gains.
2 comments
Q2 / W2: "What explains the incomplete search-cost coverage in Appendix Table 12? … or analyze whether the missing records introduce bias into the reported efficiency results?"
Per-run cost logging was added after the main-grid searches had started, so complete cost records exist for the last three seeds of each setting (11 for MOOC-2k). The seeds with complete records were fixed by the run order, so the subset is independent of the outcomes. With three seeds per setting, the efficiency trend is already clear: across all settings, a search used 43–122 evaluations, 14–71 LLM calls and on average 8.4–9.6 hours, versus 15.2 GPU-days for one EMO-NAS-CD search on ASSIST09; this gap is much larger than the seed-to-seed variation. Every new run in this rebuttal records wall-clock time, GPU training time, short/full training counts, cache hits, LLM calls and input/output tokens. Table R2 reports per seed the proxy/full evaluation counts, training hours, LLM calls, tokens and API cost at the list price of glm-5.3-flash (RMB 0.8 per million input tokens and RMB 2.8 per million output tokens, without cache discounts): RMB 0.35–0.57 per Full-Only search. At the same price, the seed-33 case-study search comes to about RMB 0.79 (Appendix G).
Q3: "Could the authors provide paired uncertainty estimates for the component comparisons in Table 3 and multi-seed, budget-matched anytime performance … against the strongest NAS baselines?"
Table 3 variants ran under the same resource caps; their evaluation counts differ because each controller decides when to stop. New Table R3 in Appendix E reports the AUC standard deviation of every Table 3 variant over the ten seeds, with 95% Welch intervals of each variant's AUC difference to the complete method, computed as in Table R1: all five component variants are significantly lower, from −0.0148 [−0.0188, −0.0108] for Write-Only to −0.0679 [−0.0736, −0.0622] for Direct LLM Search, while the DeepSeek-V4-Pro and Claude-Sonnet-5 controllers are within 0.0053 of it. To measure the effect of adaptive fidelity with uncertainty, we added a Full-Only ablation (no choice of training length) under the same budget (5,000 candidate-training epochs per seed) over 5 seeds. Full-Only evaluates 70 models on average, fewer than the complete method's 86 in Table 3, and its test AUC of 0.7980 ± 0.0172 is 0.0301 lower (95% Welch CI [−0.0513, −0.0090]). Figure 3(c) gives multi-seed anytime curves (best validation AUC against training epochs) of the Full-Only and tuned-controller runs under the same epoch budget; these runs use ASSIST09 at 70% training, the setting of the runtime and component studies in Tables 2 and 3, so that they can be compared directly with those results. For the strongest NAS baseline, Figure 3(a,b) compares anytime performance with EMO-NAS-CD on ASSIST09: on the seed-33 trajectory, Agent-NASCD reaches a validation AUC of 0.82 after 57 evaluations, whereas EMO-NAS-CD needs 2,885, and at 120 evaluations EMO-NAS-CD's best is 0.7773, which is 0.0507 lower. Because stopping EMO-NAS-CD earlier can only lower its result, the comparison at any equal budget favors Agent-NASCD. We also agree with your reading of Table 8: the gains over EMO-NAS-CD and OSCD are significant in 5/12 and 4/12 settings, and the revision changes the claims accordingly. Table R1 (General Response) adds the size of each difference with a 95% confidence interval for all 12 settings.
Thank you for the careful reading, including the appendix tables. The new results are in the revised PDF and summarized below.
We agree that Fixed-Policy is a simple heuristic. We added a graph-aware regularized-evolution controller that uses the same candidate registry, legality checks,
mutate_archneighbor generator, proxy/full options, final selection rule and per-seed epoch budget as the agent, and starts from randomly generated legal programs as in the main searches: parent selection by tournament among the 30 most recently scored candidates; adaptive fidelity, where a candidate is trained at proxy fidelity first and promoted to full training only if its proxy score is within δ of the best proxy score so far; and restarts, writing a fresh legal program with probability 0.1. Tournament size (3 or 10) and δ (0.002 or 0.01) are tuned on a separate seed that is not among the evaluation seeds, using validation AUC only; the tuning selected tournament size 3 and δ = 0.002. The controller receives 5,000 candidate-training epochs and at most 120 evaluations per seed, the same training budget as the complete method (this matches the size of its searches: 86 evaluations on average in Table 3, most of them 55-epoch proxy runs). It runs on 5 seeds, all among the ten seeds of Table 1.Result (Section 5.3, Figure 3(c), Appendix F, Table R2). The tuned controller reaches a test AUC of 0.7546 ± 0.0165 with 94.6 evaluations on average, which is 0.0735 below the complete method's ten-run ASSIST09-70% result of 0.8281 ± 0.0051 (95% Welch CI [−0.0938, −0.0533]). On the same seeds, initial programs, code and budget, the LLM-controlled Full-Only variant is 0.0434 higher than the tuned controller (paired 95% CI [+0.0019, +0.0849]) and higher on four of the five seeds, and its anytime curve stays above that of the tuned controller for almost the whole search. Fixed-Policy in Table 3 used all 120 evaluations, which exceeds this epoch budget, and it is therefore excluded from this comparison.
This comparison isolates the contribution of the LLM's sequential decisions that you asked about, since the two controllers share the tools, the graph, the candidate checks and the budget. On novelty more broadly, our response to Reviewer uczo (W1) compares which decisions the LLM controls in related methods.