acceptodds
Withdrawn from ICLR 2027

Agent-NASCD: Efficient and Reliable Agentic Architecture Search for Cognitive Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cognitive diagnosis (CD) aims to estimate students’ proficiency in knowledge concepts from response logs. However, existing evolutionary and one-shot neural architecture search (NAS) methods for designing CD interaction functions still incur substantial candidate evaluation costs. Besides, directly prompting a language model can produce invalid programs or unstable search directions. To address these issues, this paper proposes Agent-NASCD, a single-agent framework that treats CD architecture search as sequential evidence-conditioned decision making. Specifically, CD-oriented tools support program construction and mutation, a model optimization graph (MOG) records candidate lineage and evaluation results, and constrained proxy/full evaluation screens candidates at different training fidelities. Experiments on three real-world datasets show competitive predictive performance of the selected architectures and support the contributions of the MOG and diagnostic feedback to candidate-training efficiency.

Primary area:
optimization
Keywords:
Cognitive Diagnosis, Neural Architecture Search, Intelligent Education, Large Language Models, Multi-Fidelity Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_jXp3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Agent-NASCD presents a single-agent, LLM-controlled neural architecture search framework for cognitive diagnosis. A ReAct-style controller uses seven domain-specific tools to construct or mutate operator programs, retrieve candidate lineage and evaluation history from a model optimization graph (MOG), and select between proxy and full-fidelity evaluation, with legality checks and optional final confirmation.

Critical strengths
  1. Structured historical information and diagnostic feedback are supported by experimental evidence. The component studies demonstrate improvements over Direct LLM Search and Fixed-Policy search. Furthermore, the paired repair experiment shows that detailed diagnostic feedback successfully repairs more invalid programs with fewer calls (96/100 versus 89/100 successful repairs; 153 versus 174 calls). These results provide meaningful evidence for the practical value of structured search observations and actionable diagnostic feedback.
  2. The experimental evaluation was fairly thorough. The authors conduct a broad predictive evaluation against psychometric, neural, and NAS-based approaches across three datasets and four training ratios. The use of ten matched random seeds, held-out test evaluation, variance reporting, and paired statistical tests improves the credibility of the experimental findings.
Critical weaknesses
  1. The independent contribution of LLM-based sequential decision-making, beyond a strong adaptive graph-aware heuristic using the same tools and budget, remains insufficiently isolated. The proposed framework combines ReAct-style tool selection, candidate-lineage tracking, program validation, and proxy/full evaluation, which are closely related to established techniques in agentic optimization and NAS. The Fixed-Policy comparison is informative, but it is a single, relatively simple heuristic rather than a tuned or stronger graph-aware baseline. This limits the strength of the claimed methodological novelty.
  2. The generality of the reported performance and efficiency gains is not fully established. Table 8 indicates that the AUC improvements are not statistically significant in all dataset-ratio settings, and Table 3 reports component-level means without uncertainty estimates. The primary end-to-end runtime comparison (Table 2) is limited to ASSIST09 at the 70% training ratio. Table 12 further shows that complete search-cost records are available for only 12/40 ASSIST09 runs, 12/40 XES-2k runs, and 11/40 MOOC-2k runs, and monetary LLM costs are unavailable. These limitations weaken the evidence for broadly generalizable performance and efficiency improvements.
Questions for authors

Q1: Could the authors compare Agent-NASCD against a stronger, tuned graph-aware adaptive non-LLM controller using the same candidate registry, diagnostic checks, proxy/full evaluation options, and matched computational budgets? This would help clarify the contribution of LLM-based sequential decisions beyond existing adaptive search heuristics. Q2: What explains the incomplete search-cost coverage in Appendix Table 12? Could the authors provide per-seed end-to-end wall time, GPU training time, proxy/full evaluation counts, and LLM usage across all settings, or analyze whether the missing records introduce bias into the reported efficiency results? Q3: Could the authors provide paired uncertainty estimates for the component comparisons in Table 3 and multi-seed, budget-matched anytime performance across additional representative dataset-ratio settings against the strongest NAS baselines? Given that Table 8 reports statistically significant improvements in only 5/12 and 4/12 settings against EMO-A7 and OSCD-AA1, respectively, such evidence could materially affect the assessment of the reported gains.

2 comments

Thank you for the careful reading, including the appendix tables. The new results are in the revised PDF and summarized below.

Q1 / W1: "Could the authors compare Agent-NASCD against a stronger, tuned graph-aware adaptive non-LLM controller using the same candidate registry, diagnostic checks, proxy/full evaluation options, and matched computational budgets? … This limits the strength of the claimed methodological novelty."

We agree that Fixed-Policy is a simple heuristic. We added a graph-aware regularized-evolution controller that uses the same candidate registry, legality checks, mutate_arch neighbor generator, proxy/full options, final selection rule and per-seed epoch budget as the agent, and starts from randomly generated legal programs as in the main searches: parent selection by tournament among the 30 most recently scored candidates; adaptive fidelity, where a candidate is trained at proxy fidelity first and promoted to full training only if its proxy score is within δ of the best proxy score so far; and restarts, writing a fresh legal program with probability 0.1. Tournament size (3 or 10) and δ (0.002 or 0.01) are tuned on a separate seed that is not among the evaluation seeds, using validation AUC only; the tuning selected tournament size 3 and δ = 0.002. The controller receives 5,000 candidate-training epochs and at most 120 evaluations per seed, the same training budget as the complete method (this matches the size of its searches: 86 evaluations on average in Table 3, most of them 55-epoch proxy runs). It runs on 5 seeds, all among the ten seeds of Table 1.

Result (Section 5.3, Figure 3(c), Appendix F, Table R2). The tuned controller reaches a test AUC of 0.7546 ± 0.0165 with 94.6 evaluations on average, which is 0.0735 below the complete method's ten-run ASSIST09-70% result of 0.8281 ± 0.0051 (95% Welch CI [−0.0938, −0.0533]). On the same seeds, initial programs, code and budget, the LLM-controlled Full-Only variant is 0.0434 higher than the tuned controller (paired 95% CI [+0.0019, +0.0849]) and higher on four of the five seeds, and its anytime curve stays above that of the tuned controller for almost the whole search. Fixed-Policy in Table 3 used all 120 evaluations, which exceeds this epoch budget, and it is therefore excluded from this comparison.

This comparison isolates the contribution of the LLM's sequential decisions that you asked about, since the two controllers share the tools, the graph, the candidate checks and the budget. On novelty more broadly, our response to Reviewer uczo (W1) compares which decisions the LLM controls in related methods.

Q2 / W2: "What explains the incomplete search-cost coverage in Appendix Table 12? … or analyze whether the missing records introduce bias into the reported efficiency results?"

Per-run cost logging was added after the main-grid searches had started, so complete cost records exist for the last three seeds of each setting (11 for MOOC-2k). The seeds with complete records were fixed by the run order, so the subset is independent of the outcomes. With three seeds per setting, the efficiency trend is already clear: across all settings, a search used 43–122 evaluations, 14–71 LLM calls and on average 8.4–9.6 hours, versus 15.2 GPU-days for one EMO-NAS-CD search on ASSIST09; this gap is much larger than the seed-to-seed variation. Every new run in this rebuttal records wall-clock time, GPU training time, short/full training counts, cache hits, LLM calls and input/output tokens. Table R2 reports per seed the proxy/full evaluation counts, training hours, LLM calls, tokens and API cost at the list price of glm-5.3-flash (RMB 0.8 per million input tokens and RMB 2.8 per million output tokens, without cache discounts): RMB 0.35–0.57 per Full-Only search. At the same price, the seed-33 case-study search comes to about RMB 0.79 (Appendix G).

Q3: "Could the authors provide paired uncertainty estimates for the component comparisons in Table 3 and multi-seed, budget-matched anytime performance … against the strongest NAS baselines?"

Table 3 variants ran under the same resource caps; their evaluation counts differ because each controller decides when to stop. New Table R3 in Appendix E reports the AUC standard deviation of every Table 3 variant over the ten seeds, with 95% Welch intervals of each variant's AUC difference to the complete method, computed as in Table R1: all five component variants are significantly lower, from −0.0148 [−0.0188, −0.0108] for Write-Only to −0.0679 [−0.0736, −0.0622] for Direct LLM Search, while the DeepSeek-V4-Pro and Claude-Sonnet-5 controllers are within 0.0053 of it. To measure the effect of adaptive fidelity with uncertainty, we added a Full-Only ablation (no choice of training length) under the same budget (5,000 candidate-training epochs per seed) over 5 seeds. Full-Only evaluates 70 models on average, fewer than the complete method's 86 in Table 3, and its test AUC of 0.7980 ± 0.0172 is 0.0301 lower (95% Welch CI [−0.0513, −0.0090]). Figure 3(c) gives multi-seed anytime curves (best validation AUC against training epochs) of the Full-Only and tuned-controller runs under the same epoch budget; these runs use ASSIST09 at 70% training, the setting of the runtime and component studies in Tables 2 and 3, so that they can be compared directly with those results. For the strongest NAS baseline, Figure 3(a,b) compares anytime performance with EMO-NAS-CD on ASSIST09: on the seed-33 trajectory, Agent-NASCD reaches a validation AUC of 0.82 after 57 evaluations, whereas EMO-NAS-CD needs 2,885, and at 120 evaluations EMO-NAS-CD's best is 0.7773, which is 0.0507 lower. Because stopping EMO-NAS-CD earlier can only lower its result, the comparison at any equal budget favors Agent-NASCD. We also agree with your reading of Table 8: the gains over EMO-NAS-CD and OSCD are significant in 5/12 and 4/12 settings, and the revision changes the claims accordingly. Table R1 (General Response) adds the size of each difference with a 95% confidence interval for all 12 settings.

Reviewer_cfSLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper does NAS for a specific application, "cognitive diagnosis."

Critical strengths

I think the area of neural architecture search is very exciting, especially driven by modern LLM-driven methods. And it's also fine to work on NAS for a specific target application if it's well motivated and explained.

Critical weaknesses
  1. Overall, the writing quality is low, and it reads like it is AI-generated. Many concepts are not explained, or are explained in super dense jargon that seems to refer to a bunch of concepts that haven't yet been fully explained, and it's largely unreadable.It's certainly not of the level of quality it needs to be for a clear scientific manuscript, let alone a top-tier machine learning conference. For example, lines 115–117 contain a lot of pseudoscientific jargon and technical-sounding stuff that maybe makes sense, but certainly is not written in a clear enough way for a high-quality science manuscript. Scientific papers should be extremely well written and clearly introduce and communicate each concept, whereas here you just get this dump of jargon and words that has the veneer of scientific writing but actually doesn't make a lot of sense.

  2. It's not clear what the main contribution or motivation of this paper is.Is this really a paper about neural architecture search (NAS) specifically for cognitive diagnosis (CD)? Or is it just generically a neural architecture search paper that happens to use CD as a test domain? Could the NAS method work in any domain? Why or why not? You spent a lot of time specifically talking about CD and NAS for CD, but I don't see a reason why that's a unique domain that requires specialized architectures, or why this method is specialized. Knowing that would help us figure out even what the right baselines are.

  3. The paper overall, and especially the results section, just seems like a giant dump of what you did and what you found. There are tremendous numbers of comparisons and experiments, and it's hard to get a sense of what matters and how to prioritize it. It's a huge dump of information without much contextualization. But a key part of a scientific paper requires its authors to focus on what matters and really explain the meaningful results. But it's very hard to get a sense of the overall takeaway messages and whether they're informative for NAS, for CD, or for NAS-CD. What is the big-picture TL;DR of this paper? What’s the big tldr for each set of experiments? That's not coming across in the current draft.

Questions for authors

See critical weaknesses.

Also, in the questions section, the AI use statement only mentions that you used AI to polish the writing and prepare figures. Can you confirm that you did not use AI to code the experiments, launch the experiments, come up with the ideas for the methods, or assist with any other aspects of the scientific process?

5 comments

Thank you for the candid feedback. We agree that the submission was too dense. The revised PDF rewrites the passages you pointed to (the introduction, the related work on LLM controllers, the MOG description and the experiments) in plain sentences and defines each term at first use.

W1: "Many concepts are not explained, or are explained in super dense jargon … For example, lines 115–117 … [are] not written in a clear enough way for a high-quality science manuscript."

You are right that these lines pack several undefined ideas into one sentence. They now read: "These works show that a language model can take part throughout a search, not only propose a starting design. Two gaps remain for CD architecture search. First, prompting-based optimizers present past results as a flat list of (candidate, score) pairs, without recording which candidate was derived from which or whether a score came from a proxy or a full evaluation. Second, the LLM-based method for CD uses the language model inside a fixed evolutionary loop, so the model cannot decide which candidate to evaluate next or when to stop." We applied the same treatment throughout: each term is defined at first use (proxy/full evaluation, the model optimization graph, the program encoding, DOA), and the method sections are written in plain sentences.

W2: "Is this really a paper about neural architecture search (NAS) specifically for cognitive diagnosis (CD)? … Could the NAS method work in any domain? Why or why not?"

The controller loop (an LLM that queries a graph of past results and decides what to train next and when to stop) is not specific to CD, and the introduction now says so. The CD-specific parts are the search space (programs over student, exercise and concept embeddings with CD operators such as an IRT-style logistic term, Q-matrix masking and a guess/slip mixture, with a non-negative prediction head for vector outputs, following prior CD architecture search), the legality checks and their error messages, and the cost structure (each model trains in minutes, but a search trains many of them and short-training scores are noisy). We target CD because prior work showed that searched CD models beat handcrafted ones but at high search cost. In another domain, the controller loop can be kept, but the search space, the legality checks and the cost model have to be replaced. We therefore compare Agent-NASCD with handcrafted CD models and with CD-NAS methods that search the same operator space.

W3: "What is the big-picture TL;DR of this paper? What's the big tldr for each set of experiments?"

Overall: for CD architecture search, an LLM controller that queries a graph of past results and receives specific error messages reaches accuracy comparable to existing CD-NAS methods at lower search cost. The revision opens the experiments with the questions they answer and the main finding, and each experiment now states its purpose, where the results are, what they show and a one-sentence takeaway.

AI use: "Can you confirm that you did not use AI to code the experiments, launch the experiments, come up with the ideas for the methods, or assist with any other aspects of the scientific process?"

Thank you for asking; our AI Use Statement was incomplete. To answer directly:

  • Idea and method: conceived by the authors (search as sequential decision-making, the tool set, the model optimization graph, the proxy/full evaluation design).
  • Code: the authors wrote the core logic of the framework. AI coding assistants helped write the experiment scripts.
  • Experiments: AI assistants helped set experimental parameters, launch and monitor runs, and organize the results into tables and figures. The authors checked the settings and the results.
  • Writing and figures: AI tools were used to polish the writing and prepare figures, as stated, and to help draft parts of the revised text, which the authors reviewed and rewrote.

The authors take full responsibility for the method, code, results and manuscript. The revised PDF contains a corrected AI Use Statement.

I find the author's response on the AI use statement disqualifying. Their original statement did not, in any way, accurately reflect their use of AI, and because they were not honest in their description, they've lost my trust as a reviewer. The original AI use policy said AI use was very limited: "AI tools were used to polish the writing of this paper and prepare figures." The authors reply admits that much AI use was not mentioned initially. I applaud them for being honest at this point in the process, but it is not acceptable that they were not honest initially.

Science is built on trusting the integrity and accurate reporting of the authors, and it is paramount that authors are never dishonest in their reporting. I think there should be a consequence for not honestly and accurately answering that sort of a question, which explicitly asks about all the different ways AI was used. I think the consequence should be that the paper is rejected, and the authors learn from this mistake and, in all future peer-reviewed publication submissions, they're fully honest and transparent. On that issue alone, I therefore am keeping my score at a 1. Additionally, I think the authors answers are insufficient in addressing the other issues I raised, but I choose to make this the sole issue so that its importance is not lost in a sea of back and forth discussion.

AuthorsOpenReview ↗

Respecting your decision but not agreeing with it

Thank you for your further comments. We would like to clarify several important points, as we respectfully disagree with the characterization of our AI disclosure as dishonest.

  • First, we never stated that AI tools were not used for experimental setup, coding, or other research-related tasks. Our original AI Use Statement described the use of AI tools for improving the manuscript's writing and preparing figures. It did not explicitly claim that these were the only tasks for which AI was used. We acknowledge that the original statement was incomplete, but we believe there is an important difference between incomplete disclosure and intentional dishonesty.

  • Second, we have responded openly and transparently to your concerns. After you raised this question, we provided a detailed explanation of how AI tools were used throughout our research process and updated the AI Use Statement in the revised manuscript accordingly. We also clarified that the research idea, the proposed method, and the core implementation were developed by the authors. We believe these actions demonstrate our willingness to clarify and correct the disclosure, rather than an intention to conceal information.

  • Third, we respectfully disagree with the accusation of dishonesty. We fully recognize the importance of research integrity and accurate reporting. However, we believe that characterizing the authors as dishonest, without sufficient evidence of intentional misrepresentation, is neither fair nor appropriate in a scientific peer-review process. We also note that the ICLR 2027 AI policy distinguishes between required and recommended disclosures, and several of the additional uses we reported fall within the recommended category.

  • Finally, we sincerely hope that the discussion can return to an objective assessment of the paper's scientific contributions. We respect your right to maintain your recommendation, although we respectfully disagree with treating the disclosure issue as the sole basis for rejection without further consideration of the relevant facts. In addition, we have provided detailed responses to all three technical weaknesses raised in your original review and substantially revised the manuscript. If you believe that some concerns remain insufficiently addressed, we would greatly appreciate it if you could identify the specific unresolved issues. We would be happy to provide further clarification.

We believe that scientific peer review should be based on objective evidence, technical merit, and constructive discussion. We appreciate your time and sincerely hope that our work can be evaluated on this basis.

Reviewer_oaEEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Agent-NASCD, an LLM-based framework for efficient neural architecture search in cognitive diagnosis. The framework integrates CD-specific search tools, a Model Optimization Graph (MOG), and multi-fidelity evaluation to guide architecture selection. The LLM controller uses historical search results and diagnostic feedback to generate, modify, and evaluate candidate architectures. The main objective is to reduce search costs while maintaining competitive predictive performance.

Critical strengths

The paper addresses the challenge of reducing the computational cost of architecture search for cognitive diagnosis. The integration of an LLM controller with structured search history and diagnostic feedback provides a practical approach to guiding architecture exploration. The experimental evaluation includes multiple datasets, training ratios, and competitive baselines, demonstrating the effectiveness of the proposed framework. The ablation studies and reliability analysis provide useful evidence supporting the proposed components. Moreover, the substantial reduction in search time is a promising result for practical applications.

Critical weaknesses

The methodological novelty is somewhat limited, as the framework mainly combines existing LLM-based search, architecture mutation, and multi-fidelity evaluation techniques. Although the proposed method achieves the highest mean AUC, its improvements over strong NAS baselines are relatively small and often statistically insignificant. The search-efficiency comparisons are not conducted under strictly matched computational budgets, making it difficult to isolate the advantages of the proposed search strategy. Furthermore, the contribution of each component, particularly MOG and adaptive fidelity selection, is not sufficiently investigated under equal resource constraints. The evaluation also focuses mainly on predictive performance, with limited analysis of the actual cognitive diagnostic quality and interpretability of the discovered architectures.

Questions for authors

The authors should clarify the methodological novelty of Agent-NASCD compared with existing LLM-assisted NAS methods and explain what distinguishes the proposed MOG from conventional search-history mechanisms.

They should provide a stronger justification for the fairness of the computational comparisons, particularly regarding different search strategies and evaluation budgets.

The authors should also discuss the statistical significance of the improvements and avoid overstating predictive superiority where the differences are insignificant.

Further clarification is needed regarding the individual contributions of the MOG, diagnostic feedback, and multi-fidelity evaluation mechanisms.

the authors should explain how the discovered architectures support meaningful cognitive diagnosis rather than only improving response prediction.

The authors should conduct equal-budget comparisons with existing NAS baselines to provide stronger evidence of search efficiency. More comprehensive ablation experiments are needed to isolate the contributions of MOG, diagnostic feedback, and adaptive evaluation strategies.

The experimental analysis should include statistical confidence intervals and search-performance curves averaged over multiple random seeds.

Additional cognitive diagnosis metrics, such as DOA across all datasets, and analyses of proficiency interpretability and monotonicity would strengthen the educational relevance of the method.

The work appears too preliminary at this stage and i would appreicate more experiments on larger datasets or more complex search spaces would help establish the scalability and generalizability of the proposed framework.

3 comments

Thank you for the detailed list of analyses. The new results are in the revised PDF and summarized below.

Novelty and MOG: "clarify the methodological novelty of Agent-NASCD compared with existing LLM-assisted NAS methods and explain what distinguishes the proposed MOG from conventional search-history mechanisms."

Please see our response to uczo (W1), which includes a comparison table. In short, the LLM controls not only which architectures to propose but also which candidate to train next and when to stop; MOG keeps the lineage and scores needed for those decisions. The Full-Only ablation for the short/full choice is described below.

Fairness of the comparison: "provide a stronger justification for the fairness of the computational comparisons … conduct equal-budget comparisons with existing NAS baselines"

Agent-NASCD reaches comparable accuracy with far fewer candidate trainings, at most about 120 per search while EMO-NAS-CD trains on the order of 10^4, so stopping EMO-NAS-CD at our budget could only lower its result (see the General Response). We agree that Table 2 does not isolate the search strategy. We therefore added a tuned graph-aware non-LLM controller that uses the agent's tools, candidate space, training recipe and selection rule, and the same budget: 5,000 candidate-training epochs and at most 120 evaluations per seed (General Response, item 1). It reaches a test AUC of 0.7546 ± 0.0165, which is 0.0735 below the complete method (95% Welch CI [−0.0938, −0.0533]).

Statistical significance: "discuss the statistical significance of the improvements and avoid overstating predictive superiority where the differences are insignificant."

We agree: the gains over the strongest NAS baselines are significant in 5/12 (EMO-NAS-CD) and 4/12 (OSCD) settings. The revision rephrases the claim as comparable or slightly better accuracy at lower search cost. Table R1 (General Response) reports, for all 12 settings, the AUC difference to EMO-A7 and OSCD-AA1 with 95% confidence intervals next to the Wilcoxon results; the new experiments on ASSIST09-70% are reported with 95% CIs as well.

Component contributions: "Further clarification is needed regarding the individual contributions of the MOG, diagnostic feedback, and multi-fidelity evaluation mechanisms."

Table 3 isolates the graph (Flat-History), the diagnostic feedback (Generic-Error) and the LLM's action selection (Fixed-Policy, Direct LLM Search). All variants ran under the same resource caps (120 candidate trainings, a 9-hour search limit and the same LLM-call limit); the number of evaluations differs because each controller decides when to stop. The variants that used more resources still reached a lower AUC. Fixed-Policy used the full 120 evaluations and 8.06 training hours, more than the 86 evaluations and 5.35 hours of the complete method, yet reached 0.7969, below the complete method's 0.8281. Flat-History used 5.75 training hours, also more than the complete method, and still did worse. The revision states the caps explicitly and explains what each variant removes (Section 5.4, Table 3). For adaptive fidelity, which Table 3 does not isolate, we added a Full-Only ablation (every new candidate trained at full fidelity) under the same budget (5,000 candidate-training epochs per seed) over 5 seeds. Full-Only evaluates 70 models on average, fewer than the complete method's 86 in Table 3, and its test AUC of 0.7980 ± 0.0172 is 0.0301 lower (95% Welch CI [−0.0513, −0.0090]). The tuned non-LLM controller uses the same graph and tools as the agent, which separates the value of the graph from the value of the LLM's decisions.

Confidence intervals and curves: "The experimental analysis should include statistical confidence intervals and search-performance curves averaged over multiple random seeds."

Figure 3(c) shows the anytime curves of both new experiments (the tuned controller and the Full-Only ablation), averaged over 5 seeds with 95% CIs, and the test-AUC difference of each to the complete method is reported with a 95% Welch interval, computed as in Table R1.

Diagnosis, DOA and monotonicity: "explain how the discovered architectures support meaningful cognitive diagnosis … such as DOA across all datasets, and analyses of proficiency interpretability and monotonicity"

Please see our response to uczo (W4). In short, under the standard NCDM definition the DOA of the architectures found by Agent-NASCD depends on their structure. Of the two architectures already shown in the submission's appendix, the one selected on ASSIST09 seed 33 (n82) reaches 0.607 on average over ten splits (0.620 on the split where it was found), above NCDM's 0.596, because it uses a concept-wise student term close in form to NCDM; a program that depends on the student representation through cosine similarities alone (n129) is at the random level. We correct the DOA variant used in the appendix, which inflated the values, and discuss returning standard DOA to the controller as feedback as the next step. We run the standard-DOA analysis on ASSIST09 at 70% training, where both architectures shown in Appendix D were found, and the XES-2k results under the variant used in the submission remain in Table 9.

Larger datasets and search spaces: "more experiments on larger datasets or more complex search spaces would help establish the scalability and generalizability of the proposed framework."

The evaluation covers three datasets, four training ratios and ten seeds, that is, 120 independent searches per NAS method, against ten handcrafted CD models and two CD-NAS methods. The datasets contain 207K–385K responses each, the same range as prior CD architecture search (EMO-NAS-CD and OSCD use ASSIST09, about 325K responses, and SLP, about 57K), and MOOC-2k has 696 concepts. Larger CD benchmarks exist, e.g. ASSISTments 2017 with about 0.94M responses after our preprocessing. Under the same protocol, most of their cost would come from the NAS baselines, since one EMO-NAS-CD search on ASSIST09 already takes 15.2 GPU-days, while each search of Agent-NASCD trains at most about 120 candidates. The search space follows EMO-NAS-CD (25 operators, up to 12 nodes, about 4 × 10^34 raw encodings) and is far beyond enumeration. Applying Agent-NASCD to such larger benchmarks is a direct next step for this work.

Thank you for the detailed response and additional experiments. I have no further questions.

Reviewer_uczoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Agent-NASCD, an agent-based framework for neural architecture search in cognitive diagnosis. It uses an LLM to guide architecture construction and evaluation, maintains previous search results in a Model Optimization Graph (MOG), and adopts proxy and full evaluations to reduce computational cost. Experiments on three datasets show that the proposed method achieves competitive predictive performance while reducing search time compared with existing NAS approaches.

Critical strengths
  1. The paper addresses a practical problem in cognitive diagnosis. Existing NAS methods can be computationally expensive, and using an agent to guide the search based on previous results is an interesting direction.

  2. The overall framework is well designed. The CD-specific tools, MOG, and evaluation strategy work together in a reasonable way. I particularly appreciate the use of historical search results to guide subsequent decisions rather than treating each candidate independently.

  3. The experimental results are encouraging. The proposed method achieves the best average AUC across the reported settings, with particularly clear improvements in search efficiency. The paper also includes comparisons with strong baselines, ablation studies, and analyses of search behavior, which help support the main findings.

Critical weaknesses
  1. The methodological novelty is somewhat limited. Several components, such as tool-based LLM agents, search history tracking, and multi-fidelity evaluation, have been explored in previous work. The main contribution appears to be their integration into a CD-specific NAS framework. It would be useful to clarify what MOG offers beyond a conventional search history.

  2. The efficiency comparison could be more convincing. Different NAS methods follow different search procedures and evaluation budgets. Although the reported reduction in search time is substantial, a comparison under the same computational budget would provide a clearer picture of the search strategy's effectiveness.

  3. The improvements in predictive performance are relatively small in some settings. For example, on ASSIST09 with 60% training data, the AUC improvement over EMO-NAS-CD is only 0.0002. The results suggest that the main advantage of Agent-NASCD lies in search efficiency rather than a large improvement in prediction accuracy.

  4. The evaluation focuses mainly on response prediction. Since the task is cognitive diagnosis, more analysis of the diagnostic quality of the selected architectures would strengthen the paper. The current DOA evaluation is useful but limited in scope.

Questions for authors

Please address the concerns raised in the Critical Weaknesses.

6 comments

Thank you for the positive assessment and the precise questions. The new results are in the revised PDF and summarized below.

W1: "Several components, such as tool-based LLM agents, search history tracking, and multi-fidelity evaluation, have been explored in previous work. … It would be useful to clarify what MOG offers beyond a conventional search history."

We agree that tool-using LLM agents, search histories and multi-fidelity evaluation each exist in prior work. What differs is which decisions the LLM controls. In existing CD architecture search, the search loop is fixed: EMO-NAS-CD trains every candidate its evolutionary operators generate, OSCD scores candidates with shared one-shot weights, and the LLM-assisted evolutionary method for CD uses the LLM as a genetic operator inside a fixed evolutionary loop. Prompting-based optimizers such as OPRO let the LLM propose solutions, but every proposal is evaluated, in a fixed loop. In Agent-NASCD, one controller runs the whole search: from the results recorded so far it decides which architecture to build or edit, which candidate to train next, and when to stop, so the choice of which candidates the training budget is spent on is part of the search policy.

MethodWho proposes candidatesWho decides what to train next and when to stopMemory of past results
EMO-NAS-CDevolutionary operatorsfixed schedule (population × generations)population
OSCDevolutionary operators over a one-shot supernetfixed (supernet training, then subnet search)population
LLM-assisted ENAS for CDLLM as genetic operator in an evolutionary loopfixed evolutionary looppopulation and Hall of Fame
OPRO-style optimizersLLMfixed loop: every proposal is evaluated; stops at a step limit or when no better solution is foundlist of (solution, score)
Agent-NASCDLLM, via write/mutate toolsLLM, from the graph of past resultslineage graph with scores, failed and duplicate candidates

This matters in CD because each model trains in minutes but a search trains many of them, so the number of candidates trained largely determines search cost. Agent-NASCD trains at most about 120 candidates per search, versus on the order of 10^4 for EMO-NAS-CD, at comparable accuracy. Table 3 indicates that the LLM's choices matter: replacing its action selection with fixed rules (Fixed-Policy) used all 120 evaluations and 8.06 training hours yet lowered AUC from 0.8281 to 0.7969. Under the same epoch budget, a tuned graph-aware non-LLM controller with the same tools reaches a test AUC 0.0735 below the complete method (General Response, item 1).

The graph (MOG) and the CD tools are what make these decisions possible. Compared with a conventional (candidate, score) history, MOG records (1) derivation edges, so the controller can attribute a score change to a specific edit, compare siblings and return to a strong parent; (2) separate short- and full-training scores, so an under-trained candidate can be told apart from a poor architecture; (3) failed, pending and duplicate nodes, which are not re-proposed and whose results are reused; and (4) on-demand, bounded observations through tools (view_tree, inspect_node, analyze_tree), so the prompt does not grow with the search. Table 3's Flat-History variant replaces MOG with a flat history; under the same budget caps it reaches a lower test AUC while using 5.75 training hours, more than the 5.35 hours of the complete method.

W2: "Different NAS methods follow different search procedures and evaluation budgets. … a comparison under the same computational budget would provide a clearer picture of the search strategy's effectiveness."

Agent-NASCD reaches comparable accuracy with far fewer candidate trainings, at most about 120 per search while EMO-NAS-CD trains on the order of 10^4, so stopping EMO-NAS-CD at our budget could only lower its result. We agree, however, that Table 2 compares methods under their own procedures and does not isolate the search strategy. To isolate the strategy, we ran a tuned graph-aware non-LLM controller with the same tools and the same budget, 5,000 candidate-training epochs per seed (General Response, item 1). Figure 3(c) adds the anytime curves (best validation AUC vs. epochs spent, mean with 95% CI over 5 seeds) of this controller and of the LLM-controlled Full-Only variant. The tuned controller reaches a test AUC of 0.7546 ± 0.0165, against 0.8281 ± 0.0051 for the complete method (difference −0.0735, 95% Welch CI [−0.0938, −0.0533]), and the Full-Only curve stays above that of the tuned controller for almost the whole search.

W3: "The improvements in predictive performance are relatively small in some settings. For example, on ASSIST09 with 60% training data, the AUC improvement over EMO-NAS-CD is only 0.0002."

We agree. The improvement is significant against EMO-NAS-CD in 5/12 settings and against OSCD in 4/12; elsewhere, e.g. +0.0002 on ASSIST09-60% as you note, it is not. The revision presents the method as matching existing CD-NAS accuracy at lower search cost. Table R1 in the General Response gives each setting's difference with a 95% confidence interval: the differences are largest against EMO-A7 on XES-2k (+0.011 to +0.023) and within 0.004 on MOOC-2k.

W4: "Since the task is cognitive diagnosis, more analysis of the diagnostic quality of the selected architectures would strengthen the paper."

Thank you for pushing on this; re-examining DOA led to a correction and a clearer picture.

  • Metric. The DOA in Appendix Table 9 used a variant that counts pairs of students with equal responses as agreements. Under that variant even randomly assigned proficiencies score about 0.84 on XES-2k, so its absolute values are inflated. The revision reports DOA under the standard NCDM definition, which excludes such pairs (chance level about 0.50), and keeps the submission's table, labeled as this variant: under it, Agent-NASCD remains higher than EMO-A7 at all four training ratios.
  • Result. On ASSIST09 (70% training), we retrained architectures found by Agent-NASCD on the splits where they were found. n82 and n129 are the two architectures already shown, with their full encodings, in Appendix D of the submission, so they were fixed before this analysis rather than selected for it. Their standard DOA depends on the program's structure: 0.620 for n82, the architecture selected on seed 33 (test AUC 0.828, the value reported in the paper), and 0.491 for n129, the seed-7 case study. For reference, NCDM, which has an explicit per-concept proficiency layer, obtains 0.596, EMO-A7 0.514, and random proficiency 0.500 (EMO-A7 and NCDM retrained under our protocol; means over ten seeds). Retrained on all ten splits, n82 reaches a mean DOA of 0.607 ± 0.013 (test AUC 0.8256) and n129 0.493 ± 0.018 (test AUC 0.8207). The difference follows the structure: n82 multiplies a sigmoid-transformed student representation by the concept projection and averages the result, a form close to NCDM and to the computation of proficiency, whereas n129 depends on the student representation through cosine similarities alone, so its proficiency estimates are at the random level.
  • Why, and the next step. Interpretable proficiency depends on the structure of the searched program: searched programs, ours and EMO-NAS-CD's alike, have no explicit proficiency layer, the concept-level proficiency used for DOA is computed after training from the student embedding and the concept projection, and monotonicity in student ability is built in only when the output passes through the non-negative prediction head, which neither n82 nor n129 does (both end in a scalar that is clipped to [0, 1]). Diagnostic quality is therefore good when the selected program uses the student representation concept by concept, as n82 does, and poor otherwise. Because the controller acts on feedback, the standard DOA of each candidate can be returned with its evaluation and used in the controller's decisions. Section 5.5 reports n82 and n129 over ten splits, and Section 6 lists this as the next step.

Thank you for the further clarification regarding the novelty of the proposed method. The additional explanation helps me better understand the differences between MOG and conventional search history, particularly in terms of tracking candidate derivations, distinguishing evaluation results at different training stages, and managing failed and duplicate candidates. These designs allow the search controller to make better use of previous search information. The ablation study replacing MOG with a flat history also provides some empirical support for its contribution. Overall, the additional response partially addresses my concerns regarding the role of MOG. However, the main contribution still appears to lie in integrating existing techniques and applying them to cognitive diagnosis. The methodological novelty could be further strengthened.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for their careful reading. We have uploaded a revised PDF (main text within the 10-page limit). This response summarizes the new results and the shared concerns; a separate comment lists the changes by section, and the individual responses are posted under each review.

Shared concerns and what we did.

  1. A stronger, tuned non-LLM controller under the same budget (jXp3; also uczo, oaEE). On ASSIST09 (70% training), 5 seeds, we ran a graph-aware regularized-evolution controller with adaptive fidelity, tuned on a separate seed. It receives the same training budget as the complete method: 5,000 candidate-training epochs per seed (a hardware-independent unit that matches the size of the complete method's searches, 86 evaluations on average in Table 3, most of them 55-epoch proxy runs) and at most 120 candidate evaluations, the agent's own cap. As in the main experiments, it starts from randomly generated legal programs and shares the agent's candidate space, legality checks, short/full training recipes and final selection rule (validation AUC; the test split is used once, at the end). Its five seeds are among the ten seeds of Table 1. Result (Section 5.3, Figure 3(c), Appendix F, Table R2): the tuned controller reaches a test AUC of 0.7546 ± 0.0165, which is 0.0735 below the complete method's ten-run result of 0.8281 ± 0.0051 (95% Welch CI [−0.0938, −0.0533]). On the same seeds, initial programs and budget, the LLM-controlled Full-Only variant (item 2) is 0.0434 higher than the tuned controller (paired 95% CI [+0.0019, +0.0849]).
  2. Contribution of each component (oaEE, jXp3). All variants in Table 3 ran under the same budget caps; the differences in evaluations and training time come from each controller's own decision to stop, not from different budgets. We also added a Full-Only ablation (every new candidate trained at full fidelity) under the same budget (5,000 candidate-training epochs per seed) over 5 seeds, and new Table R3 (Appendix E) gives the AUC standard deviation of every Table 3 variant, with 95% Welch intervals of the differences to the complete method. Result (Section 5.4, Appendix F, Table R2): Full-Only evaluates 70 models on average, fewer than the 86 in Table 3, and its test AUC of 0.7980 ± 0.0172 is 0.0301 lower than that of the complete method (95% Welch CI [−0.0513, −0.0090]).
  3. Statistical strength of the accuracy gains (uczo, oaEE, jXp3). We agree; see part 2 of this response.
  4. Diagnostic quality (uczo, oaEE). Under the standard NCDM definition of DOA, the diagnostic quality of the architectures found by Agent-NASCD depends on their structure: of the two architectures shown in the submission's appendix, the one selected on ASSIST09 seed 33 scores 0.620 on the split where it was found and 0.607 on average over ten splits, above NCDM (0.596), whereas the seed-7 case study, whose output depends on the student embedding through cosine similarities alone, scores at the random level. We correct the DOA variant used in the appendix, which inflated the values, and treat this structural dependence as the next improvement of the framework (response to uczo, W4).
  5. Clarity and contribution (cfSL; novelty: uczo, oaEE, jXp3). See the individual responses; the revision rewrites the introduction and method sections in plain language.
  6. Scope of the evaluation (oaEE, jXp3). The main comparison covers three datasets, four training ratios and ten seeds, that is, 120 independent searches per NAS method, against ten handcrafted CD models and two CD-NAS methods, and Table 12 reports search resources on all three datasets over the four training ratios. The datasets contain 207K–385K responses each, in the same range as prior CD architecture search, and MOOC-2k has 696 concepts. The controlled comparisons, including the two new ones, use ASSIST09 at 70% training, the setting of Tables 2 and 3, so that their results can be compared directly.
AuthorsOpenReview ↗

General Response (2/3)

Accuracy claims. The gains over the strongest NAS baselines are significant in 5/12 settings against EMO-NAS-CD and 4/12 against OSCD (Wilcoxon, α = 0.05). The revision states the main finding as comparable or slightly better accuracy at lower search cost and removes wording that suggests uniform superiority.

Table R1. Per-setting AUC difference (Agent-NASCD minus baseline) with 95% confidence intervals (also in Appendix A of the revised PDF). The intervals are Welch intervals for the difference of means, computed from the ten-seed means and sample standard deviations reported in Appendix Tables 5–7, so they can be checked directly against the paper. Significance is still judged by the paired Wilcoxon signed-rank test of Table 8, shown alongside; the intervals indicate the size of each difference and its uncertainty.

Settingvs. EMO-A7: Δ AUC [95% CI]Wilcoxon (Table 8)vs. OSCD-AA1: Δ AUC [95% CI]Wilcoxon (Table 8)
ASSIST09 40%+0.0045 [+0.0036, +0.0054]++0.0037 [+0.0033, +0.0041]+
ASSIST09 50%+0.0033 [−0.0052, +0.0118]≈+0.0195 [+0.0110, +0.0280]+
ASSIST09 60%+0.0002 [−0.0103, +0.0107]≈+0.0325 [+0.0220, +0.0430]+
ASSIST09 70%+0.0070 [+0.0033, +0.0107]≈+0.0544 [+0.0507, +0.0581]+
XES-2k 40%+0.0234 [+0.0216, +0.0252]++0.0028 [+0.0007, +0.0049]≈
XES-2k 50%+0.0178 [+0.0162, +0.0194]++0.0019 [+0.0001, +0.0037]≈
XES-2k 60%+0.0139 [+0.0122, +0.0156]++0.0017 [+0.0002, +0.0032]≈
XES-2k 70%+0.0110 [+0.0087, +0.0133]++0.0027 [+0.0007, +0.0047]≈
MOOC-2k 40%+0.0031 [+0.0006, +0.0056]≈+0.0005 [−0.0019, +0.0029]≈
MOOC-2k 50%+0.0015 [−0.0000, +0.0030]≈+0.0001 [−0.0016, +0.0018]≈
MOOC-2k 60%+0.0011 [−0.0002, +0.0024]≈+0.0003 [−0.0013, +0.0019]≈
MOOC-2k 70%+0.0009 [−0.0023, +0.0041]≈+0.0009 [−0.0024, +0.0042]≈

The differences are largest against EMO-A7 on XES-2k (+0.011 to +0.023) and against OSCD-AA1 on ASSIST09 (up to +0.054). On MOOC-2k they are all within 0.004, and against OSCD-AA1 on XES-2k they are +0.002 to +0.003. Where differences are this small, we describe the accuracy as comparable.

AuthorsOpenReview ↗

General Response (3/3)

Our main claim is efficiency. A complete EMO-NAS-CD search on ASSIST09 took 15.2 GPU-days on one RTX 3090 (mean over ten runs, Table 2). Its configuration (population 100, 100 generations) trains on the order of 10^4 candidate architectures per search, whereas Agent-NASCD trains at most about 120 candidates per search (its evaluation budget). At this lower cost, Agent-NASCD's accuracy is comparable to EMO-NAS-CD's, and higher in 5/12 settings. This also bears on the equal-budget question: if EMO-NAS-CD were stopped at Agent-NASCD's budget, the best architecture it could have found by then can only be the same as or worse than the one it finds with its full budget (in Figure 3(a), its best recorded AUC at 120 evaluations is 0.7773, which is 0.0507 below Agent-NASCD). The comparison with a tuned non-LLM controller (item 1 in part 1) additionally shows that the LLM's decisions matter when both controllers share the same tools and budget.

Cost-record coverage (Table 12). Per-run cost logging was added after the main-grid searches had started, so complete cost records exist for the last three seeds of each setting (11 runs for MOOC-2k). The seeds with complete records were fixed by the order in which the runs were executed, so they are independent of the results. With three seeds per setting, the trend is already clear: in every setting a search used 43–122 candidate evaluations, 14–71 LLM calls and on average 8.4–9.6 hours (Table 12), whereas one EMO-NAS-CD search on ASSIST09 took 15.2 GPU-days. This gap of more than an order of magnitude is much larger than the variation across seeds. All new runs in this rebuttal are fully logged (wall-clock time, GPU training time, short/full training counts, cache hits, LLM calls, input/output tokens); Table R2 reports their per-seed evaluation counts, training hours, LLM calls, tokens and API cost.

AI use disclosure. The AI Use Statement in the submission was incomplete. Besides polishing the writing and preparing figures, AI assistants helped write the experiment scripts, set experimental parameters, run the experiments and organize the results. The idea, the method and the core logic of the code are the authors' own. Details are in our response to Reviewer cfSL. The revised PDF corrects the statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.