- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper automates OPC recipe development. Its VPE harness, the Pi agent harness plus domain prompts and tools, lets an LLM change global parameters, add local correction rules, or run diagnostic trials, and Calibre scores each trial once. After every trial, two other agents follow the ACE idea to look back at the result and add what they learn to a skill bank. On a FreePDK45 benchmark the method does well against six general agent methods and Bayesian optimization.
Critical strengths
- Though I don't work on OPC myself, I think the application has its value. There isn't much work on agents for OPC yet, and here a commercial lithography tool is in the loop. The constraints the paper works under, such as mask rules, SRAF printing, and caps on mean EPE and PVB, might well be what an engineer actually cares about.
- The paper provides its own evaluation protocol for agentic OPC recipe search.
Critical weaknesses
-
The comparison with the baselines in Table 1 does not show how much of the gain comes from the experience mechanism. In the supplementary code, the method presets for the memory baselines (Textual Memory, ReasoningBank, ACE) and ReAct turn off the engineering guidance that VPEvolve gets, and VPEvolve and its ablations also get a separate prompt on how to search and a longer instruction. The paper doesn't say whether these baselines get the guidance. Frozen Experience, which never updates its experience, already beats every baseline in Table 1, so much of the gap may come from the prompts rather than from what the agent learns. The continuation study in Section 4.4.1 does hold the guidance fixed, but its intervals include zero and each case is run only once.
-
It is not clear why OPC needs its own harness. The introduction lists what recipe work needs (process knowledge, layout analysis, recipe editing and tool feedback), but these could be given to a general self-evolve harness such as Hermes Agent as tools and domain prompts. Also, since the method is named for evolving, I expected some discussion of methods that optimize the harness itself, such as Meta-Harness [1] or AutoHarness [2], and neither is discussed. AnalogAgent [3], which already carries ACE-style memory into EDA, isn't discussed either. The code also reuses ACE's prompt templates (its NOTICE file says so), which the paper doesn't mention.
References
- [1] Lee et al. Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052.
- [2] Lou et al. AutoHarness: improving LLM agents by automatically synthesizing a code harness. arXiv:2603.03329.
- [3] Bao et al. AnalogAgent: Self-Improving Analog Circuit Design Automation with LLM Agents. arXiv:2603.23910.
Questions for authors
Clarification requests
- What does the VPE harness give that a general self-evolving harness with the same domain prompts can't? To demonstrate the difference, I recommend running Hermes Agent with the same prompts and tools on a few cases.
- Did the agent baselines in Table 1 other than Static Harness get the engineering guidance that VPEvolve gets? The paper doesn't say. If they did not, rerun ACE, the method VPEvolve's experience mechanism adapts, with the same guidance on a few cases.
- Why is Bayesian optimization the only traditional method in the introduction and the only numerical search baseline, and what standing does it have in OPC practice?
- Figure 7 is meant to test whether VPEvolve depends on a particular LLM (Section 4.4.4), but every API model ends behind the default 27B model on both layers. Why is that, and does the gap hold if the context length is matched? Rerun one or two API models with the full context on a few cases, or explain where the gap comes from.
- What should an NLP reader from the LLM agent community take from this work beyond OPC? For example, how much of the gain comes from domain guidance versus accumulated experience, and what kind of feedback makes the experience useful? The ablations in Table 2 start on this, and I think they deserve a short paragraph, or even a section in the experiments, on what they teach the community about agents.
Suggestions further
- Section 4.4.1 says the gains "show" that curated experience can improve later decisions, and Section 4.4.2 presents local corrections as another way to cut the worst errors, but all four paired intervals in Table A4 include zero. I would say tone both statements down to what the intervals support.
- Rewrite the contribution list. The first bullet calls recipe development "a learning problem", the text under it says "sequential search", and Section 3.1 says "sequential decision problem" (I come back to this in the next suggestion). I would pick one term and say in a sentence what it adds over the RL formulation in Intelligent OPC Engineer Assistant [4], which the paper itself cites and which framed recipe search as reinforcement learning back in 2024. The third bullet is titled as an outcome, self-evolving experience, but describes a mechanism, reflection and curation, so it should say which one is claimed. The fourth bullet is titled "an open testbed", but the recipe task itself still needs Calibre, which is closed source software. The title seems to promise more than what is released.
- I think the introduction should be rewritten around one thread. Right now the paper moves between the harness, the memory mechanism and the "learning problem" of the first contribution, and the last one confuses me a lot. The formulation has no loss function, and I don't see how it can be framed as a learning problem. At best I think it is a memory accumulation problem. The contribution also says it is a search problem, but I can't find it formally framed as one in the paper, which is really confusing.
References (continued)
- [4] Chen et al. Intelligent OPC Engineer Assistant for Semiconductor Manufacturing. arXiv:2408.12775.