- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SemanticRelaxation, a framework for optimizing discrete artifacts in agentic workflows by operating in a joint semantic embedding space. The framework consists of two components: Semantic Representation Learning, which aligns natural-language descriptions with corresponding artifacts using a Barlow Twins-style objective and a topological regularizer; and Barycentric Semantic Optimization, which estimates a semantic update direction from performance-weighted barycentric aggregation of evaluated candidates and accumulates historical directions as momentum. The resulting direction guides retrieval of relevant historical transitions, which LLM agents then use to refine the description and generate new candidate artifacts. The method is evaluated on benchmarks spanning Agentic Workflows, Code Generation, Molecular Editing, and Neural Architecture Search.
Critical strengths
- The problem motivation is genuine: translating discrete execution feedback into principled optimization directions for agentic workflows is an open and practically important challenge. The paper proposes a method that explores this direction and evaluates it across several distinct domains.
Critical weaknesses
-
The presentation of the paper is not very clear. Although the problem is intuitively understandable, neither the problem definition nor the method is rigorously established. The notation is also inconsistent. These issues make it difficult to follow the paper and to evaluate its contribution, novelty, and limitations.
-
In the method, the first step aligns descriptions and artifacts in a shared space, and BSO then estimates a direction to improve performance. The method achieves the best results on most benchmarks. However, the contribution of each component is not sufficiently supported by the experiments. For example, the effect of BSO should be studied more rigorously on the main benchmarks, such as by replacing it with a random direction or removing the momentum. Also, different tasks have different characteristics, and it is unclear why the method works well across them and when it would not.
Questions for authors
-
Could the authors elaborate on the gain from the alignment design? Since the semantic direction is only used to retrieve historical transitions, and the artifacts are still generated by the LLM agents, how much of the improvement comes from the aligned embedding space itself rather than from the retrieval-augmented refinement?
-
Compared with alternative ways of choosing the update direction, such as random directions or independent per-candidate updates, what are the experimental advantages of Barycentric Semantic Optimization?
-
Since the tasks are very diverse, what do the authors think makes the method effective across them, and in which settings would they expect it to help less?
4 comments
Q1
Could the authors elaborate on the gain from the alignment design? Since the semantic direction is only used to retrieve historical transitions, and the artifacts are still generated by the LLM agents, how much of the improvement comes from the aligned embedding space itself rather than from the retrieval-augmented refinement?
Response.
Although the LLM generates artifacts, the representation determines which editing evidence it receives and how that evidence relates to the intended update. We therefore compare representations while retaining the optimization and generation pipeline.
Alignment objectives — Section 3.4, Table 5; Appendix A.1.
| Alignment | ALFWorld | SciWorld | M3Tool |
|---|---|---|---|
| Normalized | 0.731 | 0.642 | 0.707 |
| Multi-positive contrastive | 0.761 | 0.640 | 0.683 |
| SRL | 0.784 | 0.668 | 0.756 |
The alternatives share encoders, projection architecture, training split, initialization, and minibatch schedule. Representations remain fixed during BSO and refinement, connecting alignment design to optimization outcomes. Both alternative objectives omit topology regularization, as specified in Appendix A.1; Table 4 separately evaluates the effect of removing this regularizer.
Historical context controls — Section 3.3, Table 4. Score-ordered history without SRL yields 0.694/0.489/0.622, or 9.0–17.9 points below full performance. Artifact-only optimization retains BSO and retrieval over frozen features but omits the description encoder and learned projections, yielding 0.731/0.610/0.659. This evaluates the joint representation as a whole.
Uniform selection retains eligible records but removes directional ranking, yielding 0.746/0.639/0.683, a 2.9–7.3-point reduction. These conditional effects support both representation and selection within the same refinement pipeline; their contributions interact.
Appendix E, Table 10 provides complementary semantic checks. Redundancy reduction improves matching after meaningful module changes from 28.95% to 44.74%; topology preservation raises it to 50.00% and improves rephrased-query matching from 16.67% to 35.71%. Tables 4–5 additionally test optimization outcomes.
Q2
Compared with alternative ways of choosing the update direction, such as random directions or independent per-candidate updates, what are the experimental advantages of Barycentric Semantic Optimization?
Response.
Main-benchmark controls — Section 3.3, Table 4; Appendix A.1.
| Update variant | ALFWorld | SciWorld | M3Tool |
|---|---|---|---|
| Score-target gradient, w/o BSO | 0.739 | 0.623 | 0.671 |
| Independent candidate, w/o | 0.739 | 0.635 | 0.683 |
| No momentum, w/o | 0.754 | 0.640 | 0.683 |
| Full BSO | 0.784 | 0.668 | 0.756 |
Appendix A.1 specifies the controls:
- Independent candidate: substitute a uniformly sampled candidate's normalized embedding for the barycentric direction. This omits centering, whitening, and performance weighting in direction construction, retaining the step mapping, momentum, retrieval, generation pipeline, and budget.
- No momentum: retain the current local BSO update but remove accumulated historical directions.
- Score-target gradient: retain learned representations and retrieval, replacing BSO with the specified score-target gradient without whitening or momentum.
Full BSO outperforms independent candidate updates by 3.3–7.3 percentage points, the no-momentum variant by 2.8–7.3 points, and the score-target-gradient variant by 4.5–8.5 points. The independent-candidate comparison tests the full local direction construction, while the momentum control examines accumulation more specifically.
Random directions — Section 3.4, Figure 4; Appendix A.5. On 50 paired Semantle instances with shared representation, generation pipeline, initialization, and guess budget, a same-norm random tangent query reduces discovery from 56% to 42%. Reversal yields 36%, and gain-ranked retrieval yields 46%. This pure random-direction control is on Semantle; Table 4 provides candidate-based and momentum controls on the main benchmarks.
Appendix D clarifies that reversal preserves selected records at a fixed state while exchanging labels, testing interpretation rather than selection.
Appendix E, Figure 9, compares 12 frozen states under three paired seeds. Our method and BOPRO improve the parent in 3/36 trials, versus 1/36 for gain-ranked retrieval and 0/36 for uniform selection. Especially notable are our larger positive state-level mean gains compared with BOPRO's near-zero positive means, revealing differences in gain magnitude despite identical improvement counts. Overall means remain negative, so this does not imply reliable improvement at every step.
Q3
Since the tasks are very diverse, what do the authors think makes the method effective across them, and in which settings would they expect it to help less?
Response.
The tasks share an optimization interface: an editable artifact, a description, external evaluation, and prior edits that can inform subsequent changes. Task-specific scores weight candidate updates and orient retrieved evidence, while generators, constraints, and metrics remain domain-specific.
Section 2.2 explicitly states that representations are learned separately for each domain. The experiments evaluate a common procedure with domain-specific representations, rather than one embedding model transferred unchanged across tasks. Section 3.1 and Appendix A specify these settings.
We expect benefits when descriptions capture behaviorally relevant differences, feedback distinguishes candidate quality, and prior edits offer transferable guidance. Section 3.4, Figure 5, and Appendix E, Figure 12, trace this transfer through instructions and concrete workflow changes. These examples support this interpretation without establishing a universal explanation across domains.
Appendix F, “Limitations,” now discusses poor descriptions, scarce edit records, sparse or noisy feedback, and geometry that misses task-relevant relationships. Prior SRL training also adds overhead that may limit benefits for short searches. Directional alignment does not guarantee validity or improvement.
The revised discussion includes observed limits:
- Section 3.2, Table 1: the method leads on four of six workflow benchmarks, not ALFWorld or PDDL. Macro-level guidance may insufficiently specify the concrete action sequences needed in these tasks.
- Section 3.5; Appendix E, Tables 7–8: molecular gains vary by objective. Under loose multi-objective criteria, accuracy increases from AgentDrug's 31.3% to 60.3%, but validity decreases from 78.7% to 72.3%.
- Appendix E, Figures 9–10: negative mean proposal gains and target-dependent failures show that aggregate search improvements can coexist with unsuccessful proposals and difficult targets.
We appreciate your comments, which prompted clearer definitions, more direct controls, and explicit applicability conditions.
Dear Reviewer A8a1,
We would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. We greatly appreciate your constructive feedback, which has helped us identify important areas for improvement in both the methodological presentation and experimental validation.
Guided by your valuable suggestions, we have carefully revised the problem formulation and notation, conducted additional ablation studies to examine the individual contributions of SRL and BSO, including alternative update directions and momentum, and expanded our discussion of the method's applicability and limitations across different tasks.
We sincerely understand how demanding this period can be. However, with the discussion period nearing its conclusion, we would be extremely grateful if you could kindly review our responses and share your thoughts on whether any concerns remain, should your schedule allow. Your feedback would mean a great deal to us and would be immensely helpful in further strengthening our work.
Thank you once again for your generous time, careful evaluation, and thoughtful consideration.
We sincerely thank you for your constructive feedback and for recognizing the problem's importance. We have clarified the formulation and notation, expanded component-level experiments on the main benchmarks, and specified conditions and limitations across domains.
References in our responses refer to the revised manuscript; reviewer comments are reproduced verbatim.
W1
Response.
We revised Sections 2.1–2.4 and expanded implementation details in Appendices A, C, and D.
Section 2.1, Eq. (1) now defines the constrained discrete objective: x⋆∈argmaxx∈X(c),V(x;c)=1U(x;c), where X(c) is the artifact space, V is validity, and U is task performance. We distinguish the artifact xt, descriptive summary dt, and modification instruction r, as well as evaluated candidate comparisons Bt and historical transitions Mt.
Section 2.2 distinguishes normalized embeddings from standardized cross-correlation inputs and identifies topology affinities derived from frozen descriptions. It states that SRL learns without performance labels and remains fixed during artifact optimization. Section 2.3 distinguishes the local target, tangent momentum, and accumulated latent target. Appendix C specifies regularized whitening and singular-covariance/zero-update cases.
Section 2.4, Eq. (8) defines retrieval through the alignment of the BSO query and transported edit directions, signed by observed performance changes. Algorithm 1 and Appendix D specify eligibility, absolute-score ranking, directional labels, ties, and empty retrieval. Figure 3 and Eq. (9) connect this evidence to instructions, artifact generation under task constraints, and evaluation.
The formulation separates the objective from latent guidance: candidates are generated discretely and assessed through external evaluation.
W2
Response.
We expanded Section 3.3, Table 4, with representation, direction, momentum, retrieval, and refinement ablations on ALFWorld, SciWorld, and M3Tool. Section 3.4 and Appendix E add alignment comparisons and directional-search diagnostics.
Appendix A.1 specifies shared GPT-4o-mini generation, initialization, search seed 0, evaluation populations of 134/26/82 instances or cases, and eight attempts including initialization, failures, and cache reuse. Tables 4–5 report best observed scores within this budget on the shared populations. Below, Q1 addresses alignment, Q2 alternative directions, and Q3 applicability and limitations.