acceptodds
Under review as a conference paper at ICLR 2027

TrustTrace: Code Generation and Revision with Hierarchical Program Knowledge

Abstract

Code generation requires algorithmic knowledge about a task and the computations that implement a solution. A central challenge is to preserve the relation between a solution's global strategy, the responsibilities of its components, and their executable behavior. We present TrustTrace, a framework for code generation and revision based on hierarchical program knowledge. We organize a solution into task strategies, module descriptions, and code implementations. Module descriptions specify what each component computes, while interfaces and dependencies determine how components work together. We use this structure to learn from complete solutions and to examine the behavior of individual modules when revising a program. During learning, we assign each structured response a quality weight that determines its contribution to training. During revision, we compare observed module inputs and outputs with the responsibilities stated in the descriptions. This connects the organization of algorithmic knowledge with the execution of the resulting program. Experiments show that quality weighting raises Test whole-problem accuracy over ordinary modular SFT from 62.05% to 63.05% with supplied plans and from 17.93% to 20.92% without plans, while module traces with corresponding guidance raise revision success over score-and-error feedback from 66.50% to 67.51% with five candidates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Generation, Large Language Models, Program Repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tGCuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TRUSTTRACE, which organizes programs into “task strategy–module descriptions–code implementations,” weights training examples by response-quality scores, and uses module execution traces to guide revisions. On a held-out test set from the authors’ benchmark, it outperforms standard modular SFT.

Critical strengths
  1. This paper proposes a framework called TRUSTTRACE for synthesizing data and optimizing model training.
  2. The method shows a modest performance improvement over standard SFT.
Critical weaknesses
  1. The paper applies some quality controls to the synthetic data, but does not establish that the generated code is reliably correct across all valid inputs.
  2. The fine-tuning experiments focus on Qwen2.5-Coder-7B and DeepSeek-Coder-6.7B, leaving TRUSTTRACE’s effectiveness on a broader range of open- and closed-source models unclear.
  3. The performance gains over standard SFT are modest. For example, Plan StrictAcc improves by only 1.00 percentage point, and it remains unclear whether this gain would persist at larger model scales.
Questions for authors
  1. Will the code and data for the self-constructed dataset be released?
  2. Why are comparisons with commonly used models missing for the self-constructed dataset?
Reviewer_8RA5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This research proposes a program generation and revision framework called TrustTrace based on LLM. It divides the program into multiple modules and specifies the function, input/output, and corresponding code for each module; during training. Also, the entire "description + code" answer is scored to allow the model to learn more from high-scoring answers. In real running, the input, output, and error messages of each module will be fed back to the model to help it modify the program.

Critical strengths

This research links the module function description, code implementation and operation records together. The comparison between weighted and unweighted training is sufficient, supporting the effectiveness of the weighted mechanism under the current settings. Also, the operation feedback revision has also achieved initial gains. In addition, the author separates the feedback test from the final evaluation test, and provides multiple internal controls, facilitating the judgment of the benefits and limitations of the method. Therefore, this research has certain practical significance.

Critical weaknesses
  1. What are the differences between the components of this paper and the existing research? The authors do not sufficiently elaborate on this distinction for all these components. Also, ablation experiments for each independent module are absent, so it is difficult to demonstrate whether each module is more advanced than existing solutions. In addition, the article enhances the efficiency and quality of task completion by providing explanations for the modules. However, it has not taken into account the time issue caused by the increase in running memory and token consumption. The additional cost is a problem that might affect practicality.
  2. In this paper, each function only keeps the input and output records of the first call in each test, and may miss critical fault messages in subsequent calls. This paper has not analyzed the information coverage of this logging strategy, nor evaluated its impact on fault localization and program repair through experiments. Also, some of the words and sentence structures in the article are rather obscure, making it difficult for readers who have little knowledge of this field to understand.
Questions for authors
  1. Please clarify the real boundaries between the modules in this article and existing work (e.g. CodeChain , FunCoder...).
  2. Please clarify whether the actual revision prompt includes the original module descriptions, interface specifications, and dependencies.
Reviewer_zZu7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

TRUSTTRACE represents programs as sequences of module descriptions and code blocks, using one scorer for training-response weighting, candidate reranking, and revision with module execution traces. Its dataset contains 10,034 problems reformatted from rStar-Coder. Plan supplies the reference algorithmic rationale at evaluation, while Direct supplies only the problem statement. The paper reports that weighted SFT on Qwen2.5-Coder-7B raises StrictAcc from 62.05% to 63.05% for Plan and from 17.93% to 20.92% for Direct. In a 397-problem revision experiment, adding traces raises success from 66.50% to 67.51%.

Critical strengths
  1. The design is coherent and the weighting control is explicit. One modular representation is reused for supervision, scoring, and revision. Ordinary and weighted SFT share the backbone, targets, and normalization, and all-unit weights recover the ordinary objective, making the controlled difference interpretable.
  2. The objective decomposition and negative-result reporting are valuable. Appendix A.8 decomposes the weighted objective into a uniform component and a high-weight-subset component, exposing degenerate cases. The paper also reports drops in partial test coverage, completeness, and parseability rather than presenting favorable metrics alone.
Critical weaknesses
  1. The reliability of the main gains and their attribution to traces are unestablished. The Plan improvement corresponds to about 10 of 1,004 problems, and the trace improvement to about four of 397. A single seed without per-problem paired statistics cannot establish stability. The standard error of one proportion is not the standard error of a paired difference, so it cannot by itself establish nonsignificance. More importantly, adding traces also changes revision instructions from allowing rewrites to preserving correct modules, preventing isolation of the trace contribution. Hyperparameter variation and Direct AvgAcc remaining below the untuned backbone further motivate separate interpretations by metric and condition. A validation-selected configuration failing to maximize test performance is not itself an error.
  2. Evidence for the mechanism, task scope, and semantic measurement is insufficient. The weight mapping has a transition band on the order of 10⁻⁶, making it approximately a two-level scheme, but its governing high-weight fraction q_H and the score distribution of actual training targets are unreported. Constant CodeBLEU features, threshold definitions, and data filtering need clarification. Plan measures implementation from a supplied rationale and should be distinguished from Direct generation from a problem statement. Existing correspondence metrics partly reward modular format or identifier overlap and cannot replace semantic validation. Section 5.4 lacks the joint analysis, and the full weighting method has not been established on a second backbone or independent benchmark. These issues affect both main contributions and support a clear rejection.
Questions for authors
  1. Existing paired records: Please provide seed counts and per-problem success/failure transitions for the three main comparisons, especially the discordance table for Errors versus Traces on 397 problems. Are repeated training results available?
  2. Weighting mechanism: What are q_H and the score distributions for Plan and Direct? Why do the main text and appendix specify different transition bands, and which is implemented?
  3. Trace control: Does any existing experiment hold revision instructions fixed while changing only the trace evidence? If not, please acknowledge that the present experiment does not isolate traces.
  4. Implementation and data: Are all four CodeBLEU features always zero? Please provide input, filtered, and retained counts for data construction and explain differences in candidate pools and first-choice results between Tables 3 and 14.
  5. Claim boundaries: Please distinguish the capabilities supported by Plan and Direct, explain the Direct StrictAcc/AvgAcc trade-off, and provide the analysis announced in Section 5.4 or clarify its status.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.