acceptodds
Under review as a conference paper at ICLR 2027

FlowRSI: Benchmarking Recursive Self-Improvement in Flowchart Parsing

Abstract

Improving after feedback is not the same as learning how to improve: an agent may fix serialization without recovering more visual structure, overwrite a useful strategy, or fail to transfer it to new appearances. We introduce FlowRSI, an offline benchmark of frozen-weight, agent-directed strategy revision in flowchart parsing. Agents inspect labeled examples, receive feedback from two successive mock exams, and commit prompt, memory, or skill states; we independently replay all four states on the same hidden 1,348-image test. Exact graph supervision, strict and content-aware scores, statewise replays, artifact interventions, and clean, corrupted, and unseen-writer strata distinguish feedback attribution, revision stability, and visual transfer. Across nine open systems, final strict relation-set F1 improves for four and declines for five. Qwen3.5-27B gains 55.47 points, yet its content F1 gains only 6.88 while valid output rises from 6.08% to 77.30%: much of the measured gain is contract repair. Qwen3.5-9B instead loses 25.78 points in its last revision, and removing its final artifacts restores 27.48 points in a matched replay. Even the improved 27B state scores 65.87 on held-out scan corruption but only 26.06 on unseen handwriting. The central limitation is thus not merely whether agents can change themselves, but whether they can diagnose the right failure, retain prior competence, and transfer a correction. FlowRSI turns these mechanisms into observable outcomes and suggests that future multimodal self-improvement should couple error attribution with revision safeguards and shift-aware validation.

Primary area:
datasets and benchmarks
Keywords:
Recursive Self-Improvement, Multimodal Agents, Flowchart Parsing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H5JSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FlowRSI evaluates frozen-weight agents that revise persistent prompts, memories, or skills after practice and two mock exams. It independently replays four committed states on the same hidden set of 1,348 flowcharts. The benchmark contains 5,484 images and distinguishes strict relation-set F1, recoverable content, output validity, and visual-domain transfer. Nine systems exhibit improvements, regressions, and early termination; artifact-removal experiments investigate selected final strategies.

Critical strengths
  1. The evaluation separates mechanisms that endpoint accuracy would obscure. For Qwen3.5-27B, the 55.47-point strict-F1 improvement accompanies only a 6.88-point content-F1 improvement and a substantial increase in output validity (Tables 2–3). This usefully demonstrates why serialization repair should not be equated with improved visual understanding.
  2. Independent replay, mandatory final-state reporting, and artifact-removal interventions make destructive revisions inspectable. Removing Qwen3.5-9B’s final artifacts recovers 27.48 points in the matched intervention replay (Table 5). The clean, corrupted, and unseen-writer strata add a useful transfer diagnostic.
Critical weaknesses
  1. System-level conclusions rely on one observed improvement episode per system. Repeated evaluation of fixed states and item-bootstrap intervals do not measure variability in evidence acquisition, diagnosis, or strategy revision. The paper explicitly acknowledges this limitation, but the stability of the improvement and regression patterns remains unresolved for a benchmark intended to compare these capabilities.
  2. The diagnostic value beyond simpler adaptation controls is insufficiently established. Large strict-score changes are dominated by output-contract behavior, yet the evaluation lacks a fixed format-focused strategy and an equal-budget no-feedback revision control. Content F1 and the careful non-causal interpretation are valuable, but they do not establish how much improvement requires feedback-driven strategy revision rather than a better initial instruction or additional trials.
Questions for authors
  1. How stable are final gains, destructive revisions, and early-submission patterns across independently restarted full improvement episodes?
  2. What improvement remains over a fixed format-focused strategy and an equal-budget no-feedback revision control, reporting strict F1, content F1, and validity separately?
2 comments
AuthorsOpenReview ↗

Stage-wise controls expose revision stability and destructive updates

Thank you for treating revision stability, destructive updates, and early submission as measurable outcomes.

The completed trajectories provide the requested stage-wise evidence.

Q1. Stability of gains, degradation and early submission

trajectoryH0 strict F1H1H2H3path event
27B full-feedback A0.03180.24160.57710.3820H3 decline; two early submissions
27B full-feedback B0.03770.12150.34370.3421H2/H3 nearly flat; one early submission
9B full-feedback A0.31140.26930.01750.0662destructive final revision
9B full-feedback B0.30060.00400.00450.0725forced Mock 1 submission
no-feedback C0.03170.14750.10120.2748early Mock 1 submission; partial H3 recovery

The table reports revision paths rather than only endpoint gains. The 27B trajectories finish positive but differ in their final-step behavior. The 9B trajectories finish negative. No-feedback C recovers part of its H2 loss at H3.

Q2. Fixed-format and equal-budget no-feedback controls

controlΔ strict/Content/validitystrict repair/destructioninvalid→valid / valid→invalidsubmission event
no-feedback A+0.6400/+0.0847/+0.8516237/21148/0one Mock 1 submission
no-feedback B+0.5348/+0.0667/+0.7441230/41009/6none
no-feedback C+0.2431/+0.0516/+0.3405145/14491/32one early Mock 1 submission

All three controls use the same action budget, hidden-test protocol, and evaluator, releasing only mock-exam completion. The results show that extra strategy iterations can improve output compliance while also producing destructive revisions. The fixed-format replay supplies a frozen JSON schema and is reported as a format diagnostic.

Dear Authors,

Thank you for the detailed rebuttal and the additional experiments. I appreciate the effort to provide independent trajectories, stage-wise analyses, and stronger feedback controls. These additions improve the transparency of the evaluation. However, my main concerns regarding the benchmark's construct validity and the attribution of recursive self-improvement remain unresolved.

1. The new controls raise a fundamental concern about feedback attribution.

The no-feedback trajectories achieve strict F1 improvements of +0.6400, +0.5348, and +0.2431, compared with +0.3502 and +0.3044 for the reported 27B full-feedback trajectories. More importantly, the no-feedback controls achieve Content F1 improvements of +0.0847, +0.0667, and +0.0516, while the corresponding full-feedback improvements are −0.0046 and +0.0383.

The shuffled-feedback trajectories also reach final strict F1 scores of 0.3109 and 0.4520, comparable to the full-feedback endpoints of 0.3820 and 0.3421.

Although the limited number of trajectories prevents definitive statistical comparisons, these results provide little evidence that meaningful feedback produces a consistent advantage over additional revision opportunities or corrupted feedback.

This directly affects the central construct that FlowRSI aims to evaluate: the ability of agents to use feedback to improve their strategies.

2. Improvements in output compliance remain difficult to separate from substantive self-improvement.

The additional analyses reinforce the observation that much of the strict F1 improvement is associated with output validity and serialization repair. The considerably smaller Content F1 changes, particularly under full feedback, provide limited evidence of improved visual understanding or structural reasoning.

The distinction between strict F1, Content F1, and validity is useful. However, the current evidence leaves the contribution of feedback-specific structural learning insufficiently established. The fixed-format diagnostic also leaves open the need for a quantitative, matched comparison isolating this contribution.

3. The additional trajectories improve reproducibility evidence but leave the broader benchmark conclusions insufficiently supported.

The repeated experiments provide useful examples of improvement, degradation, and early submission. Nevertheless, the substantial variation across trajectories and the limited replication across models make it difficult to draw reliable conclusions about system-level recursive self-improvement capabilities.

I recognize the value of FlowRSI as an auditable evaluation framework for persistent strategy revisions and their failures. However, its central interpretation as a benchmark of feedback-driven recursive self-improvement requires stronger evidence that the measured changes meaningfully distinguish feedback utilization from generic strategy iteration and output-contract repair.

For these reasons, I maintain my original rating of 1 (Clear Rejection). The additional experiments are valuable, but the central methodological concerns remain sufficiently substantial to affect my recommendation.

Reviewer_ej8gOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FlowRSI, a benchmark for evaluating the autonomous improvement capability of multimodal agents in flowchart parsing tasks. With model weights fixed, the agent can autonomously modify its Prompt, Memory, and Skills through practice examples and feedback from two rounds of mock exams, thereby forming multiple persistent policy states. By evaluating different policy states on the same hidden test set and combining metrics such as strict relation-set F1, Content F1, output validity rate, and artifact ablation, the paper analyzes the agentʼs performance improvement, policy degradation, and generalization ability across visual conditions, and conducts experiments on nine open-source models.

Critical strengths

The research question studied in this paper is of high value: it investigates whether an agent can truly achieve effective self-improvement after receiving feedback, rather than merely observing whether its final performance improves.

Critical weaknesses

W1. The real value of the current setting is limited.

From the existing experiments and the paperʼs descriptions, it is difficult to observe that the performance improvements reflect substantive improvements in visual understanding and structural reasoning. A typical example is Qwen3.5-27B, whose output-format validity rate improves significantly, but whose Content F1 improves by only 6 points. This indicates that the modelʼs most significant performance improvement mainly comes from fixing the JSON output format. The paper lacks detailed statistics and analysis of specific errors, and it also does not evaluate stronger closedsource models, especially since they are stronger in output formatting. Therefore, it is difficult to generalize the current conclusions to other tasks and real-world environments.

W2. Lack of validation of the improvement mechanism and stability.

For the current task and setting, I believe that simply adding some requirement specifications to the prompt can already significantly improve the modelʼs results. This raises a problem that even without feedback results, I can adjust the formulation of the structured specification based on previous run results. In other words, how much the current feedback actually contributes to improvement requires further investigation. At the same time, the paper conducts only one autonomous improvement experiment for each model. The paperʼs approach is that the model makes only one decision and then performs three evaluations on the same decision. Therefore, it is impossible to determine whether these results are stable and reproducible, and they are insufficient to reflect the agentʼs true capabilities in processes such as sample selection, error disputes, and decision revision.

Questions for authors

Q1. Could you further provide detailed statistics on different types of structural errors (such as missing relations, incorrect arrow directions, and incorrect relation types) before and after improvement, so as to more clearly demonstrate the specific changes in the agentʼs visual understanding and structural reasoning?

Q2. Could you add control experiments with no feedback or with only prompt optimization, in order to analyze how much of the current performance gain truly comes from the agentʼs utilization of feedback information?

Q3. Would you consider conducting multiple independent full selfimprovement experiments on representative models and reporting the performance changes and degradation ratios across runs, so as to verify whether the currently observed improvement and degradation phenomena are stable and reproducible?

1 comment
AuthorsOpenReview ↗

Paired diagnostics separate graph recovery from output compliance

Thank you for separating graph recovery from output-contract compliance.

The following table makes that separation explicit.

W1/Q1. Structural understanding versus formatting

trajectoryΔ strict F1Δ Content F1Δ validity
27B full-feedback A+0.3502−0.0046+0.5082
27B full-feedback B+0.3044+0.0383+0.4006
9B full-feedback A−0.2452−0.0262−0.5297
9B full-feedback B−0.2281−0.0447−0.5549
no-feedback A+0.6400+0.0847+0.8516
no-feedback B+0.5348+0.0667+0.7441
no-feedback C+0.2431+0.0516+0.3405

For the trajectories with saved structural audits, the paired transitions are:

trajectorystrict repair/destructioninvalid→valid / valid→invalid
27B full-feedback A233/4756/17
27B full-feedback B118/13328/31
no-feedback A237/21148/0
no-feedback B230/41009/6
no-feedback C145/14491/32

No-feedback C increases strict F1 by 0.2431, Content F1 by 0.0516, and validity by 0.3405. The 27B full-feedback A trajectory improves at H2 and declines at H3. These paired transitions expose contract repair, relation recovery, and destructive revision as distinct outcomes.

W2/Q2. Feedback mechanism and prompt-only controls

The three no-feedback trajectories improve the output contract by +0.6400, +0.5348, and +0.2431 strict F1. Their paths differ, and no-feedback C includes an early submission and an H2-to-H3 recovery. The shuffled-feedback controls retain the full payload shape but corrupt per-example correspondence. Their final strict F1 values are 0.3109 and 0.4520. The fixed-format replay supplies a frozen JSON schema without semantic mock feedback and serves as a separate format diagnostic.

Q3. Multiple independent episodes

Every added trajectory is reported separately. Means and standard deviations summarize families with multiple completed trajectories. Negative trajectories, forced submissions, and destructive revisions remain in the evidence.

Reviewer_VsLpOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces an offline benchmark FlowRSI, focusing on flowchart relation-extraction. The paper's methodology of using frozen-weight models (in a fixed host runtime) revising its prompts across a practice stage and two mock-exam feedback rounds does help to evaluate separately, the impact of the learning on the endpoint scoring. However, as the authors have pointed out, the lack of a multi-seed replication does cause some concern on the final conclusion.

Critical strengths

The critical strength of the paper is:

  1. The curation of a 5,484-image, exactly scored flowchart benchmark that separates structural parsing from output validity. This would be genuinely useful for further work in this area.
  2. The framing of multimodal recursive self improvement in an auditable fashion where the mock exams and same-test show the consequences of the commitment. The breakdown of the problem into intermediate steps allows us to identify the impacts of feedback at each stage.
Critical weaknesses

The critical weakness of the paper are as follows:

  1. The lack of multi-seed replications. Effectively, items are only resampled within one trajectory. Hence, we will not be able to tell if a different sampling of the agent's decision would lead to a different conclusion.
  2. The authors should consider a no-feedback control, so as to test their claim that the benchmark isolates if agents convert feedback into improvement. Also missing is a control group where the feedback gets shuffled, and whether there would be significant differences between them.
  3. The task might be too tightly designed. It seems a lot of the improvements in output of the agents are to deal with the JSON format. This might not be a good inidcation of whether they have learned the context of the material, only that of the output format.
Questions for authors
  1. Given that every result is based on one run per system, how confident are the authors of their claims?
  2. Did the authors consider adding a no-feedback or shuffling the feedback to see if there are significant differences?
  3. Would relaxing the JSON formatting lead to significant changes in the outcomes?
2 comments
AuthorsOpenReview ↗

Independent trajectories quantify revision stability and feedback attribution

Thank you for focusing on launch stability, feedback attribution, and metric interpretation.

The completed controls address these three points directly.

Q1. One run per system and confidence

The original item-level bootstrap described variation within one trajectory. We added two independent Qwen3.5-27B full-feedback trajectories and two independent Qwen3.5-9B full-feedback trajectories.

trajectoryΔ strict F1Δ Content F1Δ validity
27B trajectory A+0.3502−0.0046+0.5082
27B trajectory B+0.3044+0.0383+0.4006
9B trajectory A−0.2452−0.0262−0.5297
9B trajectory B−0.2281−0.0447−0.5549

The two 27B trajectories have mean changes of +0.3273 strict F1 (SD 0.0324), +0.0168 Content F1 (SD 0.0303), and +0.4544 validity (SD 0.0761). One reaches a higher H2 score and declines at H3. The other remains nearly flat from H2 to H3 and ends with an early submission. The two 9B trajectories are both negative. Revision stability and revision damage are measured trajectory outcomes.

Q2. No-feedback and shuffled feedback

Three no-feedback trajectories release only a mock-exam completion marker. They release no predictions, scores, gold relations, error lists, or mock images.

controlH0 strict/Content/validityH3 strict/Content/validityΔ strict/Content/validity
no-feedback A0.0358/0.6193/0.05560.6758/0.7041/0.9073+0.6400/+0.0847/+0.8516
no-feedback B0.0372/0.6164/0.05560.5720/0.6831/0.7997+0.5348/+0.0667/+0.7441
no-feedback C0.0317/0.6101/0.05790.2748/0.6618/0.3984+0.2431/+0.0516/+0.3405

No-feedback C contains one audited early-final Mock 1 submission, 145 strict repairs, 14 strict destructions, 491 invalid-to-valid transitions, and 32 valid-to-invalid transitions. Its H2 score is low and partially recovers at H3. Across the three no-feedback trajectories, the mean H3−H0 changes are +0.4726 strict F1 (SD 0.2056), +0.0677 Content F1 (SD 0.0166), and +0.6454 validity (SD 0.2695).

Two shuffled-feedback trajectories preserve the feedback payload and release timing while permuting per-example correspondence.

controlH0 strict/Content/validityH3 strict/Content/validity
shuffled-feedback A0.0279/0.6182/0.04970.3109/0.6508/0.4592
shuffled-feedback B0.0304/0.6101/0.05120.4520/0.6400/0.6335

These controls show how much contract repair can occur without reliable per-image feedback. Each trajectory is reported separately as a diagnostic.

Q3. Relaxed JSON formatting

Strict F1 measures the complete machine-executable relation set. Content F1 measures recoverable relation content. Validity measures compliance with the executable output contract. The structural audit adds missing, spurious, reversed-direction, wrong-type, partOf, and invalid-serialization counts.

Thanks for the response. Unfortunately, I remain unconvinced that my main concerns have been adequately addressed.

Seems like the no-feedback controls show improvements comparable to, or even larger than, those under full feedback? So now the question is whether the benchmark actually measures agents' ability to learn from feedback or not? And the shuffled-feedback results also reinforce this concern.

While the additional trajectories are helpful, reporting Content F1 does not directly address whether relaxing the JSON constraints would change the conclusions.

I appreciate the authors' efforts and will keep my score unchanged (in light of other reviewers' sentiment, I may have been over-lenient in the rating). However, I believe the evidence does not yet sufficiently support the paper's central claims about feedback-driven improvement, and these claims should be substantially tempered. It seems that recursive self-improvement is too much of a claim here.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We agree that the new controls do not support the stronger claim that meaningful feedback consistently outperforms generic revision opportunities or corrupted feedback. The contribution of FlowRSI is therefore best understood as an auditable benchmark for measuring staged strategy revision, failure recovery, output-contract repair, and transfer to a hidden test set—not as evidence that current agents already achieve general recursive self-improvement or reliably learn from feedback. FlowRSI provides a fixed offline protocol with persistent strategy states, two unreleased mock exams, hidden-state replay, independent trajectories, and separate strict F1, Content F1, and validity metrics. This decomposition makes it possible to determine whether an apparent improvement comes mainly from serialization compliance, relation-content recovery, or broader cross-stage transfer. The no-feedback and shuffled-feedback results are consequently not a failure of the benchmark; they reveal an important limitation of current agents and of endpoint-only RSI evaluation: improvement can arise from repeated revision and format repair without effective feedback use. We will accordingly temper the paper’s claims and present FlowRSI as a diagnostic framework for exposing and quantifying these distinct behaviors, while identifying reliable feedback utilization—and its separation from generic iteration—as an open problem for future RSI research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.