acceptodds
Under review as a conference paper at ICLR 2027

Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning

Abstract

As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fundamental challenges for AI systems: the relationships across representations are inherently one-to-many, supervision is ambiguous and costly, and model optimization lacks a principled signal that is both direction-adaptive and representation-generalizable beyond task-specific objectives. We introduce CoCoEvolve to improve consistency across chart, table, and code representations. Instead of treating cross-representation mapping as a one-to-many problem, we define explicit one-to-one correspondences and optimize models using agreement between representations, without additional annotations. During training, CoCoEvolve@Train performs co-evolution across the chart-table-code cycle, while CoCoEvolve@Test applies the same consistency objective at inference time for test-time co-optimization. We also present CoCoEvolve@Eval, an evaluation suite covering all six cross-representation tasks. Across four benchmarks, CoCoEvolve improves performance in both training-time and test-time settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
cross-representation learning, self-supervised learning, sefl-improving, self-evolving, reinforcent learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_GGRGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes CoCoEvolve, a framework for converting between chart images, data tables, and visualization code. It adds a textual constraint description to resolve the one-to-many ambiguity between these representations.

  • This paper trains a vision-language model and a language model jointly with reinforcement learning. The reward is based on cross-representation consistency, including code executability, agreement between the two models’ outputs, and visual similarity to the input chart. The same reward is used at test time to select the best candidate outputs.

  • This paper introduces an evaluation suite covering all six conversion directions, combining rule-based metrics with multi-dimensional LLM and MLLM judges.

Critical strengths
  • The paper takes advantage of the fact that visualization code can be executed and rendered, which turns agreement between chart, table, and code into an automatically computable reward. This matters because labeled cross-representation data is scarce and costly. A scalable, verifiable signal that needs no human annotation addresses a real bottleneck in chart understanding.

  • The improvements hold across many open models, RL algorithms, and both in-domain and out-of-domain chart benchmarks, including harder chart modification tasks. This breadth suggests the gains are not specific to a single model or setting, which supports the framework’s practical usefulness.

Critical weaknesses
  • The title and framing overstate the scope. The title presents a general self-supervised cross-representation learning method. However, the rewards, metrics, and all experiments are specific to the chart–table–code setting, and no evidence is given that the method transfers to other representations.

  • The core claims are not supported by evidence. The paper does not verify that constraint descriptions remove the ambiguity between representations. It also does not test whether the gains come from the two models’ mutual agreement rather than from a standard reward that compares the rendered chart with the input chart. As a result, the novelty of the contribution is unclear.

  • The evaluation may overstate the gains. The test-time method is compared unequally with baselines, the training rewards overlap with the evaluation metrics, and the reference answers are LLM-generated without human verification. This makes the true improvement difficult to assess.

Questions for authors
  1. Constraint descriptions: Section 3.1 introduces constraint descriptions to turn one-to-many mappings into one-to-one correspondences. However, reference constraints are kept only if they improve downstream code generation (Appendix C.2), which shows they are useful but not that they determine a unique answer. How do the authors verify that a table plus a constraint corresponds to a single correct chart or code?

  2. Source of the gains: The vision-language model’s reward has three main parts (Table 6): a weight of 0.7 on how closely the chart rendered from its own code matches the input chart; a weight of 0.05 on how similar its code is to the code written by the language model; a weight of 0.05 on how similar the two rendered charts are. Since the method’s novelty lies in the last two terms, how does performance change when they are removed, leaving only the comparison with the input chart?

  3. Test-time comparison: CoCoEvolve@Test selects the best of four samples (Eq. 9), while baselines report the average of four (Table 1). Moreover, the image metrics in the reward (Eq. 5) also appear in the evaluation (Appendix F.1.1). How does CoCoEvolve@Test compare with a best-of-four baseline that uses a selector independent of the evaluation metrics, such as code executability?

  4. Experimental setup: Section 4.1 lists three vision-language and six language models but does not state how they are paired. Which models were trained together for each row in Tables 1–2?

  5. Comparison with existing chart-to-code systems: The baselines in Tables 1–2 are only untuned base models. Comparisons with established systems would show whether CoCoEvolve’s gains hold against prior work:
    (1) ChartCoder (Zhao et al., ACL 2025), whose data CoCoEvolve already uses for training; (2) ChartMaster (Tan et al., arXiv:2508.17608, 2025), which also trains chart-to-code models with GRPO and a reward based on similarity between the rendered and input charts. This closely resembles the dominant part of CoCoEvolve’s reward.

6 comments
AuthorsOpenReview ↗

Author Response (1/2): Thank you for your valuable review and insightful comments

Thank you for your careful review and constructive feedback, and for recognizing our verifiable reward that needs no annotation and the breadth of our improvements across models, RL algorithms, and benchmarks. We hope to further address your remaining concerns below.

W1: Title and scope

Thank you for kindly noting this. Yes, our abstract introduces CoCoEvolve "to improve consistency across chart, table, and code representations", and §3.1 defines cross-representation understanding as six tasks among chart images, tables, and code. We make no empirical claim beyond these three representations: the design needs only representations that can be checked against each other automatically, such as code that executes into a chart. We are also happy to make our experiment scope explicit in the title, for example "Consistency-Driven Co-Evolution for Self-Supervised Chart-Table-Code Cross-Representation Learning".

W2+Q1: Constraint descriptions

Thank you for your insightful question. We hope to respectfully show how the constraint narrows the chart.

1. What one-to-one means (Q1 "a single correct chart or code"). The table fixes the data, and the constraint fixes what the table leaves open (chart type, visual encodings, colors, labels, and layout). So the one-to-one correspondence is between a table+constraint and a chart. Code is not unique, since different programs can draw the same chart. Thus, our rewards compare drawn charts and code embeddings and never require one exact program.

2. Direct measurement (Q1 "How do the authors verify", W2). Beyond the Stage 3 check (§C.2), we add a direct measurement: on in-domain test charts, base Qwen3-4B writes four codes from the reference table alone and four from the table+constraint, scored by the rule-based chart score. With the constraint, the drawn charts match the original chart better (+11.58, 95% CI [+10.46, +12.72]) and agree more with each other (+2.75, 95% CI [+1.13, +4.39]):

InputMatch to the original chartAgreement among the drawn charts
Table only70.6989.27
Table + constraint82.6492.32

W2+Q2: Source of the gains

Thank you for your insightful question. We add this ablation below.

1. Removing the agreement terms (Q2 "how does performance change when they are removed"). We train and compare both models using the same reward equation (Eq. 7), only different in with or without the agreement terms. Without them, execution and chart scores still rise over the base model, but code quality barely changes. With them, every score of both models is higher:

ModelMethodExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
MθM_\theta = Qwen3-VL-4BBase model78.4654.6841.0063.8738.58
MθM_\theta = Qwen3-VL-4BAgreement terms removed86.3554.4140.1067.5939.38
MθM_\theta = Qwen3-VL-4BCoCoEvolve@Train92.5756.4942.8170.6641.91
MψM_\psi = Qwen3-4BBase model77.6350.8639.5761.7934.89
MψM_\psi = Qwen3-4BAgreement terms removed81.2752.6937.3068.0137.64
MψM_\psi = Qwen3-4BCoCoEvolve@Train95.2556.8741.3572.3841.47

2. Small weights (Q2 "a weight of 0.05"). A small weight does not mean a small effect: GRPO normalizes rewards within each group of four candidates, so when candidates score similarly on the input-chart term, the agreement terms decide their ranking.

3. Novelty (W2 "the novelty of the contribution is unclear"). The novelty is co-evolution as a whole: the two models are coupled through the agreement terms and through sampling, as MψM_\psi learns from an evolving MθM_\theta (Eq. 8) and MθM_\theta needs to look at the input charts more carefully to provide accurate tables and constraints to better guide the evolving MψM_\psi, so both learn from unlabeled charts alone. Training MψM_\psi on its own would instead need reference tables and constraints as inputs (the "Baseline" in Tab. 5).

W3+Q3: Test-time comparison and evaluation

Thank you for your thoughtful questions. We hope to respectfully show each part below.

1. Training gains (W3 "compared unequally"). The @Train rows compare one sampled output with one sampled output (m@4 on both sides), so the training gains involve no test-time selection.

2. Best-of-four baseline (Q3 "a best-of-four baseline"). Each @Test run samples 4 MθM_\theta and 4 MψM_\psi candidates and selects with the annotation-free reward (Eq. 9), never with a reference or a judge, and m@4 averages four runs. We add your suggested best-of-four baseline on the same test set, with the same number of samples. CoCoEvolve@Test is higher on all five scores of both models, including the two judge scores that no selector sees:

(Very sorry there is not enough space left, so we respectfully put the table in the next Author Response 2/2. Many thanks!)

AuthorsOpenReview ↗

Author Response (2/2): Thank you for your valuable review and insightful comments

ModelMethodExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
MθM_\theta = Qwen3-VL-4BBest-of-four by executability90.3655.0448.3776.4160.52
MθM_\theta = Qwen3-VL-4BCoCoEvolve@Test93.5458.7351.7877.9262.96
MψM_\psi = Qwen3-4BBest-of-four by executability93.0451.0650.4074.0560.69
MψM_\psi = Qwen3-4BCoCoEvolve@Test99.0857.3855.8183.4571.04

3. Metric overlap (W3 "the training rewards overlap with the evaluation metrics", Q3). The rule-based chart score shares the reward's image metrics by design, since comparing the drawn chart with the input chart needs no annotation. In our main results, the reward never uses reference code, reference tables, or judges, and the scores it does not compute also improve, including: the LLM-as-judge code and MLLM-as-judge chart scores (tables above), the rule-based code score against the reference code (§F.1.2), and the table scores (Fig. 3).

4. LLM-generated references (W3 "LLM-generated without human verification"). Code and chart outputs (Tabs. 1-2, Fig. 9) are scored against each benchmark's original ground-truth code and charts, not against LLM-generated references. Gemini-3.1-Pro references serve only to score table and constraint outputs (Fig. 3), since ChartCoder provides no ground-truth tables and constraints are new in this work (§3.1), and each passed the quality check of Stage 3 (§C.2). On ChartNet, whose test set has its own human-verified tables, CoCoEvolve also raises MθM_\theta's table scores in every setting (Fig. 9), e.g., rule-as-judge table from 40.97 to 47.42 (@Train+@Test).

Q4: Model pairing

Thank you for kindly noting this. To keep the rows comparable, every MθM_\theta row (Qwen3-VL-2B, Qwen3-VL-4B, InternVL3.5-4B) is paired with MψM_\psi = Qwen3-4B, and every MψM_\psi row (Llama-3.2-3B, DeepSeek-Coder-1.3B and 6.7B, Qwen3-1.7B, 4B, and 8B) with MθM_\theta = Qwen3-VL-4B, in every block (@Train, @Test, @Train+@Test). Each row thus varies one model while its partner stays fixed. We have added this to §4.1.

Q5: ChartCoder and ChartMaster

Thank you for your kind suggestion. We add both comparisons below.

1. Comparison (Q5 "whether CoCoEvolve's gains hold against prior work"). ChartCoder and ChartMaster are trained for chart-to-code only, on 160K and about 240K chart-code pairs, respectively, while CoCoEvolve co-evolves two models for all six tasks from 10,298 unlabeled charts. For a fair comparison, we train CoCoEvolve on the same base model (Qwen2.5-VL-7B) as ChartMaster-7B, and at the same 7B scale as ChartCoder. All three are evaluated on the same out-of-domain chart-to-code sets (Tab. 3), with the same instruction, decoding (four samples, m@4), and judge. CoCoEvolve@Train-7B is the highest on every score:

DataSystemExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
ChartMimicChartCoder-7B79.3852.9647.8663.7845.65
ChartMimicChartMaster-7B87.3652.1449.2370.6856.23
ChartMimicCoCoEvolve@Train-7B (ours)89.6754.2653.1773.7360.71
Chart2CodeChartCoder-7B82.2049.5233.4764.2841.35
Chart2CodeChartMaster-7B88.8450.3836.7268.7250.79
Chart2CodeCoCoEvolve@Train-7B (ours)92.0752.4341.1274.7856.91
ChartNetChartCoder-7B78.0442.7943.5167.3449.22
ChartNetChartMaster-7B87.1843.5245.6875.8361.86
ChartNetCoCoEvolve@Train-7B (ours)89.8545.6149.4279.1168.31

2. ChartMaster's reward (Q5 "closely resembles the dominant part of CoCoEvolve's reward"). ChartMaster's reward compares only the drawn chart with the input chart. CoCoEvolve adds agreement between two co-evolving models across six tasks without annotations: MψM_\psi learns from an evolving MθM_\theta, which in return pushes MθM_\theta to read input charts more accurately. The second table above (W2+Q2) is the controlled comparison: with the same models, data, and budget, adding the agreement terms raises every score.

3. ChartCoder's data (Q5 "whose data CoCoEvolve already uses for training"). CoCoEvolve uses only the chart images of 10,298 ChartCoder samples, not the full 160K data that their ChartCoder is trained on. To validate cleanly, we compare on the same out-of-domain sets (table above).

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

AuthorsOpenReview ↗

Author Response: Thank you for your valuable review and insightful comments

Thank you very much for your valuable review and thoughtful comments. We sincerely hope our responses can meaningfully address your remaining concerns. As the rebuttal period is approaching its deadline, we would be more than happy to provide any further clarifications or address any remaining questions you may have. We would greatly appreciate it if you could kindly take our responses into consideration in your final assessment of our paper. Thank you again for your valuable time and consideration.

Reviewer_vHcFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoCoEvolve, a consistency-driven co-evolution framework for self-supervised cross-representation learning across chart images, tables, and visualization code. The authors introduce descriptive constraints to address the inherent one-to-many ambiguity in cross-representation mappings. During training, CoCoEvolve jointly optimizes models with rewards based on code executability, code similarity, and visual consistency. The framework also extends to test-time optimization by selecting outputs based on consistency rewards. Experiments across multiple benchmarks demonstrate improvements in cross-representation understanding and generalization across different tasks and domains.

Critical strengths

This paper demonstrates that one-to-one mapping is good for consistency-driven optimization between chart, table, and visualization code.

Critical weaknesses

The experiments conducted do not quite support the statement in the paper.

  1. The authors do not state how they apply their proposed mixed rewards to optimization algorithms in the main experiment.
  2. The authors do not compare their proposed methods with other baseline methods like SFT or GRPO, or any other models that are specifically trained for this task. The authors only compare with the vanilla model before applying the proposed method.
  3. The reward design is complicated, yet there are no ablation studies to show the effectiveness of each part in the mixed reward.
Questions for authors

The notation is overcomplicated and makes the methodology part a bit hard to understand. The authors should try to simplfy the notation.

5 comments
AuthorsOpenReview ↗

Author Response (1/2): Thank you for your valuable review and insightful comments

Thank you for your careful review and constructive feedback, and for recognizing our consistency-driven optimization between charts, tables, and visualization code. We hope to further address your remaining concerns below.

W1: How the reward is applied in the main experiments

Thank you for kindly pointing this out. We hope to respectfully show how our reward enters the optimization (§D).

1. Algorithm (W1 "in the main experiment"). All main experiments (Tabs. 1-2, Fig. 3) train with GRPO. The same reward (Eq. 7) also drives DAPO and GSPO in Fig. 4 (§4.3), where every metric improves over the base model under all three algorithms.

2. From reward to update (W1 "how they apply their proposed mixed rewards"). We apply the reward in three steps (§D):

  • Reward: every rollout receives the scalar reward of Eq. 7 (RθR_\theta for an MθM_\theta rollout, RψR_\psi for an MψM_\psi rollout), computed from the input chart and both models' outputs.
  • Advantage: an MθM_\theta rollout's reward is averaged over its Kψ=4K_\psi = 4 MψM_\psi children (Eq. 17) and normalized against the Kθ=4K_\theta = 4 MθM_\theta rollouts of the same chart (Eq. 18). An MψM_\psi rollout is normalized against the Kψ=4K_\psi = 4 siblings that share its MθM_\theta parent (Eq. 19).
  • Update: these advantages enter GRPO's clipped objective with a KL penalty for each model (Eqs. 15-16), and the co-evolution objective adds the two models' terms (Eq. 8).

To make this clearer, we will also state this in §4.1 and point to §D for details.

W2: Baselines: SFT, GRPO, and models trained for this task

Thank you for your kind suggestion. We respectfully clarify that our comparisons go beyond the vanilla model: our paper already compares with SFT, and we further add a GRPO baseline and two chart-specific systems below.

1. SFT (W2 "SFT"). Our paper compares with SFT twice. In Fig. 7 and §B.2, SFT trains on the same 10,298 charts as CoCoEvolve, with the reference code and the Gemini-3.1-Pro reference tables and constraints as targets, and is evaluated on the same test set as Tabs. 1-2. It lowers all five scores of both base models, even though it learns from references that CoCoEvolve never sees, while CoCoEvolve raises all of them (shown in the table below). SFT on 30,000 ChartNet instances (Fig. 6, §B.1) also lowers most scores.

2. GRPO (W2 "GRPO"). GRPO is the optimizer that CoCoEvolve runs on, not an alternative to it (W1). A plain GRPO baseline that trains each model on its own would need annotations: MψM_\psi writes code from a table and a constraint, so without MθM_\theta it has to read the Gemini-3.1-Pro reference tables and constraints (the "Baseline" in Tab. 5, which also takes 86 h per run vs. 77 h for one CoCoEvolve run that trains both models). We summarize the GRPO baseline results in the table below, with only the two agreement terms between the models removed, other training configurations staying the same.

3. Beyond the vanilla model (W2 "only compare with the vanilla model"). We summarize SFT and GRPO baseline results in the table below, next to the base model and CoCoEvolve@Train, with the same base models, training charts, and test set as Tabs. 1-2 (m@4 over four sampled outputs). CoCoEvolve@Train is the highest on every score for both models, and SFT the lowest. The GRPO baseline raises execution and both chart scores but barely changes code quality:

ModelMethodExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
MθM_\theta = Qwen3-VL-4BBase model78.4654.6841.0063.8738.58
MθM_\theta = Qwen3-VL-4BSFT53.2744.9626.4242.8521.51
MθM_\theta = Qwen3-VL-4BGRPO, agreement terms removed86.3554.4140.1067.5939.38
MθM_\theta = Qwen3-VL-4BCoCoEvolve@Train (GRPO)92.5756.4942.8170.6641.91
MψM_\psi = Qwen3-4BBase model77.6350.8639.5761.7934.89
MψM_\psi = Qwen3-4BSFT45.8538.9119.9536.6913.91
MψM_\psi = Qwen3-4BGRPO, agreement terms removed81.2752.6937.3068.0137.64
MψM_\psi = Qwen3-4BCoCoEvolve@Train (GRPO)95.2556.8741.3572.3841.47
AuthorsOpenReview ↗

Author Response (2/2): Thank you for your valuable review and insightful comments

4. Chart-specific models (W2 "any other models that are specifically trained for this task"). We compare with ChartCoder and ChartMaster that are specifically trained for the chart-to-code task on 160K and about 240K chart-code pairs, respectively. CoCoEvolve instead co-evolves two models for all six cross-representation tasks from 10,298 unlabeled charts. For a fair comparison, we trained CoCoEvolve on the same base model (Qwen2.5-VL-7B) as ChartMaster-7B, and at the same 7B scale as ChartCoder. All three are evaluated on the same out-of-domain chart-to-code sets (Tab. 3), with the same charts, instruction, decoding (four samples, m@4) and judge. CoCoEvolve@Train-7B is the highest on every score of all three sets:

DataSystemExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
ChartMimicChartCoder-7B79.3852.9647.8663.7845.65
ChartMimicChartMaster-7B87.3652.1449.2370.6856.23
ChartMimicCoCoEvolve@Train-7B (ours)89.6754.2653.1773.7360.71
Chart2CodeChartCoder-7B82.2049.5233.4764.2841.35
Chart2CodeChartMaster-7B88.8450.3836.7268.7250.79
Chart2CodeCoCoEvolve@Train-7B (ours)92.0752.4341.1274.7856.91
ChartNetChartCoder-7B78.0442.7943.5167.3449.22
ChartNetChartMaster-7B87.1843.5245.6875.8361.86
ChartNetCoCoEvolve@Train-7B (ours)89.8545.6149.4279.1168.31

W3: Reward ablations

Thank you for your valuable insight. We hope to respectfully show our existing reward ablations, and further add the ablation of agreement terms below.

1. Existing reward ablations (W3 "no ablation studies"). Fig. 11 (§E.1) ablates the reward weights on the main pair (MθM_\theta = Qwen3-VL-4B, MψM_\psi = Qwen3-4B): our asymmetric weights (Tab. 6) outperform symmetric weights on execution and every code score of both models. With these weights, all nine models in Tabs. 1-2 improve over their base models on execution and m@4 scores across settings. Fig. 5 also ablates the teacher module's grounding rewards.

2. Agreement terms (W3 "effectiveness of each part"). The agreement terms effectively helps: removing agreement rewards lowers every score of both models (the GRPO baseline in W2, table below):

ModelAgreement termsExecutionRule-as-judge codeLLM-as-judge codeRule-as-judge chartMLLM-as-judge chart
MθM_\theta = Qwen3-VL-4BWith (CoCoEvolve@Train)92.5756.4942.8170.6641.91
MθM_\theta = Qwen3-VL-4BWithout86.3554.4140.1067.5939.38
MψM_\psi = Qwen3-4BWith (CoCoEvolve@Train)95.2556.8741.3572.3841.47
MψM_\psi = Qwen3-4BWithout81.2752.6937.3068.0137.64

To summarize our reward ablations (W3):

Reward part (Eq. 7)AblationEvidence
Agreement between the modelsremoved for both models, everything else unchangedtable above
Balance between input-chart grounding and agreementasymmetric (main) vs. symmetric weights for the two modelsFig. 11
Teacher grounding rewards (teacher-module ablation only)on vs. offFig. 5

Q1: Notation

Thank you for kindly noting this. To make §3 easier to follow, we will simplify the notation and improve the notation table (Tab. 4) so that each symbol is clearly defined. CoCoEvolve is an efficient and effective training framework that self-supervises without external labels and co-evolves two models across six cross-representation tasks at once. To present it clearly, we will maintain separate symbols for the two models and their per-model weights.

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

AuthorsOpenReview ↗

Author Response: Thank you for your valuable review and insightful comments

Thank you very much for your valuable review and thoughtful comments. We sincerely hope our responses can meaningfully address your remaining concerns. As the rebuttal period is approaching its deadline, we would be more than happy to provide any further clarifications or address any remaining questions you may have. We would greatly appreciate it if you could kindly take our responses into consideration in your final assessment of our paper. Thank you again for your valuable time and consideration.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Author Response (1/2): Thank you for your valuable review and insightful comments

Thank you for your careful review and constructive feedback, and for recognizing our thorough experiments and the design of our method. We hope to further address your remaining concerns below.

W1+Q2: "Self-supervised" and "annotation-free"

Thank you for kindly noting this. We hope to respectfully address each point in detail.

1. What they describe (W1, Q2 "applicable boundary"). In CoCoEvolve, "self-supervised" and "annotation-free" describe the learning signal. In every CoCoEvolve result except the teacher-module ablation (Fig. 5), the training reward (Eq. 7) and the test-time selection (Eq. 9) are computed only from the input chart and the models' own outputs, through sandbox execution and rubric metrics (§3.2). As we highlighted in table captions, all other experiments are in student mode, with the teacher module disabled. Therefore, no reference table, constraint, or code, and no LLM judge is read during training or test-time selection, and training makes no API calls (Tab. 5, §E.2).

2. Teacher module (W1 "the Teacher module in §3.4 uses the ground-truth", Q2). The teacher module is enabled only in the teacher-module ablation (Fig. 5), which shows that CoCoEvolve can also smoothly accommodate supervised learning when high-quality annotations are available (§3.4). In this ablation, the "ground truth" (sgt,tgt)(s_{gt}, t_{gt}) of §3.4 is the Gemini-3.1-Pro reference tables and constraints, used as stand-in annotations. All our other experiments and ablation studies are self-supervised and use student mode, with the teacher module disabled. We also highlight this in our paper: Tab. 6 separates "Main Training (M\mathcal{M} Disabled)", with α=0\alpha = 0, from "Ablation Study: Teacher Module (M\mathcal{M} Enabled)", and the captions of Tabs. 1-2 and Fig. 3 state that "CoCoEvolve-enhanced models are optimized at train time and/or test time in student mode". To better highlight that the teacher module is an ablation study, not our main experiments, we moved the figure of this ablation to the appendix (Fig. 5, was Fig. 3).

3. Reference tables and constraints (W1 "§C.2 generates the training reference tables/constraints", Q2). The reference tables and constraints are used only for the teacher-module ablation, evaluation and data filtering (§C.2). Within the method, only the teacher module reads them, so in training and test-time selection they appear only in the teacher-module ablation (Fig. 5). In evaluation, they are the scoring references for table and constraint outputs (Fig. 3), since ChartCoder provides no ground-truth tables and constraints (§3.1). Code and chart outputs (Tabs. 1-2) are scored against each benchmark's original ground truths. During data construction, Stage 3 of §C.2 also uses these references for data filtering, in the same way for the training and test sets. This filter only performs quality control during dataset construction, and never provides a training target or reward.

4. Judges (W1 "§4.1 uses GPT-5-mini and Gemini-3-Pro as judges", Q2). The two judges are used only for evaluation (§4.1), never in training or test-time optimization. Every main result (Tabs. 1-2, Fig. 3) also includes execution success and rule-based metrics (§F.1), which use no judge and improve consistently with the judge-based columns. To validate the results, these two judges, from different providers, also agree closely (Fig. 12: Pearson r = 0.830 and 0.841, ICC = 0.794 and 0.808, weighted κ = 0.769 and 0.782 for the base models and CoCoEvolve).

5. Main setting (Q2 "How well does the method perform in a purely self-supervised setting"). Yes, the purely self-supervised setting you describe is exactly our main setting. The MθM_\theta rows of Tabs. 1-2 use none of the three resources listed in Q2 (no generated reference tables/constraints, no ground truth in the teacher module, no proprietary judges) in training, test-time selection or their execution and rule-based scores. The input is only the original chart, and the self-supervised learning signal (Eq. 7) is computed only from this chart and the models' own outputs: whether the two models' outputs are consistent with each other and with the original chart. Scored without any LLM or MLLM judge, Qwen3-VL-4B improves from 78.46 to 97.23 in execution success, from 54.68 to 60.02 in the rule-based code score, and from 63.87 to 81.21 in the rule-based chart score (base model vs. CoCoEvolve@Train+@Test, Tabs. 1-2).

AuthorsOpenReview ↗

Author Response (2/2): Thank you for your valuable review and insightful comments

6. Boundary of "annotation-free" (Q2 "applicable boundary"). Applying CoCoEvolve to new charts needs only the input chart images. References are needed only for the optional teacher module and for evaluation, and judges only for evaluation:

ResourceWhere it is usedMain training reward@Test selection
Unlabeled input charttarget of the rendered-chart rewardyesyes
Code sandbox, rubric metricscomputing the rewards (Eqs. 3 to 7)yesyes
Reference tables and constraintsteacher-module ablation; scoring references for table and constraint outputs; Stage 3 filternono
Teacher moduleteacher-module ablation only (α=0\alpha = 0 elsewhere, Tab. 6)nono
LLM/MLLM judgesLLM- and MLLM-as-judge scoresnono

7. Abstract (W1). To further avoid any confusion, we will revise your quoted abstract sentence to "optimize models using agreement between representations as an annotation-free signal at both training and test time", to state the scope of "annotation-free" more precisely.

W2+Q1: m@4 in the @Test rows

We sincerely appreciate this opportunity to further clarify.

1. How m@4 is computed (Q1 "clarify the specific computation of m@4 in the @Test row"). For @Test rows, each rollout is one complete CoCoEvolve@Test run (Eq. 9), so m@4 averages four reward-selected outputs. In every row, m@4 =14∑r=14s(yr)= \frac{1}{4}\sum_{r=1}^{4} s(y_r), where ss is the metric and yry_r is the output of run rr: one sampled output in the base-model and @Train rows, and the output selected by one complete @Test run in the @Test and @Train+@Test rows. For example, for one chart in a base-model row, four sampled codes score 50, 55, 60 and 65, so m@4 = 57.5. While in an @Test row, four @Test runs return selected codes that score 62, 58, 65 and 60, so m@4 = 61.25.

2. One @Test run (W2 "selecting the best by reward", Q1). "Produces the best outputs" is for the test-time optimization of CoCoEvolve@Test (§3.3), while "mean score over four rollouts" (Tab. 1) is for evaluating model performance after inference is done, so the two describe different phases and do not conflict. At inference, one @Test run selects the best outputs without any reference/ground truth/judge: MθM_\theta samples Kθ=4K_\theta = 4 candidates and keeps the one with the highest reward RθR_\theta, then MψM_\psi samples Kψ=4K_\psi = 4 candidates conditioned on it and keeps the one with the highest RψR_\psi (Eq. 9). RθR_\theta and RψR_\psi are computed only from the input chart and the models' outputs (§3.3), so this phase uses no reference, ground truth, or judge. After inference is done, the evaluation scores each run's output against the references and ground truths and with the judges, and m@4 averages these scores over four runs (as in #1 above).

3. Generations per run (Q1). Each @Test run uses the same Kθ=Kψ=4K_\theta = K_\psi = 4 as training (§4.1, Tab. 6): 4 MθM_\theta generations and 4 MψM_\psi generations, since MψM_\psi is sampled only for the selected MθM_\theta output. This is a stage-wise selection, not an exhaustive search over all 4×44 \times 4 pairs of MθM_\theta and MψM_\psi outputs, which would need 4 + 16 generations.

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

Thank you for the detailed explanations. My main concerns have been resolved, and I have updated my score accordingly.

I have two suggestions for the authors. Firstly, please highlight the changes from the original submission in a different color in the revised manuscript. Secondly, please add the remaining details clarified in your response to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.