acceptodds
Under review as a conference paper at ICLR 2027

Babel: Visual Language Emergence

Abstract

Can a visual language emerge from scratch rather than being designed in advance? Visual representations today are built on fixed, human-designed languages, from SVG to plotting libraries to domain-specific languages. The choice of language determines the available abstractions and the size of the search space, yet the language itself is rarely questioned. We introduce Babel, a bi-level framework in which an outer loop evolves primitive operations from expressiveness gaps while an inner loop generates visualizations with the current language. Starting from only four geometric primitives, Babel proposes, validates, and refines primitives over a scheduled stream of tasks. As gaps close, valid updates decline and growth saturates only when one registry is shared across tasks. Crucially, what emerges are mechanisms rather than policies: primitives compress repeated execution without taking task-specific decisions away from the caller. On a benchmark spanning six scientific visualization domains, the emerged language expresses the same figures in fewer calls than its seed basis and fewer than a hand-crafted DSL, at quality indistinguishable from both and from free-form Python. Its primitives also transfer across domains at no cost in quality, suggesting that what emerges is a general drawing language rather than domain-specific shortcuts. These results show that the representation space itself can be optimized during task solving, with reusable mechanism-level abstractions emerging from the task distribution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Model, Agent, Self-evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6zU6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Babel evolves a registry of visual primitives through alternating task solving and primitive proposal. Starting from four geometric operations, a VLM generates visual programs, inspects task-level evidence, and proposes reusable drawing functions. The paper evaluates program compactness, rendering quality, registry-sharing strategies, and transfer on a benchmark covering six scientific domains.

Critical strengths

Treating the drawing interface as something that can adapt to a task distribution is an interesting direction. The experiments demonstrate substantial reductions in primitive-call counts while retaining high judged quality.

The paper provides extensive supplementary analyses, including two VLM judges, clustered confidence intervals, external drawing-language comparisons, and negative results. These efforts help identify the limits of the reported findings.

Critical weaknesses
  1. The contribution beyond existing library learning and tool synthesis is insufficiently established. Many emerged primitives accept lists of explicitly specified geometric records and batch repeated drawing operations. This is useful, but fewer calls alone do not establish a substantially better representation or synthesis process. Comparisons with simple batching, macro extraction, or persistent tool-library construction are needed to identify what the proposed visual-feedback-driven search adds. The paper should also report program tokens, latency, and total cost, including registry evolution, to establish whether call-count compression translates into practical efficiency.

  2. The experimental presentation makes the central conclusion difficult to assess. The headline results use isolated per-subtopic registries, whereas the method and narrative emphasize a single shared language. Quality improvements over Seed-4 and Python are modest relative to the reported uncertainty, and the external TikZ comparison does not establish a clear compactness advantage. Existing ablations examine sharing, budget, and proposer constraints, but do not directly isolate the value of rendered visual feedback. The paper needs a clearer separation of configurations and stronger controls for its central mechanism.

Additional Presentation Concerns

  1. The writing relies heavily on broad terms such as “emergence,” “mechanisms rather than policies,” and “language,” without making their distinction from reusable function libraries sufficiently clear. A concrete example tracing a task, diagnosed gap, proposed primitive, validation, and subsequent reuse would make the contribution easier to understand.

  2. Figure 4 should be redrawn with larger text, fewer decorative icons, and clearer data-flow arrows and intermediate artifacts.

Questions for authors

How does Babel compare with a simple baseline that adds list-valued batching wrappers or extracts reusable macros from generated programs? What benefit specifically requires rendered visual feedback?

Can the main results consistently foreground the shared-registry configuration and explain how its conclusions differ from those of the isolated per-subtopic registries?

Do fewer primitive calls reduce generated tokens, latency, or API cost after accounting for primitive arguments, registry serialization, and the outer-loop evolution cost?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6zU6 (Part 1 / 3)

[Part 1/3]

We thank the reviewer for their time and detailed comments. The main question is what Babel's search adds beyond batching and existing library learning.

A few names used below: Seed-4 is the 4-primitive starting language (points, lines, circles, text). Batched Seed-4 is a new baseline that adds list versions of these 4 primitives, so many calls can be merged into one but nothing new can be drawn. Per-subtopic means one registry per subtopic (Table 2); shared means one registry for all tasks. Judge 1 / Judge 2 are two independent judges; brackets are 95% confidence intervals.


Weakness 1 / Question 1. "Comparisons with simple batching, macro extraction, or persistent tool-library construction are needed to identify what the proposed visual-feedback-driven search adds." / "How does Babel compare with a simple baseline that adds list-valued batching wrappers or extracts reusable macros from generated programs? What benefit specifically requires rendered visual feedback?"

Batching, macros and tool libraries. All baselines use the same seed language, tasks, checker and registry format, and are tested on all 1,815 tasks:

MethodPass rate (Judge 1 / Judge 2)Values the model writesProgram tokensCompared with Babel
Batched Seed-498.29 / 96.697654,264same quality; 2.0–2.4× more values, 1.65–1.88× more tokens
Macro extraction97.63 / 96.256825,692same quality; 2.2–2.5× more tokens
LILO-style (Stitch + AutoDoc)97.52 / 96.146985,422same quality; 2.1–2.4× more tokens
TroVE-style (growing toolbox, with pruning)78.90 / 77.913372,18417.91 [12.73, 23.42] points lower (Judge 1, vs. shared)
Babel (per-subtopic / shared)98.24 / 97.25; 96.80 / 96.36323 / 3772,267 / 2,576—

Babel is the only method that is both accurate and short. Batching, macros and LILO can only package existing primitives, so their programs stay long. The TroVE-style toolbox is short but loses almost 18 points.

What Babel adds is new drawing abilities. 8 of the 10 shared primitives draw things Seed-4 cannot: filled regions, curves from points or formulas, ellipses, box plots, and node-link diagrams. 87.7% of Babel programs use them. When we restart evolution from Batched Seed-4, it finds 7 more such primitives, which cut written values by 2.13× [1.91, 2.38] and program tokens by 2.00× [1.82, 2.19] at the same quality. So the search finds more than batching does.

What needs feedback. We changed only what the proposer sees, keeping the budget, rules, checker and task order fixed:

What the proposer seesPass rate (Judge 1 / Judge 2)Difference from Babel (Judge 1, points)
Task descriptions only, whole library at once93.88 / 92.18−4.85 [−8.48, −1.76]
Task descriptions only, one task at a time93.22 / 90.85−5.51 [−9.37, −2.20]
Programs written for the tasks, no render result98.46 / 97.41−0.28 [−0.94, 0.39]
Programs + text execution log (3 runs)98.55 / 96.99−0.06 [−0.68, 0.55]
Babel: programs + render result (3 runs)98.60 / 97.69—
Babel + the rendered image (3 runs)98.92 / 97.80+0.31 [−0.15, 0.83]
  • Feedback is needed. Designing from descriptions alone loses 4.85–5.51 points (5.90–7.22 under Judge 2).
  • The key is seeing how the model actually attempted the tasks. Seeing the programs alone gives +5.23 [1.87, 9.15] points over description-only design. Adding a text log or the rendered image on top makes no measurable difference. So the benefit comes from attempted tasks, and a text-only proposer is enough, which keeps evolution cheap.

We will describe the feedback in §3 and B.4 in these terms and add the full table.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 6zU6 (Part 2 / 3)

[Part 2/3]

Weakness 1 (cont.) / Question 3. "The paper should also report program tokens, latency, and total cost, including registry evolution ..." / "Do fewer primitive calls reduce generated tokens, latency, or API cost after accounting for primitive arguments, registry serialization, and the outer-loop evolution cost?"

Yes. Program tokens drop 2.25–2.57×. Cost per task drops about 10% on our main model, which spends most of its tokens on reasoning, and about 30% on models that reason less, where runs are also about 1.7× faster. All numbers include the registry description, and the cost of evolution is paid back after about 1,530 tasks.

Why calls go down. "Calls" is the number of primitive calls in the program for one figure: about 117 for Seed-4 and 28–31 for Babel. There are two reasons:

  • Merged calls. Drawing 100 points becomes one list call, but all 100 coordinates are still written.
  • New primitives. A smooth curve becomes one formula instead of hundreds of short segments, so less is written.

Only the second reason saves tokens and money, as the batching numbers below show.

Main model, compared with Seed-4, 1,815 tasks:

MeasurePer-subtopicShared
Program tokens2.57× fewer2.25× fewer
Total tokens, measured (input + reasoning + output)1.19× fewer1.09× fewer
Registry description in the prompt1,398 tokens (average)2,177 tokens (about $0.002 per task)
Cost per task, measured1.10× lower ($0.203 → $0.185)1.17× lower
Time per task, measured1.15× lower1.11× lower
One-time evolution costabout $770about $45
Tasks needed to pay it backabout 43,000about 1,530

This model writes about 17–18k reasoning tokens per task in every language, so the saving is smaller. With models that reason less, shorter programs save directly (a ratio above 1 means Babel is cheaper or faster):

ModelCost, Seed-4 / BabelTime, Seed-4 / BabelPass rate, Seed-4 → Babel (Judge 1)
claude-sonnet-4-61.44 [1.26, 1.64]1.70 [1.39, 2.08]51.0 → 72.5
gemini-3.7-flash1.35 [1.28, 1.42]1.75 [1.54, 2.06]83.2 → 94.6
gpt-4o0.98 [0.84, 1.13]1.10 [0.94, 1.29]7.4 → 14.1
  • Batched Seed-4 saves nothing on claude-sonnet (0.97 [0.88, 1.05]) and 3% on the main model. Merging calls alone does not save money.
  • On gpt-4o, cost per task is the same, but Babel passes almost twice as often, so the cost per correct figure falls from $2.41 to $1.30.
  • With output length capped, shorter programs also fail less (pass rate %, Judge 1, 149 tasks; caps are percentiles of Seed-4's output length):
Output capSeed-4Batched Seed-4BabelBabel minus Seed-4 (points)
25th (17,238 tokens)24.224.841.6+17.45 [10.14, 25.00]
50th (24,963 tokens)60.460.477.2+16.78 [7.43, 26.62]
75th (32,040 tokens)86.686.694.6+8.05 [1.42, 15.60]

Weakness 2 / Question 2. "The headline results use isolated per-subtopic registries, whereas the method and narrative emphasize a single shared language." / "Can the main results consistently foreground the shared-registry configuration and explain how its conclusions differ from those of the isolated per-subtopic registries?"

Yes. We now lead with the shared registry.

  • Same budget (15 evolution tasks per subtopic). Shared and per-subtopic have the same quality (−1.43 [−2.92, 0.22] points, Judge 1), but the shared registry needs half as many calls (14.4 vs. 28.3 per figure). It is also 8.71 [5.51, 12.29] points above Human-28.
  • Stable. Three full reruns of the shared pipeline with different task orders vary by only 0.14 points (standard deviation).
  • What each shows. The per-subtopic registries are 121 small libraries (6.4 primitives on average) and show how far compression can go inside one subtopic. The shared registry is one language and supports the claims about reuse across fields and a vocabulary that stops growing. We now use each only for its own claims.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 6zU6 (Part 3 / 3)

[Part 3/3]

Weakness 2 (cont.). "Quality improvements over Seed-4 and Python are modest relative to the reported uncertainty, and the external TikZ comparison does not establish a clear compactness advantage."

Quality. All baselines were run with the same output limit and file handling (1,815 tasks; Babel per-subtopic: 98.24 / 97.25):

BaselinePass rate (Judge 1 / Judge 2)Babel minus baseline, Judge 1 (points)Babel minus baseline, Judge 2 (points)
Seed-497.25 / 95.59+0.99 [−0.66, 2.26]+1.65 [0.11, 3.09]
Python96.75 / 94.93+1.49 [−0.44, 3.14]+2.31 [0.33, 4.30]

Babel is at least as good as both under both judges, and clearly better under Judge 2. We agree that pass rate is not the main contribution. We present quality as "no worse than strong baselines", and the main contribution as shorter programs, fewer failures under output limits, and lower cost.

TikZ, on all 1,815 tasks, split by whether TikZ uses its pgfplots plotting library (pass rate %, Judge 1):

TasksCountTikZ passBabel passTikZ commands (loops expanded)Babel commands
Charts (TikZ uses pgfplots)47894.9898.9514.4839.88
All other tasks1,33797.8397.9842.0624.12
All tasks (Judge 1 / Judge 2)1,81597.08 / 96.7598.24 / 97.2534.828.3
  • Quality is the same: TikZ minus Babel is −1.17 [−2.61, 0.61] points (Judge 1).
  • TikZ's low command count comes from pgfplots, which draws axes and grids for free on the 26% of tasks that are standard charts; there TikZ passes about 4 points less often. On the other 74%, with loops expanded, TikZ needs 1.74× more commands than Babel.
  • TikZ/PGF was built by hand over decades. Babel reaches the same quality automatically from 4 primitives for about $45, and its programs are typed records that can be checked automatically and extended for new tasks.

Presentation 1. "The writing relies heavily on broad terms such as 'emergence,' 'mechanisms rather than policies,' and 'language' ... A concrete example tracing a task, diagnosed gap, proposed primitive, validation, and subsequent reuse would make the contribution easier to understand."

Definitions (added to §3):

  • Emergence: a primitive that is not in Seed-4, was proposed during evolution, and passed the checker.
  • Mechanisms rather than policies: a primitive makes no choice the caller cannot override, such as a fixed color or automatic layout. An automatic check (rules plus an LLM) agrees with human labels on 10 of 10 shared primitives and 282 of 294 per-subtopic primitives.
  • Language rather than library: one registry used across fields, whose vocabulary stops growing, and whose primitives are reused.

Worked example (new appendix): draw_filled_regions.

  1. In round 21, a UV-Vis task had to fill the area under a peak with many thin lines.
  2. The proposer noticed that filled regions could not be drawn.
  3. It proposed draw_filled_regions, which takes a list of regions.
  4. The checker accepted it. A later edit added an optional hatch field.
  5. It was then used in 1,165 test figures across 6 fields. On the 1,142 tasks that also have a Seed-4 program, it needs 3.90× fewer calls, 1.77× fewer written values and 2.29× fewer tokens.

Presentation 2. "Figure 4 should be redrawn with larger text, fewer decorative icons, and clearer data-flow arrows and intermediate artifacts."

Done. Figure 4 now uses larger text and no decorative icons. Arrows show each step and what it produces: task → program → render and check → missing ability found → new primitive → registry.


We thank the reviewer again. The new comparisons with batching, macros, LILO, TroVE and description-only design make the contribution much clearer: learning from attempted tasks finds new drawing abilities the alternatives cannot, giving shorter programs at the same quality, fewer failures under output limits, and lower cost. We would be grateful if the reviewer would consider raising the score.

Reviewer_LU7fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Babel studies how a drawing language can be adapted to a distribution of scientific visualization tasks. Starting from four geometric primitives, it alternates between generating programs under a frozen registry and proposing additions or modifications based on the resulting programs and renderings. The model weights remain fixed; adaptation takes place in the primitive interfaces and their executable implementations.

The evaluation compares the resulting libraries with the seed primitives, a hand-designed DSL, and existing drawing languages. The isolated configuration, which maintains separate registries for individual subtopics, reduces the average number of primitive calls from 114.9 to 28.3 while achieving a high VLM-judged pass rate. The paper also examines registry sharing, cross-domain transfer, and the effect of interface constraints that leave geometry and layout decisions to the caller.

Critical strengths

1. The distinction between abstraction and caller control is valuable.
The most interesting idea is that a primitive can encapsulate repeated drawing operations without taking over task-specific decisions. The node-link and filled-region examples illustrate this well: the caller specifies the geometry, while the primitive handles the drawing.

2. The evaluation separates library construction from test-time use.
The registry is frozen before held-out evaluation, with no primitives added, refined, or repaired in response to test tasks. This is an appropriate protocol for assessing whether the learned library remains useful beyond the instances used to construct it, rather than measuring the benefit of creating a new tool for each test case.

3. The results show substantial cross-task reuse and shorter call sequences.
In the shared setting, newly introduced primitives account for 48.6% of all primitive calls in the held-out evaluation. draw_filled_regions, introduced by a chemistry spectroscopy task, is used in 1,165 figures across all six domains, while draw_node_link_diagram is used in 587 figures. These usage statistics provide concrete evidence that the primitives capture recurring drawing structures rather than individual examples. The shared and isolated configurations also reduce the average command count to 31.2 and 28.3, respectively, compared with 114.9 for Seed-4.

Critical weaknesses

1. The compression result needs a batching control.
Nine of the ten shared primitives accept lists of per-element records. Under a call-count metric, moving a loop inside a primitive can substantially reduce the reported count while leaving most of the coordinates and attributes to be specified by the model. Batching is useful, but the current experiments do not establish how much Babel adds beyond this simple interface change. A Seed-4 baseline with batched calls or a generic map operation would be an important control. The character-count reduction in Appendix B.17 is encouraging, but the main benchmark should also report complete program length and the prompt overhead of describing the registry.

2. The contribution of feedback-driven discovery is not isolated.
The main comparisons—Seed-4, Human-28, Python, and Babel—show that the resulting libraries are useful, but do not establish that Babel is a better way to construct them. A useful control would ask the same model to design a library from the emergence-task descriptions, under the same interface constraints and a comparable budget, but without rendered feedback.

The distinction from prior library-learning methods also needs stronger evidence. LILO (Grand et al., 2024) builds reusable libraries through iterative synthesis, compression, and documentation, including for compositional graphics tasks. TroVE (Wang et al., 2024) studies training-free toolbox construction through tool reuse, expansion, and periodic pruning. Against this background, the paper needs to establish what its visual feedback and evolutionary loop contribute beyond existing approaches to library learning and tool induction.

3. Some headline claims are broader than the supporting results.
The strongest compression result comes from 121 subtopic-specific registries, rather than the single shared language emphasized in the paper. Moreover, the shared and isolated configurations used for the headline results have different emergence budgets. Both settings are informative, but their results should not be treated as interchangeable evidence for a shared language.

Similarly, the favorable cross-domain transfer average excludes Chemistry as a source. Its registry achieves an average pass rate of only 56.7% across the six target domains in Table 4. Although the authors discuss this failure, it substantially qualifies the abstract’s claim that transfer comes “at no cost in quality.” The results support successful transfer for most of the tested source libraries, not an unqualified claim of domain generality.

Questions for authors
  1. How much of the compression advantage remains against a batched Seed-4 baseline, when measuring complete program length rather than only primitive calls?

  2. How does Babel compare with library design from task descriptions alone under the same interface constraints and a comparable budget? For Table 8, can the effect of preserving caller control be separated from the accompanying changes toward per-element records?

  3. Can the headline shared and isolated configurations be compared with matched budgets and proposal opportunities? How much do their results vary across complete emergence runs and different task orders?

  4. See the cons above. I’d be willing to raise my score if some of these concerns are addressed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LU7f (Part 1 / 3)

[Part 1/3]

We thank the reviewer for their time, for the concrete suggestions, and for being open to raising the score. We ran every check the reviewer asked for.

A few names used below: Seed-4 is the 4-primitive starting language (points, lines, circles, text). Batched Seed-4 is a new baseline that adds list versions of these 4 primitives, so many calls can be merged into one but nothing new can be drawn. Per-subtopic means one registry per subtopic (Table 2); shared means one registry for all tasks. Judge 1 / Judge 2 are two independent judges; brackets are 95% confidence intervals.


Weakness 1 / Question 1. "A Seed-4 baseline with batched calls or a generic map operation would be an important control ... the main benchmark should also report complete program length and the prompt overhead of describing the registry." / "How much of the compression advantage remains against a batched Seed-4 baseline, when measuring complete program length rather than only primitive calls?"

We built Batched Seed-4 as suggested. Each list primitive takes records with the same fields as the original and draws them one by one with the original code; sampled outputs are pixel-identical. On all 1,815 tasks:

MeasureSeed-4 / Batched Seed-4Batched Seed-4 / Babel (per-subtopic)Batched Seed-4 / Babel (shared)
Calls6.31×0.67×0.61×
Values the model writes (coordinates, colors, labels)0.87×2.37× [2.15, 2.62]2.03× [1.82, 2.26]
Full program tokens1.37×1.88× [1.73, 2.05]1.65× [1.50, 1.82]

The reviewer is right that batching explains the drop in calls. But batching does not reduce what the model writes; it actually writes 15% more values. Measured by full program length, Babel programs are still 1.65–1.88× shorter than Batched Seed-4. We now use full program tokens and written values as the main measures.

To test directly what evolution adds beyond batching, we restarted evolution from Batched Seed-4. It found 7 new primitives (filled polygons, curves, ellipses, box plots, arrows, rounded boxes). Compared with Batched Seed-4, they cut written values by 2.13× [1.91, 2.38] and program tokens by 2.00× [1.82, 2.19], with the same pass rate (98.62 / 97.13). We did not build a separate map operation; starting from an already batched language is a harder test.

Registry prompt cost. The registry description is 731 tokens for Seed-4, 2,177 for the shared registry, and 1,398 on average per subtopic, about $0.002 per task. We add these to the main table and include them in all cost numbers.

Does it matter in practice? With output length capped, Batched Seed-4 is no better than Seed-4, but Babel is (pass rate %, Judge 1, 149 tasks):

Output cap (percentile of Seed-4 output length)Seed-4Batched Seed-4Babel
25th (17,238 tokens)24.224.841.6
50th (24,963 tokens)60.460.477.2
75th (32,040 tokens)86.686.694.6

Cost follows the same pattern. On claude-sonnet-4-6, Babel is 1.44× [1.26, 1.64] cheaper per task than Seed-4, while Batched Seed-4 is not cheaper at all (0.97× [0.88, 1.05]).

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer LU7f (Part 2 / 3)

[Part 2/3]

Weakness 2 (a) / Question 2 (a). "A useful control would ask the same model to design a library from the emergence-task descriptions, under the same interface constraints and a comparable budget, but without rendered feedback." / "How does Babel compare with library design from task descriptions alone ...?"

We ran this control. We also added steps between "description only" and Babel, changing only what the proposer sees. The model, budget, rules, checker and task order were the same throughout.

What the proposer seesPass rate (Judge 1 / Judge 2)Difference from Babel (Judge 1 / Judge 2, points)
Task descriptions only, whole library at once93.88 / 92.18−4.85 [−8.48, −1.76] / −5.90 [−9.97, −2.42]
Task descriptions only, one task at a time (same budget)93.22 / 90.85−5.51 [−9.37, −2.20] / −7.22 [−11.13, −3.80]
Programs written for the tasks, no render result98.46 / 97.41−0.28 [−0.94, 0.39] / −0.66 [−1.49, 0.17]
Programs + text execution log (3 runs)98.55 / 96.99−0.06 [−0.68, 0.55] / −0.70 [−1.41, 0.00]
Babel: programs + render result (3 runs)98.60 / 97.69—
Babel + the rendered image (3 runs)98.92 / 97.80+0.31 [−0.15, 0.83] / +0.11 [−0.46, 0.72]
  • Designing from descriptions alone is clearly worse, by 4.85–7.22 points, under both judges.
  • What matters is seeing how the model actually attempted the tasks. Seeing the programs alone gives +5.23 [1.87, 9.15] points over description-only design (Judge 2: +6.56 [3.09, 10.52]). Adding a text log or the image on top makes no measurable difference, so a text-only proposer is enough, which keeps evolution cheap.

We will describe the feedback in §3 and B.4 in these terms and add the full table.


Weakness 2 (b). "The distinction from prior library-learning methods also needs stronger evidence ... LILO ... TroVE ..."

We implemented LILO-style, TroVE-style and macro-extraction baselines. All use the same seed language, tasks, checker and registry format, and are tested on all 1,815 tasks:

MethodPass rate (Judge 1 / Judge 2)Program tokensDifference from Babel shared (Judge 1, points)Primitives that add a new drawing ability
TroVE-style (import / create / skip, with pruning)78.90 / 77.912,184−17.91 [−23.42, −12.73]—
LILO-style (Stitch compression + AutoDoc, 2 rounds)97.52 / 96.145,422+0.72 [−0.83, 2.31]0%
Macro extraction97.63 / 96.255,692about equal0%
Babel shared96.80 / 96.362,576—8 of 10; used in 87.7% of programs

LILO and macro extraction look for call patterns that repeat in programs already written, and package them. They can only combine existing primitives, so they cannot express things like filled regions or smooth curves. Babel instead looks for what the current primitives cannot draw, and adds primitives for it. As a result, Babel matches LILO, the strongest of these baselines, with programs half as long, and is far ahead of TroVE. Note that macro extraction has no AutoDoc and only 21.5% of programs use its macros, so we treat LILO as the main baseline.


Question 2 (b). "For Table 8, can the effect of preserving caller control be separated from the accompanying changes toward per-element records?"

Yes. We tested the two changes separately and together, in the paper's per-subtopic setting (1,815 tasks; pass rate %, Judge 1 / Judge 2):

Without per-element recordsWith per-element records
Without caller-control rule93.33 / 92.4586.72 / 85.34
With caller-control rule90.52 / 89.2098.24 / 97.25
  • Caller-control rule alone, averaged: +4.35 [1.54, 7.30] / +4.33 [1.38, 7.30] points
  • Per-element records alone, averaged: +0.55 / +0.47 points
  • Extra gain from using both together: +14.33 [8.37, 20.66] / +15.15 [9.14, 21.60] points

The two changes work mainly in combination. With both in place, the share of primitives that make choices the caller cannot override falls from 67.1% to 5.1%. We will rewrite B.11 to say this.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer LU7f (Part 3 / 3)

[Part 3/3]

Weakness 3 (a) / Question 3 (a). "The strongest compression result comes from 121 subtopic-specific registries, rather than the single shared language ... the shared and isolated configurations used for the headline results have different emergence budgets." / "Can the headline shared and isolated configurations be compared with matched budgets and proposal opportunities?"

We compared the two with the same budget:

Comparison (same budget)Pass rate difference (Judge 1 / Judge 2, points)Calls per figure
Shared vs. per-subtopic, 15 evolution tasks per subtopic−1.43 [−2.92, 0.22] / −1.65 [−3.64, 0.17] (no clear difference)14.4 vs. 28.3
Shared vs. per-subtopic, 1 evolution task per subtopic+0.88 (no clear difference)—
Shared (15 per subtopic) vs. Human-28+8.71 [5.51, 12.29] / +9.86 [6.23, 13.83]—

With the same budget, one shared registry matches 121 per-subtopic registries in quality and needs half as many calls. This supports the paper's focus on a single shared language, and we now make this the main result. The per-subtopic registries are kept to show how far compression can go within one subtopic.


Question 3 (b). "How much do their results vary across complete emergence runs and different task orders?"

Very little. We reran the whole shared pipeline (evolution, generation and judging) three times with different task orders:

Task orderJudge 1Judge 2
198.7398.07
298.6297.25
398.4697.74
Standard deviation0.14 points0.42 points

This spread is far below the smallest difference our test can detect (about 1.4 points). All three reruns are 1.7–1.9 points above the number in the paper, so the paper's figure is on the low side. Different task orders lead to different primitive names (31% overlap), but with the same quality. The method reliably finds a good language, though not always the same one.


Weakness 3 (b). "The favorable cross-domain transfer average excludes Chemistry as a source ... it substantially qualifies the abstract's claim that transfer comes 'at no cost in quality.'"

We now report the full transfer matrix, including Chemistry. With Chemistry included, the 30 cross-domain cells average 88.4%, against 94.3% when each registry is used in its own domain (difference −6.0 points [−13.3, +0.2]).

The low Chemistry row comes from that one evolved registry, not from Chemistry itself. We evolved the Chemistry registry again three times with the same budget and different task orders. Its average transfer pass rate rose from 54.4% to 96.8% (+19.4 to +61.1 points per target domain; McNemar p ≈ 1e-9 to 0.02). The original registry scored about 54.4% every time we re-tested it, so this is not test noise.

We will update the abstract and §5.4 to give the full numbers and this finding, and recommend evolving more than one registry and picking one on a small validation set.


To sum up: (1) beyond batching, Babel programs are still about 2× shorter, and evolution keeps finding new primitives even from a batched start; (2) learning from attempted tasks clearly beats designing from descriptions, and sets Babel apart from LILO and TroVE; (3) with the same budget, one shared language matches 121 per-subtopic ones with half the calls, and results are stable across runs. We would be grateful if the reviewer would consider raising the score.

Reviewer_5xyTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Babel, a system that, rather than using a fixed visualization language, automatically adds and refines the drawing primitives necessary for solving a task, thereby evolving the visualization language itself. Babel consists of an “inner loop” that generates visualizations using the current language, and an “outer loop” that detects expressive limitations in the results and updates a shared primitive registry. Evaluation across six scientific fields, 121 subtopics, and 3,630 visualization tasks showed that the generated language maintained high rendering quality with fewer commands than the initial four primitives or manually designed DSLs. Furthermore, the authors demonstrated that the resulting primitives are formed as general-purpose drawing mechanisms reusable across multiple fields, rather than as policies that embody domain-specific judgments.

Critical strengths
  1. The approach of evolving the visual language itself in a task-driven manner is novel. Whereas it was previously common to generate programs within fixed frameworks such as SVG or DSLs, this paper treats the language itself as the target of optimization. Furthermore, the authors propose a mechanism for discovering the necessary level of abstraction from task distributions by adding or modifying primitives based on a lack of expressiveness. This paper aims to create a language that can be shared and reused.

  2. This paper comprehensively evaluates properties such as compressibility, sharing, transferability, and abstraction through a large-scale benchmark. The authors prepared 3,630 scientific visualization tasks across 6 fields and 121 subtopics, evaluating not only generation quality but also the number of commands and registry growth. Furthermore, through comparisons of shared and isolated registries, cross-domain transfer, and analysis of the emergence budget, they also investigate how primitives are reused.

Critical weaknesses
  1. There are serious concerns regarding the validity of evaluating compression based on command count. For example, when multiple per-element records or arguments can be passed within a single call, a simple command count does not allow for a fair comparison of information density across languages. The authors should first clearly explain what practical benefit command compression is intended to provide. Furthermore, if the goal is to reduce the computational cost of the VLM, the claim that the Bable's representation has truly become more compact cannot be sufficiently demonstrated without at least a comparison based on token count.

  2. The proposed method’s advantages over existing methods are unclear. According to Table 7, there is no difference in “Pass” compared to TikZ/PGF, and TikZ/PGF actually has fewer commands. It is unclear where Babel’s practical advantages lie compared to TikZ/PGF.

Questions for authors

Please see the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5xyT (Part 1 / 2)

[Part 1/2]

We thank the reviewer for their time and for the clear comments on how compression should be measured.

A few names used below: Seed-4 is the 4-primitive starting language (points, lines, circles, text). Batched Seed-4 is a new baseline that adds list versions of these 4 primitives, so many calls can be merged into one but nothing new can be drawn. Per-subtopic means one registry per subtopic (Table 2); shared means one registry for all tasks. Judge 1 / Judge 2 are two independent judges; brackets are 95% confidence intervals.


Weakness 1 (a). "There are serious concerns regarding the validity of evaluating compression based on command count ... the claim that the Babel's representation has truly become more compact cannot be sufficiently demonstrated without at least a comparison based on token count."

We agree that call count can mislead, since one call can carry many records. We now also count how many values the model has to write (coordinates, colors, labels), and the number of program tokens. Merging calls does not change either number.

Seed-4 compared with Babel, on 1,815 tasks:

MeasureSeed-4 / Babel (per-subtopic)Seed-4 / Babel (shared)
Calls4.17×3.75×
Values the model writes2.09×1.77×
Program tokens2.57×2.25×

So by token count, Babel programs are still 2.3–2.6× shorter.

To check how much of this is just batching, we compared against Batched Seed-4:

MeasureSeed-4 / Batched Seed-4Batched Seed-4 / Babel (per-subtopic)Batched Seed-4 / Babel (shared)
Calls6.31×0.67×0.61×
Values the model writes0.87×2.37× [2.15, 2.62]2.03× [1.82, 2.26]
Program tokens1.37×1.88× [1.73, 2.05]1.65× [1.50, 1.82]

Batching explains the drop in calls, but it does not reduce what the model writes. Even compared with Batched Seed-4, Babel programs are 1.65–1.88× shorter. The reason is that evolved primitives draw things Seed-4 cannot. For example, a smooth curve becomes one formula instead of hundreds of short line segments. When we restart evolution from Batched Seed-4, it finds 7 such primitives and cuts program tokens by a further 2.00× [1.82, 2.19] at the same quality.

In the paper, call count will be one of these measures, not the main one.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer 5xyT (Part 2 / 2)

[Part 2/2]

Weakness 1 (b). "The authors should first clearly explain what practical benefit command compression is intended to provide."

Shorter programs help in two ways: they fail less often when output length is limited, and they cost less when the model does not spend most of its tokens on reasoning.

Output-length limits. We capped total output at the 25th, 50th and 75th percentile of Seed-4's measured output length, on 149 tasks (pass rate %, Judge 1 / Judge 2):

Output capSeed-4Batched Seed-4BabelBabel minus Seed-4 (Judge 1, points)
17,238 tokens24.2 / 23.524.8 / 24.841.6 / 42.3+17.45 [10.14, 25.00]
24,963 tokens60.4 / 59.160.4 / 58.477.2 / 77.2+16.78 [7.43, 26.62]
32,040 tokens86.6 / 83.986.6 / 85.294.6 / 93.3+8.05 [1.42, 15.60]

Batched Seed-4 does not help at any cap, so the gain comes from writing less, not from merging calls. Other models show the same pattern. Babel beats Seed-4 by 21.48 [12.84, 30.38] points with claude-sonnet-4-6, by 11.41 [3.11, 20.13] with gemini-3.7-flash, and by 6.71 [0.66, 12.90] with gpt-4o.

Measured cost and time. These numbers come from the call logs of the same runs. They cover generation only, including retries and excluding judging. A ratio above 1 means Babel is cheaper or faster.

ModelCost, Seed-4 / BabelTime, Seed-4 / BabelPass rate, Seed-4 → BabelCost, Seed-4 / Batched Seed-4
claude-sonnet-4-61.44 [1.26, 1.64]1.70 [1.39, 2.08]51.0 → 72.50.97 [0.88, 1.05]
gemini-3.7-flash1.35 [1.28, 1.42]1.75 [1.54, 2.06]83.2 → 94.61.10 [1.04, 1.17]
gpt-4o0.98 [0.84, 1.13]1.10 [0.94, 1.29]7.4 → 14.10.92 [0.82, 1.04]
main model (long reasoning)1.10 ($0.203 → $0.185)1.15about equal1.03 [1.02, 1.04]
  • On claude-sonnet and gemini-flash, Babel is more accurate, 26–31% cheaper and about 1.7× faster. Batched Seed-4 saves nothing on claude-sonnet.
  • On gpt-4o, cost per task is the same. Babel passes almost twice as often, so the cost per correct figure falls from $2.41 to $1.30.
  • The main model writes about 17–18k reasoning tokens per task in every language, so the saving is smaller there.

Registry and evolution cost. The registry description adds 731 tokens for Seed-4 and 2,177 for the shared registry, about $0.002 per task. This is already included above. Evolving the shared registry once costs about $45, which is paid back after about 1,530 tasks on the main model, and sooner on the others.

We will add these results to §5.2.


Weakness 2. "According to Table 7, there is no difference in 'Pass' compared to TikZ/PGF, and TikZ/PGF actually has fewer commands. It is unclear where Babel's practical advantages lie compared to TikZ/PGF."

We reran the comparison on all 1,815 tasks instead of the 121-task subset. We also split the tasks by whether TikZ uses its pgfplots plotting library (pass rate %, Judge 1):

TasksCountTikZ passBabel passTikZ commands (loops expanded)Babel commands
Charts (TikZ uses pgfplots)47894.9898.9514.4839.88
All other tasks1,33797.8397.9842.0624.12
All tasks (Judge 1 / Judge 2)1,81597.08 / 96.7598.24 / 97.2534.828.3
  1. Quality is the same. TikZ minus Babel is −1.17 [−2.61, 0.61] points (Judge 1) and −0.50 [−2.11, 1.22] (Judge 2).
  2. TikZ's low command count comes from pgfplots. On the 26% of tasks that are standard charts, pgfplots draws axes and grids for free. There TikZ needs far fewer commands, but it passes about 4 points less often. On the other 74%, once \foreach loops are expanded, TikZ needs 42.06 commands against Babel's 24.12. Table 7 counted TikZ commands without expanding loops. We now report both ways.
  3. What Babel offers. TikZ/PGF was built by hand over decades and comes with a dedicated plotting library. Babel starts from 4 primitives and, after about $45 of automatic evolution, matches TikZ in quality and uses fewer commands outside standard charts. Its programs are also typed records that can be checked automatically and extended for new tasks. So Babel is a way to build such a language automatically, which matters most in fields that have no mature drawing language.

We hope these results answer both concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_5xyTOpenReview ↗

Response to Authors

I am deeply grateful to the authors for their thoughtful response and additional experiments. These additional experiments have largely alleviated my concerns regarding Weakness 1.

However, my concerns regarding Weakness 2 remain. While the authors compare the number of commands in TikZ/PGF and Babel, I question whether comparing the number of operations required to expand TikZ loops with the number of primitive calls in Babel constitutes a fair evaluation method. To fairly compare generation efficiency, I think it would be preferable to directly compare the number of output tokens and generation times between TikZ/PGF and Babel.

AuthorsOpenReview ↗

We thank Reviewer 5xyT for the thoughtful comments and constructive suggestions.

We thank the reviewer for reading our response so carefully, and we are glad the new experiments addressed Weakness 1.

"I question whether comparing the number of operations required to expand TikZ loops with the number of primitive calls in Babel constitutes a fair evaluation method. To fairly compare generation efficiency, I think it would be preferable to directly compare the number of output tokens and generation times between TikZ/PGF and Babel."

We agree, and we now report exactly this. Both languages use the same model and settings, and the comparison covers the same 130 tasks (per task; brackets are 95% confidence intervals of the ratio):

TikZ/PGFBabel (per-subtopic)TikZ / Babel
Total output tokens (reasoning + program)18,01619,5120.92 [0.86, 0.99]
of which reasoning16,81917,2270.98 [0.90, 1.06]
of which program1,1972,2850.52 [0.47, 0.58]
Generation time134 s159 s0.85 [0.72, 0.96]

TikZ is somewhat more efficient to generate: about 8% fewer output tokens and 15% less time.

We think this is the expected result, and it does not affect the main claim. TikZ is a mature language that the model has seen in enormous amounts of training data, and it comes with a hand-built plotting library. Babel is a new language the model has never seen; it learns it only from a 1–2k-token description in the prompt, and it was built automatically from 4 primitives for about $45. Getting within about 10% of TikZ's generation cost under these conditions is the point of the paper.

What Babel gives in return is reliability. On all 1,815 tasks:

TikZ/PGFBabel (per-subtopic)
Programs that fail to render36 (35 LaTeX compile errors)3
Pass rate (Judge 1 / Judge 2)97.08 / 96.7598.24 / 97.25
Lowest pass rate across the 6 fields94.17 (Geo-engineering)96.32 (Physics)
Fields where it has the higher pass rate14 (1 tie)

Because every Babel program is a list of typed records checked against the registry, it almost never fails to render, about 12× less often than TikZ. Its quality is also more even across fields. This matters in practice: when a generated figure fails to compile, the whole generation cost is lost and has to be paid again.

So our claim against TikZ is not that Babel is cheaper to generate, but that an automatically built language reaches TikZ-level quality and generation cost, with fewer failures and more even quality across fields, in domains where no mature language like TikZ exists. We will state this plainly in the revision. We hope this answers the reviewer's remaining concern, and we would be grateful if the reviewer would consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.