- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Babel evolves a registry of visual primitives through alternating task solving and primitive proposal. Starting from four geometric operations, a VLM generates visual programs, inspects task-level evidence, and proposes reusable drawing functions. The paper evaluates program compactness, rendering quality, registry-sharing strategies, and transfer on a benchmark covering six scientific domains.
Critical strengths
Treating the drawing interface as something that can adapt to a task distribution is an interesting direction. The experiments demonstrate substantial reductions in primitive-call counts while retaining high judged quality.
The paper provides extensive supplementary analyses, including two VLM judges, clustered confidence intervals, external drawing-language comparisons, and negative results. These efforts help identify the limits of the reported findings.
Critical weaknesses
-
The contribution beyond existing library learning and tool synthesis is insufficiently established. Many emerged primitives accept lists of explicitly specified geometric records and batch repeated drawing operations. This is useful, but fewer calls alone do not establish a substantially better representation or synthesis process. Comparisons with simple batching, macro extraction, or persistent tool-library construction are needed to identify what the proposed visual-feedback-driven search adds. The paper should also report program tokens, latency, and total cost, including registry evolution, to establish whether call-count compression translates into practical efficiency.
-
The experimental presentation makes the central conclusion difficult to assess. The headline results use isolated per-subtopic registries, whereas the method and narrative emphasize a single shared language. Quality improvements over Seed-4 and Python are modest relative to the reported uncertainty, and the external TikZ comparison does not establish a clear compactness advantage. Existing ablations examine sharing, budget, and proposer constraints, but do not directly isolate the value of rendered visual feedback. The paper needs a clearer separation of configurations and stronger controls for its central mechanism.
Additional Presentation Concerns
-
The writing relies heavily on broad terms such as “emergence,” “mechanisms rather than policies,” and “language,” without making their distinction from reusable function libraries sufficiently clear. A concrete example tracing a task, diagnosed gap, proposed primitive, validation, and subsequent reuse would make the contribution easier to understand.
-
Figure 4 should be redrawn with larger text, fewer decorative icons, and clearer data-flow arrows and intermediate artifacts.
Questions for authors
How does Babel compare with a simple baseline that adds list-valued batching wrappers or extracts reusable macros from generated programs? What benefit specifically requires rendered visual feedback?
Can the main results consistently foreground the shared-registry configuration and explain how its conclusions differ from those of the isolated per-subtopic registries?
Do fewer primitive calls reduce generated tokens, latency, or API cost after accounting for primitive arguments, registry serialization, and the outer-loop evolution cost?
5 comments
Response to Reviewer 6zU6 (Part 2 / 3)
[Part 2/3]
Weakness 1 (cont.) / Question 3. "The paper should also report program tokens, latency, and total cost, including registry evolution ..." / "Do fewer primitive calls reduce generated tokens, latency, or API cost after accounting for primitive arguments, registry serialization, and the outer-loop evolution cost?"
Yes. Program tokens drop 2.25–2.57×. Cost per task drops about 10% on our main model, which spends most of its tokens on reasoning, and about 30% on models that reason less, where runs are also about 1.7× faster. All numbers include the registry description, and the cost of evolution is paid back after about 1,530 tasks.
Why calls go down. "Calls" is the number of primitive calls in the program for one figure: about 117 for Seed-4 and 28–31 for Babel. There are two reasons:
- Merged calls. Drawing 100 points becomes one list call, but all 100 coordinates are still written.
- New primitives. A smooth curve becomes one formula instead of hundreds of short segments, so less is written.
Only the second reason saves tokens and money, as the batching numbers below show.
Main model, compared with Seed-4, 1,815 tasks:
| Measure | Per-subtopic | Shared |
|---|---|---|
| Program tokens | 2.57× fewer | 2.25× fewer |
| Total tokens, measured (input + reasoning + output) | 1.19× fewer | 1.09× fewer |
| Registry description in the prompt | 1,398 tokens (average) | 2,177 tokens (about $0.002 per task) |
| Cost per task, measured | 1.10× lower ($0.203 → $0.185) | 1.17× lower |
| Time per task, measured | 1.15× lower | 1.11× lower |
| One-time evolution cost | about $770 | about $45 |
| Tasks needed to pay it back | about 43,000 | about 1,530 |
This model writes about 17–18k reasoning tokens per task in every language, so the saving is smaller. With models that reason less, shorter programs save directly (a ratio above 1 means Babel is cheaper or faster):
| Model | Cost, Seed-4 / Babel | Time, Seed-4 / Babel | Pass rate, Seed-4 → Babel (Judge 1) |
|---|---|---|---|
| claude-sonnet-4-6 | 1.44 [1.26, 1.64] | 1.70 [1.39, 2.08] | 51.0 → 72.5 |
| gemini-3.7-flash | 1.35 [1.28, 1.42] | 1.75 [1.54, 2.06] | 83.2 → 94.6 |
| gpt-4o | 0.98 [0.84, 1.13] | 1.10 [0.94, 1.29] | 7.4 → 14.1 |
- Batched Seed-4 saves nothing on claude-sonnet (0.97 [0.88, 1.05]) and 3% on the main model. Merging calls alone does not save money.
- On gpt-4o, cost per task is the same, but Babel passes almost twice as often, so the cost per correct figure falls from $2.41 to $1.30.
- With output length capped, shorter programs also fail less (pass rate %, Judge 1, 149 tasks; caps are percentiles of Seed-4's output length):
| Output cap | Seed-4 | Batched Seed-4 | Babel | Babel minus Seed-4 (points) |
|---|---|---|---|---|
| 25th (17,238 tokens) | 24.2 | 24.8 | 41.6 | +17.45 [10.14, 25.00] |
| 50th (24,963 tokens) | 60.4 | 60.4 | 77.2 | +16.78 [7.43, 26.62] |
| 75th (32,040 tokens) | 86.6 | 86.6 | 94.6 | +8.05 [1.42, 15.60] |
Weakness 2 / Question 2. "The headline results use isolated per-subtopic registries, whereas the method and narrative emphasize a single shared language." / "Can the main results consistently foreground the shared-registry configuration and explain how its conclusions differ from those of the isolated per-subtopic registries?"
Yes. We now lead with the shared registry.
- Same budget (15 evolution tasks per subtopic). Shared and per-subtopic have the same quality (−1.43 [−2.92, 0.22] points, Judge 1), but the shared registry needs half as many calls (14.4 vs. 28.3 per figure). It is also 8.71 [5.51, 12.29] points above Human-28.
- Stable. Three full reruns of the shared pipeline with different task orders vary by only 0.14 points (standard deviation).
- What each shows. The per-subtopic registries are 121 small libraries (6.4 primitives on average) and show how far compression can go inside one subtopic. The shared registry is one language and supports the claims about reuse across fields and a vocabulary that stops growing. We now use each only for its own claims.
(continued in the next comment)
Response to Reviewer 6zU6 (Part 3 / 3)
[Part 3/3]
Weakness 2 (cont.). "Quality improvements over Seed-4 and Python are modest relative to the reported uncertainty, and the external TikZ comparison does not establish a clear compactness advantage."
Quality. All baselines were run with the same output limit and file handling (1,815 tasks; Babel per-subtopic: 98.24 / 97.25):
| Baseline | Pass rate (Judge 1 / Judge 2) | Babel minus baseline, Judge 1 (points) | Babel minus baseline, Judge 2 (points) |
|---|---|---|---|
| Seed-4 | 97.25 / 95.59 | +0.99 [−0.66, 2.26] | +1.65 [0.11, 3.09] |
| Python | 96.75 / 94.93 | +1.49 [−0.44, 3.14] | +2.31 [0.33, 4.30] |
Babel is at least as good as both under both judges, and clearly better under Judge 2. We agree that pass rate is not the main contribution. We present quality as "no worse than strong baselines", and the main contribution as shorter programs, fewer failures under output limits, and lower cost.
TikZ, on all 1,815 tasks, split by whether TikZ uses its pgfplots plotting library (pass rate %, Judge 1):
| Tasks | Count | TikZ pass | Babel pass | TikZ commands (loops expanded) | Babel commands |
|---|---|---|---|---|---|
| Charts (TikZ uses pgfplots) | 478 | 94.98 | 98.95 | 14.48 | 39.88 |
| All other tasks | 1,337 | 97.83 | 97.98 | 42.06 | 24.12 |
| All tasks (Judge 1 / Judge 2) | 1,815 | 97.08 / 96.75 | 98.24 / 97.25 | 34.8 | 28.3 |
- Quality is the same: TikZ minus Babel is −1.17 [−2.61, 0.61] points (Judge 1).
- TikZ's low command count comes from pgfplots, which draws axes and grids for free on the 26% of tasks that are standard charts; there TikZ passes about 4 points less often. On the other 74%, with loops expanded, TikZ needs 1.74× more commands than Babel.
- TikZ/PGF was built by hand over decades. Babel reaches the same quality automatically from 4 primitives for about $45, and its programs are typed records that can be checked automatically and extended for new tasks.
Presentation 1. "The writing relies heavily on broad terms such as 'emergence,' 'mechanisms rather than policies,' and 'language' ... A concrete example tracing a task, diagnosed gap, proposed primitive, validation, and subsequent reuse would make the contribution easier to understand."
Definitions (added to §3):
- Emergence: a primitive that is not in Seed-4, was proposed during evolution, and passed the checker.
- Mechanisms rather than policies: a primitive makes no choice the caller cannot override, such as a fixed color or automatic layout. An automatic check (rules plus an LLM) agrees with human labels on 10 of 10 shared primitives and 282 of 294 per-subtopic primitives.
- Language rather than library: one registry used across fields, whose vocabulary stops growing, and whose primitives are reused.
Worked example (new appendix): draw_filled_regions.
- In round 21, a UV-Vis task had to fill the area under a peak with many thin lines.
- The proposer noticed that filled regions could not be drawn.
- It proposed
draw_filled_regions, which takes a list of regions. - The checker accepted it. A later edit added an optional
hatchfield. - It was then used in 1,165 test figures across 6 fields. On the 1,142 tasks that also have a Seed-4 program, it needs 3.90× fewer calls, 1.77× fewer written values and 2.29× fewer tokens.
Presentation 2. "Figure 4 should be redrawn with larger text, fewer decorative icons, and clearer data-flow arrows and intermediate artifacts."
Done. Figure 4 now uses larger text and no decorative icons. Arrows show each step and what it produces: task → program → render and check → missing ability found → new primitive → registry.
We thank the reviewer again. The new comparisons with batching, macros, LILO, TroVE and description-only design make the contribution much clearer: learning from attempted tasks finds new drawing abilities the alternatives cannot, giving shorter programs at the same quality, fewer failures under output limits, and lower cost. We would be grateful if the reviewer would consider raising the score.
Response to Reviewer 6zU6 (Part 1 / 3)
[Part 1/3]
We thank the reviewer for their time and detailed comments. The main question is what Babel's search adds beyond batching and existing library learning.
A few names used below: Seed-4 is the 4-primitive starting language (points, lines, circles, text). Batched Seed-4 is a new baseline that adds list versions of these 4 primitives, so many calls can be merged into one but nothing new can be drawn. Per-subtopic means one registry per subtopic (Table 2); shared means one registry for all tasks. Judge 1 / Judge 2 are two independent judges; brackets are 95% confidence intervals.
Batching, macros and tool libraries. All baselines use the same seed language, tasks, checker and registry format, and are tested on all 1,815 tasks:
Babel is the only method that is both accurate and short. Batching, macros and LILO can only package existing primitives, so their programs stay long. The TroVE-style toolbox is short but loses almost 18 points.
What Babel adds is new drawing abilities. 8 of the 10 shared primitives draw things Seed-4 cannot: filled regions, curves from points or formulas, ellipses, box plots, and node-link diagrams. 87.7% of Babel programs use them. When we restart evolution from Batched Seed-4, it finds 7 more such primitives, which cut written values by 2.13× [1.91, 2.38] and program tokens by 2.00× [1.82, 2.19] at the same quality. So the search finds more than batching does.
What needs feedback. We changed only what the proposer sees, keeping the budget, rules, checker and task order fixed:
We will describe the feedback in §3 and B.4 in these terms and add the full table.
(continued in the next comment)