- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARTA proposes a "training-free" LLM agent to map tensor programs onto spatial accelerators, aiming to replace per-task RL tuning (like Soter). The LLM uses a human-authored strategy text to propose spatial allocations, loop priorities, and tile caps. These proposals are snapped to valid hardware constraints by a deterministic builder and evaluated analytically via Timeloop to minimize the EDP. The authors rigorously evaluate this against RL and heuristic baselines across 90 layers on three modeled architectures. They find that while the LLM beats the RL baseline, the real heavy lifting is done by the human-written strategy text, with the LLM adding only a marginal improvement over a non-LLM annealer given the exact same text.
Critical strengths
-
They systematically deconstruct why LLM wins by introducing controls like the strategy-seeded annealer. This effectively proves that the human-authored text is doing the heavy lifting, serving as a much-needed reality check on the current hype around LLMs natively optimizing spatial architectures.
-
The paper convincingly demonstrates that the Soter RL Tuner is an incredibly weak baseline. By showing that an unconstrained random search at 500 evaluations easily beats the Tuner, the authors provide a valuable recalibration for the community regarding how we evaluate mapping frameworks.
Critical weaknesses
-
The framework relies entirely on Timeloop's analytical scores, assuming perfectly fluid spatial fanouts that ignore physical routing constraints. Treating a 64-wide spatial fanout as a frictionless allocation ignores the routing congestion and multiplexing overhead inherent in driving an interconnect or NoC. Furthermore, the claim of search efficiency relies entirely on minimizing evaluator calls. In wall-clock time, the LLM API latency makes this approach drastically slower than local heuristic searches, which can execute roughly 2,000 evaluations in the same timeframe and reach parity.
-
The paper framed in the introduction as a novel LLM framework, but the actual narrative is that LLMs are mostly just parsing human intuition and barely beat traditional search heuristics. This narrative is buried under an overwhelming details and the writing makes it exhausting to extract the core points. For example, the text unleashes a barrage of unanchored ratios (1.85x, 1.61x, 1.66x, 1.21x, 1.06x, 0.92x) across different evaluation budgets (200 calls, 320 calls, 500 calls) in the very first paragraph. Also, the paper switches between differently constrained baselines, comparing the LLM to the "fair Tuner," "latency-directed Tuner," "unported agent," "ported agent," and "strategy-seeded annealer". This forces the reader to memorize highly specific terms to understand the contents.
Questions for authors
- Why is the paper framed as an LLM systems paper when the ablation data strongly supports an anti-LLM, pro-heuristic-with-priors conclusion?
- Can you provide a wall-clock time vs EDP Pareto frontier figure in the main text, plotting the LLM against the random/annealer baselines?
- Have you tested these LLM-proposed mappings on cycle-accurate simulators to see if the spatial allocations hold up against actual hardware that includes non-ideal assumptions such as interconnect routing congestion?
Thank you for the careful review and the clear reading of our results. We agree with it, and the revision adopts it: we priced the on-chip network, added the wall-clock Pareto figure to the main text, reframed and rewrote the paper as a measurement study, and counted which mappings fit real hardware; new content is marked in blue in the revised PDF. Throughout, a ratio is the other method's EDP divided by CARTA's, so above 1 means CARTA is better; brackets are 95% intervals.
1. Network realism (W1)
Once the on-chip network is priced, simple search ties or beats CARTA. On Simba, the 64-wide level you mention is the multiply lanes inside each processing element, not a routed network; the other wide levels are real networks, and our Timeloop build charges every method nothing for them (Sec. 4). In a pre-registered experiment we charged wire energy, reduction adders and link bandwidth to first order and re-scored every method's best mapping on all 270 cases (intervals in App. F.5):
The paper now states every CARTA lead over these controls as holding only for a free network. Two caveats: this is a first-order model and does not capture congestion; and the re-run controls matched only 94.9-97.9% of stored values against our 99% bar, so by our own rule they are flagged not reproducible, although their ratios agree with the stored ones within 0.23% (Sec. 5.2, App. F.5).
2. Wall-clock time and the Pareto figure (W1, Q2)
Added: Figure 9 in the main text plots EDP against wall-clock time per layer for CARTA, random search, SA, the same-text annealer and the GP. CARTA has no established wall-clock lead over the annealer: at equal time the annealer has 1.030× CARTA's EDP at the mean call cost and ties at the median-layer cost (0.996; post hoc, modelled). An evaluator call takes a median 0.34-0.56 s on our host, not the 0.1 s the submission assumed. You were also right about parallel search: with 4 parallel evaluators random search is within the margin of error, and with 16 it reaches parity (Sec. 5.5, App. F.7).
3. Framing and writing (W2, Q1)
The paper should not have been framed as an LLM systems paper, and the revision changes that. It is now a measurement study, retitled "CARTA: The Strategy Text, Not the LLM, Carries Most of the Gain in Tensor Mapping". The abstract opens with this finding, and CARTA is described as a measurement instrument, not a new method. Our pre-registered tests of what the LLM adds are inconclusive on two current models (Sec. 5.3). The writing is rewritten: three ratios in the abstract, one name per concept, no internal codes in the main text, and a table of five questions and answers at the start of Sec. 5.
One correction on "human-authored text": the two port texts were drafted by an LLM coding assistant in one call each, in sessions that had seen some test results (so the port results may be optimistic, though CARTA and the annealer share the same text). A pre-registered test of DeepSeek writing the text itself did not pass its main test: its text was worse than the assistant-drafted one (Sec. 5.4).
4. Cycle-accurate simulation (Q3)
Not run; we claim no cycle-level fidelity. Besides the network pricing, we counted mappings that fit published designs. On GemminiLike almost none fit Gemmini's fixed dataflow (CARTA 0/60, Tuner 0/30 per seed), because our model relaxes it for all methods. On Simba, the annealer and the GP fit about as many as CARTA (53-57 against 48-60 of 90 per run; same 95–98% re-run match note; App. F.6). For the camera-ready we commit to RTL simulation of every method's best mappings on 10 GemminiLike layers. This is planned for the camera-ready, not current evidence.
Thank you again for your time and feedback. We are happy to answer any further questions.