acceptodds
Under review as a conference paper at ICLR 2027

Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

Abstract

Long-form subtitle translation presents challenges beyond those of conventional machine translation: a single episode may contain hundreds of sentences whose meanings depend on long-range discourse and cultural context spanning the entire episode or even series, while translation quality also requires maintaining consistent terminology and style throughout the series. Existing approaches remain limited. Single-LLM methods operate at the sentence level, lacking long-context understanding and consistent terminology across episodes. Multi-agent methods often rely on static workflows that fail to adapt to scene complexity. In addition, both paradigms ignore the production context, e.g., genre. We propose SMART, a Self-evolving Multi-Agent system for long-foRm subtitle Translation. SMART operates in two stages: during test-time training, it builds a persistent series-level memory and iteratively translates a subset of sentences through a dynamic graph router and a Mixture-of-Agents translation layer, equipped with tool-calling modules for terminology verification, subtitle constraint validation, and contextual retrieval; a judge-refiner loop scores each candidate translation and back-propagates textual critiques that refine agent prompts and the routing policy, without retraining the underlying LLMs. During test-time inference, this evolved configuration translates the remaining sentences of the series. To evaluate long-form subtitle translation at scale, we introduce Subtitle Arena, covering 14 genres, 2–198 episodes per series, production years from 1959 to 2023, and 15 target locales, together with SubMQM, a subtitle-adapted MQM framework with seven dimensions and 19 error categories. SMART achieves the best overall MQM score in all 15 Subtitle Arena directions, reducing the average penalty by 6.9% over the strongest competing agent system. On a public benchmark, MuSC, SMART obtains the best model result across all 4 language pairs. SMART also achieves the best result in human evaluation with an overall score of 4.50/5.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Subtitle Translation, Multi-Agent Systems, Test-Time Adaptation, Long-Form Translation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2qLyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SMART, a multi-agent system for long-form subtitle translation that combines series-level memory, dynamic routing, tool use, and test-time updates to prompts and routing policies. It also introduces Subtitle Arena, a benchmark constructed from OpenSubtitles2024.

Critical strengths
  1. SMART achieves the lowest Overall SubMQM penalty across all 15 English-to-target-language directions in Subtitle Arena, and the best model results on all four MuSC directions.
  2. In a small-scale human evaluation, SMART ranks first in fidelity, consistency, language quality, and subtitle quality.
Critical weaknesses
  1. The methodological novelty appears limited: SMART primarily integrates and adapts existing techniques to long-form subtitle translation.
  2. Although the paper presents Subtitle Arena as a benchmark, it is derived from OpenSubtitles2024. Its contribution appears to lie primarily in reconstructing and reorganizing existing subtitle alignments at the episode and series levels
  3. The Subtitle Arena experiments do not report common metrics such as spBLEU or COMET.
Questions for authors
  1. The main experiments use a newly constructed benchmark and a custom evaluation metric, but the paper does not state whether the code and data will be made publicly available.
  2. The baselines in Table 3 are outdated; please include Gemma 4 31B and DeepSeek V4.1.
  3. Table 4 does not report Claude Sonnet 4.6 performance on MuSC, despite its use as the paper’s default API model.
6 comments

We thank the reviewer for the constructive comments. We appreciate the recognition that SMART has the lowest Overall SubMQM penalty in all 15 English-to-locale directions of Subtitle Arena and the best model results in all four MuSC directions, as well as the acknowledgment that it ranks first on all four criteria in the human evaluation. Our responses to each weakness and question follow.


1. The methodological novelty appears limited: SMART primarily integrates and adapts existing techniques to long-form subtitle translation.
We would like to clarify that the main methodological novelty of SMART lies in constrained test-time evolution of its prompts and multi-agent structure within one long-form task. The techniques that SMART builds on typically do not target this setting. The closest prior approaches either adapt a single LLM to subtitles or use multi-agent systems. A single LLM can be fine-tuned for subtitles, as in ALPO, or prompted with neighboring dialogue, genre, or plot summaries. In multi-agent systems such as TransAgent, specialized roles generate and refine translations. For long-form subtitles, these approaches fall short in three ways. First, without persistent series-level context, recurring names and terms drift across episodes. Second, with roles and refinement steps fixed in advance, the workflow does not adapt to scene complexity. Third, current systems have limited ability to retrieve and use production context such as genre. Self-evolving agents and prompt optimizers learn from feedback, but they typically optimize for future tasks or a fixed objective. Long-form subtitles instead call for adapting within one series while preserving the behavior acquired earlier.

SMART therefore runs in two stages on each series. During test-time training on the first 30%, judge feedback updates both the agent prompts and the multi-agent structure. The structure update rewrites the routing table, which sets which translators and tools are active for each category of sentence. It removes an agent from a category where it keeps underperforming and reassigns tools (Sec. 3.5 and Appendix B.8). Both updates are constrained rewrites. Each agent keeps its role and workflow, and each routing category keeps at least two agents. During test-time inference on the remaining 70%, the prompts and routing stay frozen. The series memory keeps accumulating names, terms, and style to preserve consistency across episodes. In both stages, the router selects translators and tools for each sentence. SMART's contextual retrieval tools supply production context, such as genre or historical setting. The differences from translation agents, self-evolving agents, and prompt optimizers are also stated in Sec. 2.

2. Although the paper presents Subtitle Arena as a benchmark, it is derived from OpenSubtitles2024.
We would like to clarify that the core of our paper is the method, SMART. As stated in Sec. 1 of the original submission, we built Subtitle Arena to evaluate SMART on long-form series. Its source, OpenSubtitles2024, is a sentence-level parallel corpus in many languages without an evaluation protocol (Table 9). Some of its sentence pairs are noisy or misaligned. We used the IMDb identifier and the season and episode indices of each document as the episode key. We then rebuilt episode-level SRT streams with the original timecodes and discarded empty or one-sided alignments. Finally, we grouped the episodes into 192 series of 2–198 episodes each. We split each series into a 30% prefix for SMART's test-time training and a 70% held-out portion for evaluation.

Beyond OpenSubtitles2024, which contains subtitles released before 2024, the original submission also evaluated SMART on 200 TV series released in 2025–2026. We collected this set separately, and it shares no title with Subtitle Arena. It includes titles released after the reported knowledge cutoff of Claude Sonnet 4.6, SMART's default backbone (Sec. 5.2 and Appendix F.4). The table reports the Overall SubMQM penalty in each direction (lower is better).

DirectionOnlineTransAgentSMART
en→zh1.870.480.44
en→de2.140.940.87
en→ko1.480.440.42
en→it2.330.810.78
en→es2.860.920.90
en→fr2.271.000.99

SMART had a lower Overall penalty than TransAgent in all six directions, with an average of 0.73 compared with 0.77 (Table 19).

3. Its contribution appears to lie primarily in reconstructing and reorganizing existing subtitle alignments at the episode and series levels.
The episode- and series-level reorganization is what the evaluation of SMART needs, because it makes long-range consistency measurable. Among the resources in Table 9, Subtitle Arena is the only one with series context. MuST-Cinema covers TED talks for speech-to-subtitle translation. MuSC, the resource closest to our setting, consists of multi-line utterances evaluated at the segment level. These resources are not designed to fully capture three properties of long-form subtitles. These are the use of long-range context, the consistency of names, terms, and style across a series, and respect for display constraints. Subtitle Arena is built for these three properties, and SubMQM adds an evaluation with 19 error types, which we checked against human MQM.

For example, we tracked recurring entities and terms on the held-out 70% of each series in the six directions of Table 6. An occurrence is consistent if it matches the system's first rendering. The cross-episode distance d is the number of episodes since the first occurrence. The table reports the consistency in percent.

Consistencyd = 0d = 1–2d = 3–5d = 6–10d > 10
TransAgent92.677.871.467.963.2
SMART w/o Memory94.081.275.671.168.3
SMART96.394.893.993.191.7

TransAgent's consistency fell from 92.6% within the episode of the first occurrence to 63.2% more than ten episodes later, compared with 96.3% to 91.7% for SMART. Without the series structure, this drop could not be measured. The evaluation protocol is also checked against human MQM. In a study of six systems in four directions, SubMQM agreed with the human annotations better than any other automatic metric, with a segment-level τ of 0.42 against 0.35 for AutoMQM. We have added the consistency analysis to the new Appendix F.8 (Table 22) and the human-MQM study to Appendix F.10 (Table 24).

4. The Subtitle Arena experiments do not report common metrics such as spBLEU or COMET.
Thank you for pointing this out. The only references in Subtitle Arena are the Online community subtitles. As stated in Sec. 5.1 of the original submission, these subtitles are collected in the wild and vary in quality. We therefore do not treat them as a controlled human upper bound. Reference-based metrics such as spBLEU and COMET would score every system against these noisy references. The original submission therefore used SubMQM, an LLM-as-a-judge evaluation with typed errors (Sec. 4.2). Following your suggestion, we additionally computed spBLEU and COMET-22 for five systems on all 15 English-to-locale directions, with the Online subtitles as the reference. The table reports the averages over the 15 directions, where higher is better.

MethodspBLEUCOMET-22
Claude Sonnet 4.623.980.4
GPT-5.524.780.8
DRT24.681.0
TransAgent25.381.6
SMART26.081.9

SMART achieves the best average on both metrics, with 26.0 spBLEU and 81.9 COMET-22, compared with 25.3 and 81.6 for TransAgent, the strongest baseline. It is the best in 13 of the 15 directions on spBLEU and the best or tied for best in 14 of the 15 directions on COMET-22. We keep SubMQM as the primary metric because it correlates better with human MQM. The table reports the agreement of each metric with the human labels in our human MQM study of six systems in four directions. Pairwise accuracy is the share of system pairs that a metric orders like the annotators.

MetricSegment-level τSystem-level rPairwise accuracy
SubMQM0.420.9491.7%
AutoMQM0.350.9186.7%
COMET-220.310.8683.3%
spBLEU0.170.7173.3%

SubMQM's segment-level τ was 0.42, compared with 0.31 for COMET-22 and 0.17 for spBLEU. We have added both metrics, with per-direction values, to the new Appendix F.12 (Table 28).

5. The main experiments use a newly constructed benchmark and a custom evaluation metric, but the paper does not state whether the code and data will be made publicly available.
Yes, the SMART and SubMQM code is available, and Subtitle Arena will be released as a reconstruction recipe. The SMART implementation and the SubMQM evaluator are in the anonymous repository, which Appendix I.1 of the original submission links to. The implementation covers all 30 directions and includes the agent prompts, initial routing table, tool interfaces, and series memory. The evaluator includes the 19-type rubric with its aggregation weights, the judge prompt, and the alignment and scoring code. The repository also contains the test-time training and inference scripts and one synthetic example SRT file.

We are adding the Subtitle Arena construction pipeline with the episode keys and splits, the SubMQM prompts, and the evolved SMART configurations. We will post a comment here once the upload is complete. As stated in Appendix I.2, we do not redistribute the subtitle text. Once the pipeline is uploaded, users can download OpenSubtitles2024 themselves and rebuild the episode-level SRT streams and the per-series splits.

The baselines in Table 3 are outdated; please include Gemma 4 31B and DeepSeek V4.1.
Thank you for suggesting these baselines. The single-call LLMs in Table 3 of the original submission included GPT-5.5 and Claude Opus 4.8. We evaluated Gemma 4 31B and DeepSeek V4.1 under the same setting as the other single-call LLMs in Table 3. The table reports the mean Overall SubMQM over the 15 English-to-locale directions, where lower is better. The GPT-5.5 and SMART means are from Sec. 5.2.

MethodMean Overall
GPT-5.51.44
Gemma 4 31B1.82
DeepSeek V4.11.46
SMART1.11

DeepSeek V4.1 performs comparably to GPT-5.5 (1.46 vs. 1.44), while Gemma 4 31B reaches 1.82. SMART keeps the lowest mean at 1.11. It also has the lowest Overall penalty in all 15 English-to-locale directions and in the four reverse directions of Table 3. We have added both models to Sec. 5.1 and to Tables 3, 11, and 12, which report the per-direction values.

6. Table 4 does not report Claude Sonnet 4.6 performance on MuSC, despite its use as the paper's default API model.
Thank you for suggesting this comparison. For Table 4, we followed the original MuSC benchmark (Cui et al., 2026b) and listed the systems it evaluated, which do not include Claude Sonnet 4.6. We evaluated Claude Sonnet 4.6 with the same ICL protocol as the other ICL models in Table 4. The table compares it with SMART on Accuracy, Naturalness, and Vividness, where higher is better. The SMART values are from Table 4 of the original submission.

DirectionClaude Sonnet 4.6 AccuracySMART AccuracyClaude Sonnet 4.6 NaturalnessSMART NaturalnessClaude Sonnet 4.6 VividnessSMART Vividness
en→zh91.694.386.188.268.780.4
ko→zh83.286.882.085.961.978.4
zh→en89.492.386.488.671.883.7
zh→th90.294.285.187.070.380.5

SMART achieves higher Accuracy, Naturalness, and Vividness than Claude Sonnet 4.6, its own backbone, in every direction. In every column of Table 4, Claude Sonnet 4.6 scores below the strongest prior system. These results suggest that the MuSC gain comes from our framework rather than from the backbone. We have added Claude Sonnet 4.6 to Table 4.

Reviewer_hMD9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SMART, a self-evolving multi-agent framework for long-form subtitle translation. It combines persistent series-level memory, dynamic routing, specialized translation agents, and test-time prompt optimization. The authors also introduce Subtitle Arena and SubMQM for evaluating multilingual subtitle translation. Extensive experiments report consistent improvements over existing systems.

Critical strengths
  1. The paper addresses an important problem in long-form subtitle translation, particularly cross-episode consistency and contextual adaptation.

  2. The proposed framework integrates several complementary mechanisms and provides extensive ablation studies.

  3. The evaluation covers multiple languages, independent datasets, and human assessments, showing promising improvements.

Critical weaknesses
  1. Limited methodological novelty. The main components, including multi-agent translation, persistent memory, dynamic routing, and prompt optimization, have been studied previously. The specific novelty of their integration and test-time evolution needs clearer justification.

  2. High computational cost. SMART requires approximately 4,970 LLM calls and 14M input tokens per episode. Its quality gains should be evaluated against baselines under comparable computational budgets.

  3. Insufficient validation of SubMQM. Although robustness across two LLM judges is examined, the agreement between SubMQM and human MQM annotations is not established. This is important because the main improvements are relatively modest.

  4. Limited validation of the incremental contribution of self-evolution. The ablation study demonstrates its benefit, but its improvement is smaller than those of memory and multi-agent translation. Further comparisons with existing prompt optimization methods would help establish its distinct contribution.

Questions for authors
  1. How does SMART compare with existing multi-agent translation systems under matched inference costs or token budgets?

  2. How well do SubMQM scores and error classifications correlate with human MQM judgments?

  3. What distinguishes the proposed self-evolution mechanism from existing iterative prompt optimization approaches, and what additional benefits does it provide under comparable computational conditions?

5 comments

We appreciate your thoughtful review and are encouraged by your recognition of our work: that the paper tackles an important problem in long-form subtitle translation, especially cross-episode consistency and contextual adaptation, that the framework combines complementary mechanisms and is supported by extensive ablations, and that the evaluation spans multiple languages, independent datasets and human assessments. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We believe the remaining concerns can be addressed, and we provide detailed responses below.


1. Limited methodological novelty. The main components, including multi-agent translation, persistent memory, dynamic routing, and prompt optimization, have been studied previously. The specific novelty of their integration and test-time evolution needs clearer justification.
We would like to clarify that the novelty of SMART lies in how and when it evolves at test time within one series, changing both its agent prompts and its multi-agent structure. SMART is a self-evolving multi-agent system for long-form subtitle translation. Prior subtitle translation mainly follows two lines. One adapts a single LLM through subtitle-specific fine-tuning or context-aware prompting. The other uses multi-agent systems such as TransAgent, in which specialized roles generate and refine translations. Self-evolving agents and prompt optimizers can make a system adaptive. However, they typically optimize for future tasks or fixed objectives, rather than adapting during one long task while keeping the behavior acquired earlier.

We designed SMART around three gaps of prior systems on long-form content. First, single-LLM methods mainly reason over local sentences without persistent series-level context, so recurring names and terms drift across episodes. The series memory keeps names, terms, and style across episodes. Second, multi-agent systems often fix their roles and refinement steps in advance, although translation difficulty varies across scenes. The dynamic router selects translators and tools for each sentence. Third, current systems have limited ability to retrieve and use production context, such as genre or historical setting. The tools retrieve this context.

Test-time evolution then changes both the agent prompts and the multi-agent structure within one series, without undoing what was learned earlier. The structure update rewrites the routing table, which sets which translators and tools are active for each category of sentence. It removes an agent from a category where it keeps underperforming, promotes one that scores higher, and reassigns tools (Sec. 3.5 and Appendix B.8). On the first 30% of a series, the prompt rewrite cannot delete a workflow step. The routing-table rewrite must keep at least two agents in every category. The series memory is carried across training epochs, so each update builds on what SMART has learned about the series. On the remaining 70%, the prompts and routing stay frozen while the memory keeps growing. Held-out feedback therefore does not leak into the prompts or routing. The differences from translation agents with fixed workflows, general self-evolving agents, and conventional prompt optimization are also stated in Sec. 2.

2. High computational cost. SMART requires approximately 4,970 LLM calls and 14M input tokens per episode. Its quality gains should be evaluated against baselines under comparable computational budgets. How does SMART compare with existing multi-agent translation systems under matched inference costs or token budgets?
Thank you for raising this concern. At inference, SMART uses fewer calls per episode than TransAgent (4,970 vs. 5,694). TransAgent is the agent baseline with the lowest mean penalty in Tables 3 and 11. In the original submission, we reimplemented it with SMART's backbone, Claude Sonnet 4.6, so both call counts come from the same model. Test-time training adds 6,389 calls per held-out episode, since SMART runs the inference pipeline for three epochs over the 30% training prefix. The table gives the calls per held-out episode.

SystemModel calls per held-out episode
TransAgent5,694
SMART, inference4,970
SMART, test-time training6,389
SMART, total11,359

Following your suggestion, we compared SMART and TransAgent at each other's budget on the six Table 6 directions. SMART-lite scales SMART down to TransAgent's budget, with two translators, one refinement round, and one training epoch. TransAgent++ scales TransAgent up to SMART's budget by adding SMART's memory, tools, and extra refinement rounds, with the same backbone and 30% training prefix. We also built two static baselines with SMART's budget. Best-of-8 samples one translator eight times and adds selection and refinement. Static SMART turns off self-evolution and spends the training budget on extra candidates. All call counts include test-time training. The table reports the six-direction means of Overall SubMQM, where lower is better, and each system's difference to SMART with a 95% confidence interval. The TransAgent and SMART scores are from Tables 3 and 11.

MethodCalls per held-out episodeMean OverallΔ vs. SMART [95% CI]
TransAgent5,6941.00+0.04
+0.03,+0.06+0.03, +0.06

| | SMART-lite | 5,827 | 0.98 | +0.02

+0.01,+0.03+0.01, +0.03

| | TransAgent++ | 10,983 | 0.99 | +0.03

+0.01,+0.04+0.01, +0.04

| | Best-of-8 | 11,521 | 1.03 | +0.07

+0.05,+0.08+0.05, +0.08

| | Static SMART | 11,197 | 1.21 | +0.25

+0.23,+0.27+0.23, +0.27

| | SMART | 11,359 | 0.96 | – |

At TransAgent's budget, SMART-lite had a lower penalty than TransAgent (0.98 vs. 1.00). At SMART's budget, TransAgent++ reached 0.99, Best-of-8 1.03, and Static SMART 1.21, compared with 0.96 for SMART. SMART therefore had a lower penalty than TransAgent at both budgets, and the lowest penalty of all these systems. We have revised Appendix F.6 to state the cost breakdown and added these comparisons as Table 21.

3. Insufficient validation of SubMQM. Although robustness across two LLM judges is examined, the agreement between SubMQM and human MQM annotations is not established. This is important because the main improvements are relatively modest. How well do SubMQM scores and error classifications correlate with human MQM judgments?
Thank you for raising this concern. The blind human study of Sec. 5.5 in the original submission compared final outputs and did not collect error annotations. Following your suggestion, we ran a human MQM study in the en→zh, zh→en, en→ko, and ko→en directions to measure SubMQM's agreement with human MQM. It covers the outputs of Gemma 3 4B, DeepSeek-V3.2, Claude Sonnet 4.6, GPT-5.5, TransAgent, and SMART on 100 held-out windows per direction (400 in total). This gives 2,400 items annotated with our 19-type rubric (Table 10). Chinese and Korean annotators labeled the en→zh and en→ko items. Bilingual annotators labeled the zh→en and ko→en items. SubMQM and AutoMQM used the paper's default SubMQM judge, Claude Sonnet 4.6. Pairwise accuracy is the share of system pairs that a metric orders like the annotators. The 60 pairs are all pairs of the six systems in each direction. The table reports the agreement of each metric with the human labels on all 2,400 items.

MetricSegment-level τSystem-level rPairwise accuracy
SubMQM0.420.9491.7%
AutoMQM0.350.9186.7%
COMET-220.310.8683.3%
spBLEU0.170.7173.3%

For the error classifications, we compared each metric with a human ceiling. We double-annotated 20.0% of the items (480) to obtain this ceiling. On these items, we compared the second annotator and each metric with the first annotator, by segment-level τ and by the dimension macro-F1 of the error classification.

MetricSegment-level τDimension macro-F1
Second annotator0.500.72
SubMQM0.410.64
AutoMQM0.340.57
COMET-220.28–
spBLEU0.18–

Among the automatic metrics, SubMQM agreed best with the human labels in every column of both tables. For the scores, its segment-level τ was 0.42, compared with 0.35 for AutoMQM. For the error classifications, its dimension macro-F1 was 0.64, compared with 0.57 for AutoMQM and 0.72 for the second annotator.

We also checked them on these human labels. In the blind human study of Sec. 5.5, annotators ranked SMART first on all four criteria. A paired bootstrap test of SMART against TransAgent on the human MQM labels found a lower penalty for SMART in all four directions. The pooled difference was significant (95% CI [0.02, 0.16], p=0.013). We have added the study and the test to Appendix F.10 (Table 24), with a pointer in Sec. 4.2.

4. Limited validation of the incremental contribution of self-evolution. The ablation study demonstrates its benefit, but its improvement is smaller than those of memory and multi-agent translation. Further comparisons with existing prompt optimization methods would help establish its distinct contribution. What distinguishes the proposed self-evolution mechanism from existing iterative prompt optimization approaches, and what additional benefits does it provide under comparable computational conditions?
Thank you for raising this concern. SMART's self-evolution differs from iterative prompt optimizers in what it updates and when. Optimizers such as APO, TextGrad, REVOLVE, SIPDO, and GEPA refine instructions from natural-language feedback, usually toward one fixed task or objective. In SMART, the two update operators are constrained rewrites of the agent prompts and of a finite routing table. They run once per training epoch on aggregated feedback rather than after every sentence, act only on the first 30% of a series, and build on the series memory carried across epochs.

In Table 6, removing self-evolution increased the penalty in all six directions, by +0.32 to +0.38, with a mean of +0.36, while memory and multi-agent translation stayed in place. Following your suggestion, we compared SMART's update operators with these optimizers under the same conditions. On the six Table 6 directions, we replaced the two operators with APE, OPRO, TextGrad, and GEPA in its DSPy implementation. All runs used the same pipeline, memory, tools, critiques, and training prefix, while each optimizer added its own calls. We also ran SMART with the prompt rewrite only, without the routing-table rewrite. The regression rate is the share of held-out episodes on which a method is worse than SMART with both update operators disabled. The table reports the means over the six directions.

MethodOptimizer calls per seriesMean OverallRegression rate
APE2,3261.1813.1%
OPRO1,9631.1410.6%
TextGrad1,8481.0814.2%
GEPA1,2371.057.4%
SMART, prompt rewrite only≤151.014.9%
SMART≤180.963.8%

SMART had the lowest penalty on average and in each of the six directions (per-direction values in Table 23). Among the existing optimizers, GEPA came closest (1.05 vs. 0.96). SMART with the prompt rewrite only also had a lower penalty than GEPA (1.01 vs. 1.05) with fewer optimizer calls. SMART regressed on the fewest held-out episodes (3.8%), compared with 7.4% for GEPA and 14.2% for TextGrad. With fewer optimizer calls, SMART's constrained operators therefore gave both a lower penalty and fewer held-out regressions than the existing optimizers. We have added this comparison to a new Appendix F.9 (Table 23).

Reviewer_SPS7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SMART addresses long-form subtitle translation through persistent series memory, specialized translators, tools, and a judge–refiner loop. It uses the first 30% of each series to revise prompts and routing, then freezes the configuration to translate the remainder without updating model weights. The authors introduce Subtitle Arena and SubMQM, reporting a 6.9% reduction in average penalty over

TransAgent across 15 English-to-locale directions, alongside MuSC, temporal generalization, multimodal, and human-ranking experiments. The results support the complete system’s practical value, but the credibility of its optimization signal and the independent contribution of self-evolution remain unclear.

Critical strengths
  1. The paper targets a meaningful failure mode. Recurring names, domain terminology, and pragmatic interpretation require persistent context. The memory and tool interfaces are concrete, and the benchmark’s series structure and timestamps fit the intended task.

  2. The evaluation has useful breadth. Agent baselines share the same backbone, with additional checks using weaker backbones, an alternative evaluator, newly released series, and blind human rankings. These results suggest that the gains extend beyond backbone strength.

Critical weaknesses
  1. The optimization signal is uncalibrated. Training roles and evaluation use the same default model, rewrites lack independent quality validation, and existing translations become consistency references. Evaluator substitution and short-passage rankings do not adequately rule out judge preferences or reinforcement of incorrect translations.
  2. The mechanism attribution is weak. SMART uses approximately 4,970 model calls per episode but lacks strong static baselines with matched budgets, memory, and tools. MoA ablations also reduce candidate counts. The ablation discussion further confuses the contributions of memory and self-evolution, weakening its diagnostic value.
Questions for authors
  1. How well do training judge scores correlate with independent human error annotations? Does SMART retain its advantage when TransAgent and DRT are also rescored by the alternative evaluator?

  2. Under matched total budgets, adaptation prefixes, memory, and retrieval access, how much does SMART outperform a strong static pipeline or best-of-N with refinement? Do the reported costs include adaptation?

  3. Does the variant without self-evolution retain three passes over the training prefix and equivalent memory? Why does the ablation discussion report a mean memory effect of 0.78 when the corresponding values indicate approximately 0.36?

  4. Can you report entity and terminology correctness, consistency, and erroneous-memory correction rates on held-out episodes, stratified by cross-episode distance, to directly verify the claimed long-range benefits?

6 comments

We thank the reviewer for the constructive and precise comments. We appreciate the recognition that the paper addresses a meaningful failure mode with concrete memory and tool interfaces, as well as the acknowledgment that the evaluation is broad and indicates gains beyond backbone strength. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of the raised weaknesses and questions below.


1. The optimization signal is uncalibrated. How well do training judge scores correlate with independent human error annotations?
Thank you for raising this concern. We calibrated SMART's judge, whose scores drive test-time training, against independent human annotations. In a blind audit on 600 held-out sentences where the refiner changed the selected candidate, annotators chose the best MoA candidate and labeled every candidate with human MQM. The table compares a random candidate, the judge's top candidate, and the annotators' choice.

CandidateMatch with annotatorsHuman MQM
Random41.3%1.46
Judge78.2%1.07
Annotators' choice–0.94

The judge's top candidate matched the annotators' choice in 78.2% of the sentences, compared with 41.3% for a random choice. Its human MQM penalty was 1.07, compared with 1.46 for a random candidate and 0.94 for the annotators' choice. Within each sentence, the judge's scores also ordered the candidates like their human MQM penalties, with a mean Kendall τ of 0.47. We have added this audit to Appendix F.11 (Table 25).

2. Training roles and evaluation use the same default model.
To test whether using the same model for SMART's judge and for the evaluator favors SMART, we re-ran test-time training in four directions with GPT-5.5 as SMART's judge. We then scored the outputs with both the default evaluator, Claude Sonnet 4.6, and GPT-5.5. The table reports Overall SubMQM per direction and the mean over the four directions, where lower is better. TransAgent has no test-time training, so only its evaluator changes.

SMART's judgeEvaluatoren→zhen→deen→koen→itMean
Claude Sonnet 4.6 (default)Claude Sonnet 4.60.480.870.921.130.850
Claude Sonnet 4.6 (default)GPT-5.50.470.870.911.130.845
GPT-5.5Claude Sonnet 4.60.490.870.921.140.855
GPT-5.5GPT-5.50.470.860.911.120.840
TransAgent (no test-time training)Claude Sonnet 4.60.520.950.971.170.903
TransAgent (no test-time training)GPT-5.50.520.940.961.150.893

With GPT-5.5 as SMART's judge, SMART's margin over TransAgent was 0.053 when the evaluator was also GPT-5.5, and 0.048 when it was Claude Sonnet 4.6. Self-preference therefore accounts for less than 10% of SMART's margin. We have added this control to Appendix F.11 (Table 27) and revised Sec. 5.3 to report it.

3. Rewrites lack independent quality validation.
SMART rewrites translations during refinement, and its prompts and routing table during self-evolution. We validated both rewrites on held-out data. For refinement, annotators labeled the judge-selected candidate and the refined output with human MQM on 600 held-out sentences where the refiner changed the selected candidate. The table reports how refinement changed the human MQM penalty.

Refinement outcomeShare of the 600 sentences
Improved37.7%
Unchanged56.0%
Degraded6.3%

Refinement lowered the penalty for 37.7% of the sentences and raised it for 6.3%. For self-evolution, a rewrite is adopted when it meets the structural constraints. The critiques behind a rewrite name failure modes and give no corrected outputs. A rewrite therefore cannot carry a judge-preferred translation into later sentences. Held-out feedback never enters the rewrites. On the held-out episodes of the six Table 6 directions, the evolved system was worse than the same system with its update operators disabled on 3.8% of the episodes. We have added the refinement audit to Appendix F.11 (Table 25) and the held-out comparison to Appendix F.9 (Table 23).

4. Existing translations become consistency references.
The consistency references are SMART's own earlier renderings in series memory. Memory counts how often each stored rendering occurs. The judge then takes the established rendering as the reference for the rest of the series. To check whether wrong renderings are reinforced, we audited the memory in two steps. A locale annotator first checked 600 sampled terminology and character entries. We then audited the memory entries of all tracked entities and terms in the held-out episodes, which recur in 41,730 later occurrences. The table reports both audits.

Memory auditResult
Sampled entries that are correct (600 entries)95.2%
All tracked entries that are wrong4.7%
Later repetitions per wrong entry, on average4.5
Later occurrences that repeat a wrong entry3.1%
Incorrect later occurrences, SMART w/o Memory22.2%
Incorrect later occurrences, SMART9.5%

Of the sampled entries, 95.2% were correct, and 4.7% of all tracked entries were wrong. A wrong entry was repeated in 4.5 later occurrences on average before it was corrected or the series ended. These repetitions make up 3.1% of all later occurrences. Without memory, SMART rendered 22.2% of the same occurrences incorrectly, compared with 9.5% with memory, repetitions included. We have added the sampled audit to Appendix F.11 (Table 25) and the full audit to Appendix F.8.

5. Does SMART retain its advantage when TransAgent and DRT are also rescored by the alternative evaluator?
Yes, SMART keeps the lowest penalty when TransAgent and DRT are also rescored with the alternative evaluator. In the original submission, the evaluator substitution rescored SMART's outputs but not the baselines'. Following your suggestion, we also rescored TransAgent and DRT with GPT-5.5 on all 15 English-to-locale directions. The table reports the 15-direction means of Overall SubMQM, where lower is better.

MethodClaude Sonnet 4.6 judgeGPT-5.5 judge
DRT1.301.25
TransAgent1.201.14
SMART1.111.06

SMART kept the lowest penalty under both judges, and in 15/15 directions under GPT-5.5. The human study of Sec. 5.5 ranked short passages and did not label errors. To test judge preferences against human error annotations, we ran a human MQM study in four directions, namely en→zh, zh→en, en→ko, and ko→en. We sampled 100 held-out five-sentence windows per direction (400 in total) and collected the outputs of six systems, giving 2,400 items. The systems were Gemma 3 4B, DeepSeek-V3.2, Claude Sonnet 4.6, GPT-5.5, TransAgent, and SMART. Chinese and Korean annotators labeled the en→zh and en→ko items. Bilingual annotators labeled the zh→en and ko→en items. All annotators used our 19-type rubric (Table 10). SubMQM and AutoMQM used the default judge, Claude Sonnet 4.6. Pairwise accuracy is the share of system pairs that a metric orders like the annotators. The 60 pairs are all pairs of the six systems in each direction. The table reports the agreement of each metric with the human labels on all 2,400 items.

MetricSegment-level τSystem-level rPairwise accuracy
SubMQM0.420.9491.7%
AutoMQM0.350.9186.7%
COMET-220.310.8683.3%
spBLEU0.170.7173.3%

We double-annotated 20.0% of the items (480) to obtain a human ceiling. On these items, we compared the second annotator and each metric with the first annotator.

MetricSegment-level τ
Second annotator0.50
SubMQM0.41
AutoMQM0.34
COMET-220.28
spBLEU0.18

Among the automatic metrics, SubMQM agreed best with the human labels in both tables. AutoMQM came closest, with a segment-level τ of 0.35, compared with 0.42 for SubMQM. SubMQM ordered 55 of the 60 system pairs like the annotators. On the double-annotated items, its τ of 0.41 was 82% of the second annotator's 0.50. With GPT-5.5 as the judge, its segment-level τ was 0.40 and its system-level r 0.93. On the human labels themselves, SMART had a lower penalty than TransAgent in all four directions. A paired bootstrap test found the pooled difference significant (95% CI [0.02, 0.16], p=0.013). For the reinforcement of incorrect translations, our full memory audit found that repeated wrong entries make up 3.1% of later occurrences. We have added the study and the test to Appendix F.10 (Table 24), with pointers in Sec. 4.2 and Appendix E.

6. The mechanism attribution is weak. SMART uses approximately 4,970 model calls per episode but lacks strong static baselines with matched budgets, memory, and tools. Under matched total budgets, adaptation prefixes, memory, and retrieval access, how much does SMART outperform a strong static pipeline or best-of-N with refinement?
Thank you for suggesting these comparisons. On the six Table 6 directions, we built three systems at SMART's budget and one at TransAgent's budget. All of them use SMART's backbone, memory, tools including contextual retrieval, and 30% training prefix. Best-of-8 samples one translator eight times and adds candidate selection and refinement. Static SMART turns off self-evolution and spends the test-time training budget on extra candidates. TransAgent++ adds SMART's memory, tools, and extra refinement rounds to TransAgent. SMART-lite scales SMART down to TransAgent's budget, with two translators, one refinement round, and one training epoch. All call counts include test-time training. The table reports the six-direction means and each system's difference to SMART, with a 95% confidence interval from a paired bootstrap over held-out episodes (10,000 resamples). The TransAgent and SMART scores are from Tables 3 and 11.

MethodCalls per held-out episodeMean OverallΔ vs. SMART [95% CI]
TransAgent5,6941.00+0.04
+0.03,+0.06+0.03, +0.06

| | SMART-lite | 5,827 | 0.98 | +0.02

+0.01,+0.03+0.01, +0.03

| | TransAgent++ | 10,983 | 0.99 | +0.03

+0.01,+0.04+0.01, +0.04

| | Best-of-8 | 11,521 | 1.03 | +0.07

+0.05,+0.08+0.05, +0.08

| | Static SMART | 11,197 | 1.21 | +0.25

+0.23,+0.27+0.23, +0.27

| | SMART | 11,359 | 0.96 | – |

At SMART's budget, Best-of-8 reached 1.03, Static SMART 1.21, and TransAgent++ 0.99, compared with 0.96 for SMART. All five intervals exclude 0, and SMART had the lowest penalty in each of the six directions. At TransAgent's budget, SMART-lite reached 0.98, compared with 1.00 for TransAgent. These results suggest that SMART's gain does not come from more compute alone. We have added this comparison, with per-direction values, to Appendix F.6 (Table 21).

7. MoA ablations also reduce candidate counts.
Thank you for pointing this out. In Table 6, w/o MoA and w/ Combined Prompt produce one candidate per sentence, while the full system produces 2.6 on average. Their increases therefore combine the loss of specialist diversity with fewer candidates. To separate the two, we re-ran both variants with as many candidates as the full system. For each sentence, we sampled the single translator once for each translator that the router selects in the full system. The table reports the mean increase in Overall SubMQM over the six directions.

SettingSingle candidateSame candidate count
w/o MoA+0.62+0.42
w/ Combined Prompt+0.41+0.29

With the same number of candidates, removing MoA increased the mean penalty by 0.42. Replacing MoA with the combined prompt increased it by 0.29. Most of each effect therefore comes from specialist diversity. We have added both variants to Table 6 and their definitions to Appendix F.5.

8. The ablation discussion further confuses the contributions of memory and self-evolution, weakening its diagnostic value. Why does the ablation discussion report a mean memory effect of 0.78 when the corresponding values indicate approximately 0.36?
Sorry for the confusion. The 0.78 and the 0.36 are the mean increases of two different components, so they are not the same quantity. In the original submission, Table 6 sat beside the text of Sec. 5.4, and its w/o Self-Evolution row was printed next to the sentence with the mean memory effect of 0.78, which may have made the two look connected. The table gives both increases with their ranges over the six directions.

Removed componentMean increaseRange over the six directions
Series-level memory (w/o Memory)+0.78+0.54 to +0.95
Self-evolution (w/o Self-Evolution)+0.36+0.32 to +0.38

The 0.78 is the mean increase when series-level memory is removed. The 0.36 is the mean increase when self-evolution is removed. We have rewritten the ablation paragraph of Sec. 5.4 so that each number sits next to its component.

Comments on the paper

1 comment

Dear reviewers,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Thank you for your contribution!

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.