- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SMART, a multi-agent system for long-form subtitle translation that combines series-level memory, dynamic routing, tool use, and test-time updates to prompts and routing policies. It also introduces Subtitle Arena, a benchmark constructed from OpenSubtitles2024.
Critical strengths
- SMART achieves the lowest Overall SubMQM penalty across all 15 English-to-target-language directions in Subtitle Arena, and the best model results on all four MuSC directions.
- In a small-scale human evaluation, SMART ranks first in fidelity, consistency, language quality, and subtitle quality.
Critical weaknesses
- The methodological novelty appears limited: SMART primarily integrates and adapts existing techniques to long-form subtitle translation.
- Although the paper presents Subtitle Arena as a benchmark, it is derived from OpenSubtitles2024. Its contribution appears to lie primarily in reconstructing and reorganizing existing subtitle alignments at the episode and series levels
- The Subtitle Arena experiments do not report common metrics such as spBLEU or COMET.
Questions for authors
- The main experiments use a newly constructed benchmark and a custom evaluation metric, but the paper does not state whether the code and data will be made publicly available.
- The baselines in Table 3 are outdated; please include Gemma 4 31B and DeepSeek V4.1.
- Table 4 does not report Claude Sonnet 4.6 performance on MuSC, despite its use as the paper’s default API model.
6 comments
3. Its contribution appears to lie primarily in reconstructing and reorganizing existing subtitle alignments at the episode and series levels.
The episode- and series-level reorganization is what the evaluation of SMART needs, because it makes long-range consistency measurable. Among the resources in Table 9, Subtitle Arena is the only one with series context. MuST-Cinema covers TED talks for speech-to-subtitle translation. MuSC, the resource closest to our setting, consists of multi-line utterances evaluated at the segment level. These resources are not designed to fully capture three properties of long-form subtitles. These are the use of long-range context, the consistency of names, terms, and style across a series, and respect for display constraints. Subtitle Arena is built for these three properties, and SubMQM adds an evaluation with 19 error types, which we checked against human MQM.
For example, we tracked recurring entities and terms on the held-out 70% of each series in the six directions of Table 6. An occurrence is consistent if it matches the system's first rendering. The cross-episode distance d is the number of episodes since the first occurrence. The table reports the consistency in percent.
| Consistency | d = 0 | d = 1–2 | d = 3–5 | d = 6–10 | d > 10 |
|---|---|---|---|---|---|
| TransAgent | 92.6 | 77.8 | 71.4 | 67.9 | 63.2 |
| SMART w/o Memory | 94.0 | 81.2 | 75.6 | 71.1 | 68.3 |
| SMART | 96.3 | 94.8 | 93.9 | 93.1 | 91.7 |
TransAgent's consistency fell from 92.6% within the episode of the first occurrence to 63.2% more than ten episodes later, compared with 96.3% to 91.7% for SMART. Without the series structure, this drop could not be measured. The evaluation protocol is also checked against human MQM. In a study of six systems in four directions, SubMQM agreed with the human annotations better than any other automatic metric, with a segment-level τ of 0.42 against 0.35 for AutoMQM. We have added the consistency analysis to the new Appendix F.8 (Table 22) and the human-MQM study to Appendix F.10 (Table 24).
4. The Subtitle Arena experiments do not report common metrics such as spBLEU or COMET.
Thank you for pointing this out. The only references in Subtitle Arena are the Online community subtitles. As stated in Sec. 5.1 of the original submission, these subtitles are collected in the wild and vary in quality. We therefore do not treat them as a controlled human upper bound. Reference-based metrics such as spBLEU and COMET would score every system against these noisy references. The original submission therefore used SubMQM, an LLM-as-a-judge evaluation with typed errors (Sec. 4.2). Following your suggestion, we additionally computed spBLEU and COMET-22 for five systems on all 15 English-to-locale directions, with the Online subtitles as the reference. The table reports the averages over the 15 directions, where higher is better.
| Method | spBLEU | COMET-22 |
|---|---|---|
| Claude Sonnet 4.6 | 23.9 | 80.4 |
| GPT-5.5 | 24.7 | 80.8 |
| DRT | 24.6 | 81.0 |
| TransAgent | 25.3 | 81.6 |
| SMART | 26.0 | 81.9 |
SMART achieves the best average on both metrics, with 26.0 spBLEU and 81.9 COMET-22, compared with 25.3 and 81.6 for TransAgent, the strongest baseline. It is the best in 13 of the 15 directions on spBLEU and the best or tied for best in 14 of the 15 directions on COMET-22. We keep SubMQM as the primary metric because it correlates better with human MQM. The table reports the agreement of each metric with the human labels in our human MQM study of six systems in four directions. Pairwise accuracy is the share of system pairs that a metric orders like the annotators.
| Metric | Segment-level τ | System-level r | Pairwise accuracy |
|---|---|---|---|
| SubMQM | 0.42 | 0.94 | 91.7% |
| AutoMQM | 0.35 | 0.91 | 86.7% |
| COMET-22 | 0.31 | 0.86 | 83.3% |
| spBLEU | 0.17 | 0.71 | 73.3% |
SubMQM's segment-level τ was 0.42, compared with 0.31 for COMET-22 and 0.17 for spBLEU. We have added both metrics, with per-direction values, to the new Appendix F.12 (Table 28).
5. The main experiments use a newly constructed benchmark and a custom evaluation metric, but the paper does not state whether the code and data will be made publicly available.
Yes, the SMART and SubMQM code is available, and Subtitle Arena will be released as a reconstruction recipe. The SMART implementation and the SubMQM evaluator are in the anonymous repository, which Appendix I.1 of the original submission links to. The implementation covers all 30 directions and includes the agent prompts, initial routing table, tool interfaces, and series memory. The evaluator includes the 19-type rubric with its aggregation weights, the judge prompt, and the alignment and scoring code. The repository also contains the test-time training and inference scripts and one synthetic example SRT file.
We are adding the Subtitle Arena construction pipeline with the episode keys and splits, the SubMQM prompts, and the evolved SMART configurations. We will post a comment here once the upload is complete. As stated in Appendix I.2, we do not redistribute the subtitle text. Once the pipeline is uploaded, users can download OpenSubtitles2024 themselves and rebuild the episode-level SRT streams and the per-series splits.
The baselines in Table 3 are outdated; please include Gemma 4 31B and DeepSeek V4.1.
Thank you for suggesting these baselines. The single-call LLMs in Table 3 of the original submission included GPT-5.5 and Claude Opus 4.8. We evaluated Gemma 4 31B and DeepSeek V4.1 under the same setting as the other single-call LLMs in Table 3. The table reports the mean Overall SubMQM over the 15 English-to-locale directions, where lower is better. The GPT-5.5 and SMART means are from Sec. 5.2.
| Method | Mean Overall |
|---|---|
| GPT-5.5 | 1.44 |
| Gemma 4 31B | 1.82 |
| DeepSeek V4.1 | 1.46 |
| SMART | 1.11 |
DeepSeek V4.1 performs comparably to GPT-5.5 (1.46 vs. 1.44), while Gemma 4 31B reaches 1.82. SMART keeps the lowest mean at 1.11. It also has the lowest Overall penalty in all 15 English-to-locale directions and in the four reverse directions of Table 3. We have added both models to Sec. 5.1 and to Tables 3, 11, and 12, which report the per-direction values.
6. Table 4 does not report Claude Sonnet 4.6 performance on MuSC, despite its use as the paper's default API model.
Thank you for suggesting this comparison. For Table 4, we followed the original MuSC benchmark (Cui et al., 2026b) and listed the systems it evaluated, which do not include Claude Sonnet 4.6. We evaluated Claude Sonnet 4.6 with the same ICL protocol as the other ICL models in Table 4. The table compares it with SMART on Accuracy, Naturalness, and Vividness, where higher is better. The SMART values are from Table 4 of the original submission.
| Direction | Claude Sonnet 4.6 Accuracy | SMART Accuracy | Claude Sonnet 4.6 Naturalness | SMART Naturalness | Claude Sonnet 4.6 Vividness | SMART Vividness |
|---|---|---|---|---|---|---|
| en→zh | 91.6 | 94.3 | 86.1 | 88.2 | 68.7 | 80.4 |
| ko→zh | 83.2 | 86.8 | 82.0 | 85.9 | 61.9 | 78.4 |
| zh→en | 89.4 | 92.3 | 86.4 | 88.6 | 71.8 | 83.7 |
| zh→th | 90.2 | 94.2 | 85.1 | 87.0 | 70.3 | 80.5 |
SMART achieves higher Accuracy, Naturalness, and Vividness than Claude Sonnet 4.6, its own backbone, in every direction. In every column of Table 4, Claude Sonnet 4.6 scores below the strongest prior system. These results suggest that the MuSC gain comes from our framework rather than from the backbone. We have added Claude Sonnet 4.6 to Table 4.
We thank the reviewer for the constructive comments. We appreciate the recognition that SMART has the lowest Overall SubMQM penalty in all 15 English-to-locale directions of Subtitle Arena and the best model results in all four MuSC directions, as well as the acknowledgment that it ranks first on all four criteria in the human evaluation. Our responses to each weakness and question follow.
1. The methodological novelty appears limited: SMART primarily integrates and adapts existing techniques to long-form subtitle translation.
We would like to clarify that the main methodological novelty of SMART lies in constrained test-time evolution of its prompts and multi-agent structure within one long-form task. The techniques that SMART builds on typically do not target this setting. The closest prior approaches either adapt a single LLM to subtitles or use multi-agent systems. A single LLM can be fine-tuned for subtitles, as in ALPO, or prompted with neighboring dialogue, genre, or plot summaries. In multi-agent systems such as TransAgent, specialized roles generate and refine translations. For long-form subtitles, these approaches fall short in three ways. First, without persistent series-level context, recurring names and terms drift across episodes. Second, with roles and refinement steps fixed in advance, the workflow does not adapt to scene complexity. Third, current systems have limited ability to retrieve and use production context such as genre. Self-evolving agents and prompt optimizers learn from feedback, but they typically optimize for future tasks or a fixed objective. Long-form subtitles instead call for adapting within one series while preserving the behavior acquired earlier.
SMART therefore runs in two stages on each series. During test-time training on the first 30%, judge feedback updates both the agent prompts and the multi-agent structure. The structure update rewrites the routing table, which sets which translators and tools are active for each category of sentence. It removes an agent from a category where it keeps underperforming and reassigns tools (Sec. 3.5 and Appendix B.8). Both updates are constrained rewrites. Each agent keeps its role and workflow, and each routing category keeps at least two agents. During test-time inference on the remaining 70%, the prompts and routing stay frozen. The series memory keeps accumulating names, terms, and style to preserve consistency across episodes. In both stages, the router selects translators and tools for each sentence. SMART's contextual retrieval tools supply production context, such as genre or historical setting. The differences from translation agents, self-evolving agents, and prompt optimizers are also stated in Sec. 2.
2. Although the paper presents Subtitle Arena as a benchmark, it is derived from OpenSubtitles2024.
We would like to clarify that the core of our paper is the method, SMART. As stated in Sec. 1 of the original submission, we built Subtitle Arena to evaluate SMART on long-form series. Its source, OpenSubtitles2024, is a sentence-level parallel corpus in many languages without an evaluation protocol (Table 9). Some of its sentence pairs are noisy or misaligned. We used the IMDb identifier and the season and episode indices of each document as the episode key. We then rebuilt episode-level SRT streams with the original timecodes and discarded empty or one-sided alignments. Finally, we grouped the episodes into 192 series of 2–198 episodes each. We split each series into a 30% prefix for SMART's test-time training and a 70% held-out portion for evaluation.
Beyond OpenSubtitles2024, which contains subtitles released before 2024, the original submission also evaluated SMART on 200 TV series released in 2025–2026. We collected this set separately, and it shares no title with Subtitle Arena. It includes titles released after the reported knowledge cutoff of Claude Sonnet 4.6, SMART's default backbone (Sec. 5.2 and Appendix F.4). The table reports the Overall SubMQM penalty in each direction (lower is better).
SMART had a lower Overall penalty than TransAgent in all six directions, with an average of 0.73 compared with 0.77 (Table 19).