- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TimeVista, a framework and benchmark for evaluating time-series forecasts using VLMs as judges. Forecasts are rendered jointly with history and ground truth, then assessed using explicit rubrics at a Micro-Level covering trend, motif, phase, and magnitude, and at a Macro-Level capturing domain-dependent operational requirements. TimeVista contains 5,563 series, while Meta-TimeVista contains 1,025 forecast instances independently rated by five time-series researchers.
The meta-evaluation is fairly extensive. Five annotators provide ratings under the same five-point criteria used by VLM judges, yielding mean Krippendorff’s . The authors additionally conduct a rubric-free pairwise preference study, where rubric-derived human scores agree with intuitive preferences in 88.29% of comparisons. Six VLMs are evaluated, with the strongest judge attaining Spearman correlations between roughly 0.83 and 0.91 across the five dimensions.
The paper further compares VLM judging against pointwise, alignment-based, structure-aware, and deterministic rubric-based evaluators. The VLM evaluator performs best across all reported dimensions, including a substantial advantage for the domain-aware General score. Ablations examine numerical versus visual inputs, rubric perturbations, rendering changes, and repeated-call consistency.
Critical strengths
-
The paper identifies an important limitation of standard forecast metrics: similar numerical errors can conceal substantially different trend, phase, motif, magnitude, or domain-level behavior.
-
Six VLM judges, multiple numerical and structure-aware baselines, deterministic rubric rules, five human annotators, and inter-annotator agreement provide a strong empirical foundation.
-
The comparison is not limited to MSE or MAE. Rubric-based deterministic rules are substantially stronger, yet VLM judging still improves human alignment across all dimensions.
-
Visual input improves token efficiency, while rubric simplification substantially hurts alignment and deliberately incorrect rules reverse it, revealing both strengths and important failure modes.
-
Aspect ratio, resolution, padding, markers, axes, and legend perturbations have relatively modest effects, reducing concern that results depend strongly on one plotting template.
Critical weaknesses
-
Experts and VLMs primarily score using the same authored rubrics, so high correlation demonstrates shared rubric interpretation more directly than fully independent human preference alignment.
-
The 88.29% rubric-free pairwise agreement is important, but its sampling scale and disagreement composition need clearer main-text reporting to establish independent human validity.
-
Macro-Level rubrics encode operational preferences across healthcare, finance, energy, transportation, and other domains, but annotators are described primarily as time-series researchers rather than domain practitioners.
-
Incorrect domain rules drive General-score correlation from to , showing that authoritative-looking but erroneous criteria can systematically redirect evaluation.
-
Most detailed ablations use Gemini-3.1-Pro, while judge performance varies materially across models, so evaluator robustness across weaker or structurally different VLMs is less established.
Questions for authors
My main question is about independent validation of the Macro-Level rubrics. Were the domain-specific requirements reviewed by practitioners with expertise in the corresponding application areas, particularly where the rubric encodes asymmetric preferences such as one error direction being safer than another? If not, how should readers interpret “domain-aware” human alignment versus agreement with a researcher-authored operationalization?
Second, could the authors report the size and sampling procedure of the rubric-free pairwise preference experiment more prominently, including the fraction of genuine ordering reversals rather than disagreements caused by rubric ties? This result seems central because it provides the cleanest evidence that the rubrics themselves reflect human preferences rather than merely inducing agreement.
Finally, have the authors repeated the rubric and rendering ablations with at least one substantially different VLM family? Given the variation across judges in Table 1, evidence that the main robustness conclusions survive across models would materially strengthen my confidence in TimeVista as an evaluator-independent benchmark.
4 comments
As clarified in Appendix B of the revised manuscript, time-series researchers developed the Macro-Level rubrics, and practitioners in the corresponding application domains reviewed them. The Micro-Level rubrics assess trend, motif, phase, and magnitude separately. Each Macro-Level rubric specifies which discrepancies along these dimensions matter most for a given planning objective. In TSFM evaluation, a higher General score therefore indicates that a forecast better meets that objective. For example, the depth of a speed trough indicates how severe congestion is, while its timing indicates when it occurs. A predicted trough that is too shallow understates congestion severity, so the congestion-warning rubric penalizes errors in that direction more heavily. Users can then judge whether underestimating congestion should carry a greater penalty in their own application.
In meta-evaluation, domain-aware alignment measures how closely VLM ratings agree with the researchers’ ratings under the stated criteria. In the incorrect-rule experiment, we held the forecast plots and the original human ratings fixed and changed only the domain rules given to Gemini-3.1-pro. Its Spearman correlation with human General ratings fell from 0.865 to −0.508 (). This reversal supports the judge’s ability to follow the supplied rubric even when its instructions conflict with the original planning objective.
W5 & Q3: Robustness Across VLM Families
Using a fixed evaluator gives models a common basis for comparison. FID uses the same Inception-v3 feature extractor across generative models [1]. In language model evaluation, MT-Bench uses GPT-4 to judge responses across models [2], and AlpacaEval 2.0 uses a GPT-4 Turbo-based evaluator [3]. Following the same principle, we use a single judge to evaluate all forecasting models. We selected Gemini-3.1-pro because, among the six VLMs we evaluated, it achieved the highest rank correlation with human ratings on four of the five dimensions ().
To test whether the robustness observed for Gemini-3.1-pro extends to another VLM family, we repeated the rendering and rubric paraphrasing experiments with Qwen3.6-Plus on the same 480 forecasts. We tested the 11 rendering changes in that preserve the identities of the forecast and ground-truth curves. Across these settings, Spearman’s ρ with human Magnitude ratings ranged from 0.895 to 0.917 for Gemini-3.1-pro (default 0.907) and from 0.850 to 0.888 for Qwen3.6-Plus (default 0.862). With the paraphrased rubric, ρ was 0.901 for Gemini-3.1-pro and 0.850 for Qwen3.6-Plus. Across the tested changes in rendering and rubric wording, each judge’s rank correlation with human Magnitude ratings remained close to its own baseline.
| Factor | Condition | Gemini-3.1-pro (ρ / τ-b) | Qwen3.6-Plus (ρ / τ-b) |
|---|---|---|---|
| Baseline | Default | 0.907 / 0.849 | 0.862 / 0.794 |
| Aspect ratio | 6:5 | 0.896 / 0.836 | 0.857 / 0.783 |
| 18:5 | 0.903 / 0.846 | 0.886 / 0.823 | |
| 24:5 | 0.907 / 0.851 | 0.887 / 0.823 | |
| Vertical padding | 50% | 0.917 / 0.862 | 0.876 / 0.807 |
| 100% | 0.912 / 0.855 | 0.887 / 0.815 | |
| Plot annotations | Axes removed | 0.906 / 0.850 | 0.880 / 0.811 |
| Legend removed | 0.895 / 0.836 | 0.850 / 0.782 | |
| Resolution | 50 DPI | 0.902 / 0.843 | 0.861 / 0.784 |
| 300 DPI | 0.896 / 0.835 | 0.882 / 0.817 | |
| Markers | None | 0.908 / 0.851 | 0.888 / 0.823 |
| Every 3 steps | 0.906 / 0.846 | 0.886 / 0.820 | |
| Rubric wording | Paraphrased | 0.901 / 0.840 | 0.850 / 0.779 |
The default plot has a 12:5 aspect ratio and 5% vertical padding. Each visual perturbation changes one rendering parameter at a time.
References
[1] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30. Paper
[2] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. Paper
[3] Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. Conference on Language Modeling. Paper
Thanks for the clarification. I have updated my rating accordingly.
Our rubrics specify which forecast properties to assess and how to score them. To evaluate whether annotators could apply the criteria consistently, we measured agreement among five researchers who independently scored the forecasts using the rubrics. Mean Krippendorff’s α across the five dimensions was 0.718 (Table 6 of submission), supporting the consistency of rubric-guided scoring.
In a separate task, annotators compared forecast pairs without access to the rubrics, the existing scores, or the rankings derived from them. Their modal preference agreed with the rubric-derived ordering in 1,320 of 1,495 valid comparisons (88.29%, Table 7 of submission). Only 0.80% of comparisons yielded opposite strict preferences (Table 8 of submission). These results indicate that the rubric-derived rankings generally reflected the preferences these participants expressed in direct comparisons.
Finally, we assessed how closely each VLM reproduced the annotators’ rubric-guided rankings. For Gemini-3.1-pro, Spearman’s ρ with the modal human ratings ranged from 0.831 to 0.907 across the five dimensions (Table 1 of submission). This agreement with the annotators’ rankings supports Gemini-3.1-pro’s ability to follow the rubric instructions when scoring forecasts.
These findings support the validity and reliability of our VLM-as-a-judge approach for ranking forecasts under the specified criteria.
Following your suggestion, we revised Section 4.2 to describe how we sampled forecast pairs and to distinguish disagreements involving ties from genuine ranking reversals. We paired forecasts that shared the same historical window and ground truth, yielding 744 candidate pairs for each of the four Micro-Level dimensions and 854 for General. Within each dimension, we grouped the pairs by their rubric-based ordering (A preferred, tied, or B preferred) and randomly sampled 100 pairs per group without replacement, for a total of 300 comparisons per dimension and 1,500 overall.
Annotators compared the two forecasts directly without being exposed to the rubrics, the existing scores, or the orderings derived from those scores. They could select either forecast or indicate a tie. Of the 1,500 comparisons, 1,495 had votes from all five annotators and a unique modal preference. In this stratified sample, the rubric-based ordering agreed with the annotators’ modal preference in 1,320 of 1,495 comparisons (88.29%, Table 7 of submission).
Of the 175 disagreements, 121 occurred when the rubric scores were tied but the annotators preferred one forecast, and 42 occurred when the annotators’ modal preference was a tie but the rubric scores favored one forecast. In only 12 cases did the rubric scores favor one forecast and the annotators prefer the other. These genuine reversals accounted for 0.80% of all valid comparisons (12/1,495, Table 8 of submission). Most disagreements thus involved a tie in one of the two assessments.
Pairwise agreement and ranking reversals