acceptodds
Under review as a conference paper at ICLR 2027

TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting

Abstract

High-quality time series forecasting is pivotal for real-world decision-making. However, traditional numerical metrics often fail to reveal complex temporal patterns and are limited in their ability to align with human intuitive preferences. While the “LLM-as-a-Judge” paradigm has revolutionized text evaluation by providing flexible, human-aligned judgment, its application to time series remains largely unexplored. In this paper, we leverage Vision-Language Models (VLMs) as judges for time series forecasting, harnessing their ability to comprehend time series plots grounded in textual constraints. Specifically, we propose a novel framework integrating Micro- and Macro-Level judgments informed by contextual information to evaluate time series forecasting. To this end, we introduce TimeVista, comprising 5,563 time series samples with evaluation rubrics, and Meta-TimeVista, comprising 1,025 forecast instances annotated by five experts. Meta-evaluation shows that VLM judges possess better alignment with human judgments than point-wise and structure-aware numerical baselines. Further analyses examine the effects of visual rendering and rubric design on VLM judgments. Building upon our benchmark, we comprehensively assess recent Time Series Foundation Models (TSFMs) under the VLM-as-a-Judge paradigm. Our results demonstrate that VLMs serve as robust and interpretable judges, providing a comprehensive, human-aligned standard for evaluating time series models.

Primary area:
datasets and benchmarks
Keywords:
VLM-as-a-judge, Time Series Forecasting, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_LpujOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TimeVista, a framework and benchmark for evaluating time-series forecasts using VLMs as judges. Forecasts are rendered jointly with history and ground truth, then assessed using explicit rubrics at a Micro-Level covering trend, motif, phase, and magnitude, and at a Macro-Level capturing domain-dependent operational requirements. TimeVista contains 5,563 series, while Meta-TimeVista contains 1,025 forecast instances independently rated by five time-series researchers.

The meta-evaluation is fairly extensive. Five annotators provide ratings under the same five-point criteria used by VLM judges, yielding mean Krippendorff’s α=0.718\alpha=0.718. The authors additionally conduct a rubric-free pairwise preference study, where rubric-derived human scores agree with intuitive preferences in 88.29% of comparisons. Six VLMs are evaluated, with the strongest judge attaining Spearman correlations between roughly 0.83 and 0.91 across the five dimensions.

The paper further compares VLM judging against pointwise, alignment-based, structure-aware, and deterministic rubric-based evaluators. The VLM evaluator performs best across all reported dimensions, including a substantial advantage for the domain-aware General score. Ablations examine numerical versus visual inputs, rubric perturbations, rendering changes, and repeated-call consistency.

Critical strengths
  • The paper identifies an important limitation of standard forecast metrics: similar numerical errors can conceal substantially different trend, phase, motif, magnitude, or domain-level behavior.

  • Six VLM judges, multiple numerical and structure-aware baselines, deterministic rubric rules, five human annotators, and inter-annotator agreement provide a strong empirical foundation.

  • The comparison is not limited to MSE or MAE. Rubric-based deterministic rules are substantially stronger, yet VLM judging still improves human alignment across all dimensions.

  • Visual input improves token efficiency, while rubric simplification substantially hurts alignment and deliberately incorrect rules reverse it, revealing both strengths and important failure modes.

  • Aspect ratio, resolution, padding, markers, axes, and legend perturbations have relatively modest effects, reducing concern that results depend strongly on one plotting template.

Critical weaknesses
  • Experts and VLMs primarily score using the same authored rubrics, so high correlation demonstrates shared rubric interpretation more directly than fully independent human preference alignment.

  • The 88.29% rubric-free pairwise agreement is important, but its sampling scale and disagreement composition need clearer main-text reporting to establish independent human validity.

  • Macro-Level rubrics encode operational preferences across healthcare, finance, energy, transportation, and other domains, but annotators are described primarily as time-series researchers rather than domain practitioners.

  • Incorrect domain rules drive General-score correlation from 0.8650.865 to −0.508-0.508, showing that authoritative-looking but erroneous criteria can systematically redirect evaluation.

  • Most detailed ablations use Gemini-3.1-Pro, while judge performance varies materially across models, so evaluator robustness across weaker or structurally different VLMs is less established.

Questions for authors

My main question is about independent validation of the Macro-Level rubrics. Were the domain-specific requirements reviewed by practitioners with expertise in the corresponding application areas, particularly where the rubric encodes asymmetric preferences such as one error direction being safer than another? If not, how should readers interpret “domain-aware” human alignment versus agreement with a researcher-authored operationalization?

Second, could the authors report the size and sampling procedure of the rubric-free pairwise preference experiment more prominently, including the fraction of genuine ordering reversals rather than disagreements caused by rubric ties? This result seems central because it provides the cleanest evidence that the rubrics themselves reflect human preferences rather than merely inducing agreement.

Finally, have the authors repeated the rubric and rendering ablations with at least one substantially different VLM family? Given the variation across judges in Table 1, evidence that the main robustness conclusions survive across models would materially strengthen my confidence in TimeVista as an evaluator-independent benchmark.

4 comments

W1: Human–VLM Preference Alignment

Our rubrics specify which forecast properties to assess and how to score them. To evaluate whether annotators could apply the criteria consistently, we measured agreement among five researchers who independently scored the forecasts using the rubrics. Mean Krippendorff’s α across the five dimensions was 0.718 (Table 6 of submission‾\underline{\text{Table 6 of submission}}), supporting the consistency of rubric-guided scoring.

In a separate task, annotators compared forecast pairs without access to the rubrics, the existing scores, or the rankings derived from them. Their modal preference agreed with the rubric-derived ordering in 1,320 of 1,495 valid comparisons (88.29%, Table 7 of submission‾\underline{\text{Table 7 of submission}}). Only 0.80% of comparisons yielded opposite strict preferences (Table 8 of submission‾\underline{\text{Table 8 of submission}}). These results indicate that the rubric-derived rankings generally reflected the preferences these participants expressed in direct comparisons.

Finally, we assessed how closely each VLM reproduced the annotators’ rubric-guided rankings. For Gemini-3.1-pro, Spearman’s ρ with the modal human ratings ranged from 0.831 to 0.907 across the five dimensions (Table 1 of submission‾\underline{\text{Table 1 of submission}}). This agreement with the annotators’ rankings supports Gemini-3.1-pro’s ability to follow the rubric instructions when scoring forecasts.

These findings support the validity and reliability of our VLM-as-a-judge approach for ranking forecasts under the specified criteria.

W2 & Q2: Pairwise Sampling Procedure and Results

Following your suggestion, we revised Section 4.2 to describe how we sampled forecast pairs and to distinguish disagreements involving ties from genuine ranking reversals. We paired forecasts that shared the same historical window and ground truth, yielding 744 candidate pairs for each of the four Micro-Level dimensions and 854 for General. Within each dimension, we grouped the pairs by their rubric-based ordering (A preferred, tied, or B preferred) and randomly sampled 100 pairs per group without replacement, for a total of 300 comparisons per dimension and 1,500 overall.

Annotators compared the two forecasts directly without being exposed to the rubrics, the existing scores, or the orderings derived from those scores. They could select either forecast or indicate a tie. Of the 1,500 comparisons, 1,495 had votes from all five annotators and a unique modal preference. In this stratified sample, the rubric-based ordering agreed with the annotators’ modal preference in 1,320 of 1,495 comparisons (88.29%, Table 7 of submission‾\underline{\text{Table 7 of submission}}).

Of the 175 disagreements, 121 occurred when the rubric scores were tied but the annotators preferred one forecast, and 42 occurred when the annotators’ modal preference was a tie but the rubric scores favored one forecast. In only 12 cases did the rubric scores favor one forecast and the annotators prefer the other. These genuine reversals accounted for 0.80% of all valid comparisons (12/1,495, Table 8 of submission‾\underline{\text{Table 8 of submission}}). Most disagreements thus involved a tie in one of the two assessments.

Pairwise agreement and ranking reversals

DimensionAgreement (%)Linearly weighted κReversals (%)
Trend88.930.8740.34
Motif90.330.8890.33
Phase94.670.9400.00
Magnitude85.620.8222.01
General81.880.7901.34

As clarified in Appendix B of the revised manuscript, time-series researchers developed the Macro-Level rubrics, and practitioners in the corresponding application domains reviewed them. The Micro-Level rubrics assess trend, motif, phase, and magnitude separately. Each Macro-Level rubric specifies which discrepancies along these dimensions matter most for a given planning objective. In TSFM evaluation, a higher General score therefore indicates that a forecast better meets that objective. For example, the depth of a speed trough indicates how severe congestion is, while its timing indicates when it occurs. A predicted trough that is too shallow understates congestion severity, so the congestion-warning rubric penalizes errors in that direction more heavily. Users can then judge whether underestimating congestion should carry a greater penalty in their own application.

In meta-evaluation, domain-aware alignment measures how closely VLM ratings agree with the researchers’ ratings under the stated criteria. In the incorrect-rule experiment, we held the forecast plots and the original human ratings fixed and changed only the domain rules given to Gemini-3.1-pro. Its Spearman correlation with human General ratings fell from 0.865 to −0.508 (Table 3 of submission‾\underline{\text{Table 3 of submission}}). This reversal supports the judge’s ability to follow the supplied rubric even when its instructions conflict with the original planning objective.

W5 & Q3: Robustness Across VLM Families

Using a fixed evaluator gives models a common basis for comparison. FID uses the same Inception-v3 feature extractor across generative models [1]. In language model evaluation, MT-Bench uses GPT-4 to judge responses across models [2], and AlpacaEval 2.0 uses a GPT-4 Turbo-based evaluator [3]. Following the same principle, we use a single judge to evaluate all forecasting models. We selected Gemini-3.1-pro because, among the six VLMs we evaluated, it achieved the highest rank correlation with human ratings on four of the five dimensions (Table 1 of submission‾\underline{\text{Table 1 of submission}}).

To test whether the robustness observed for Gemini-3.1-pro extends to another VLM family, we repeated the rendering and rubric paraphrasing experiments with Qwen3.6-Plus on the same 480 forecasts. We tested the 11 rendering changes in Table 4 of submission‾\underline{\text{Table 4 of submission}} that preserve the identities of the forecast and ground-truth curves. Across these settings, Spearman’s ρ with human Magnitude ratings ranged from 0.895 to 0.917 for Gemini-3.1-pro (default 0.907) and from 0.850 to 0.888 for Qwen3.6-Plus (default 0.862). With the paraphrased rubric, ρ was 0.901 for Gemini-3.1-pro and 0.850 for Qwen3.6-Plus. Across the tested changes in rendering and rubric wording, each judge’s rank correlation with human Magnitude ratings remained close to its own baseline.

FactorConditionGemini-3.1-pro (ρ / τ-b)Qwen3.6-Plus (ρ / τ-b)
BaselineDefault0.907 / 0.8490.862 / 0.794
Aspect ratio6:50.896 / 0.8360.857 / 0.783
18:50.903 / 0.8460.886 / 0.823
24:50.907 / 0.8510.887 / 0.823
Vertical padding50%0.917 / 0.8620.876 / 0.807
100%0.912 / 0.8550.887 / 0.815
Plot annotationsAxes removed0.906 / 0.8500.880 / 0.811
Legend removed0.895 / 0.8360.850 / 0.782
Resolution50 DPI0.902 / 0.8430.861 / 0.784
300 DPI0.896 / 0.8350.882 / 0.817
MarkersNone0.908 / 0.8510.888 / 0.823
Every 3 steps0.906 / 0.8460.886 / 0.820
Rubric wordingParaphrased0.901 / 0.8400.850 / 0.779

The default plot has a 12:5 aspect ratio and 5% vertical padding. Each visual perturbation changes one rendering parameter at a time.

References

[1] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30. Paper

[2] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. Paper

[3] Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. Conference on Language Modeling. Paper

Thanks for the clarification. I have updated my rating accordingly.

Reviewer_myozOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the use of Vision-Language Models (VLMs) as judges for time-series forecasting, integrating the assessment of temporal structures with domain-specific evaluation criteria. The authors introduce the TimeVista benchmark, along with an expert-annotated meta-evaluation dataset Meta-TimeVista. They compare the performance of six VLM-based judges against numerical evaluation metrics and apply this paradigm to 13 different forecasting models. Final findings demonstrate that, when guided by specific evaluation criteria, vision-based judgments can provide effective feedback that goes beyond traditional forecasting error metrics

Critical strengths
  1. This evaluation design reflects a profound understanding of time-series forecasting and the key elements essential for assessing forecast quality. By distinguishing between trends, motifs, phases, and amplitudes at a micro-level, while incorporating application-specific considerations at a macro-level, this work offers a logically coherent perspective on the evaluation of forecasts.
  2. TimeVista addresses a significant gap in forecasting evaluation: relying solely on numerical accuracy fails to capture a model’s structural fidelity or its alignment with the requirements of practical application. Through an analysis of 13 models, TimeVista demonstrates how the proposed evaluation method reveals model strengths and weaknesses that would otherwise be obscured by conventional numerical metrics.
  3. The empirical evaluation is comprehensive and compelling. Comparisons with baseline methods, based on numerical metrics and predefined rules, validate the value of evaluations grounded in VLMs, while studies on human alignment and robustness analysis further reinforce these findings.
Critical weaknesses
  1. Comparisons with human ratings are primarily based on rank correlation. However, this fails to capture systematic tendencies toward over- or under-scoring.
  2. While the distribution of ratings itself offers some insight, employing a weighted Cohen’s kappa coefficient or a rating confusion matrix provides a clearer picture of consistency at the level of individual ratings.
Questions for authors
  1. How do the authors recommend combining VLM scores with conventional numerical metrics in practice? When they disagree on which forecasting model is better, how should users interpret the difference?
  2. Some time series datasets may have appeared in VLM pretraining. Could the authors clarify how the evaluation setup mitigates potential contamination?
3 comments

W1: Scoring Bias

We initially examined scoring tendencies by comparing the distributions of ratings from Gemini-3.1-pro and human annotators in the violin plots (Figure 4(b) of submission‾\underline{\text{Figure 4(b) of submission}}). To quantify over- and under-scoring, we now report the mean signed score difference for each judge and dimension, calculated by subtracting the modal human rating from the VLM score for each forecast. On average, all six judges scored Trend below the human reference and Magnitude above it. On the other three dimensions, the direction varied by judge. For example, Gemini-3.1-pro scored Motif 0.375 points above the human reference on average, while Llama scored it 0.342 points below.

VLM judgeTrendMotifPhaseMagnitudeGeneral
Gemini-3.1-pro-0.335+0.375+0.169+0.256-0.121
Qwen3.6-Plus-0.077+0.002+0.165+0.290+0.064
GPT-5.4-0.146-0.104-0.298+0.165+0.209
Claude-Sonnet-4.6-0.369-0.315-0.733+0.044+0.083
Seed-1.8-0.323-0.315-0.613+0.171+0.026
Llama-4-Maverick-0.215-0.342-0.683+0.013+0.290

W2: Item-Level Rating Agreement

For each forecast, we paired each judge’s score with the modal rating from five human annotators to assess agreement on individual forecasts. We computed linearly weighted Cohen’s κ and a rating confusion matrix for each judge and dimension. Weighted κ accounts for chance agreement and weights disagreements in proportion to the absolute difference between scores. Gemini-3.1-pro’s κ ranged from 0.665 to 0.779, with the highest point estimate on four of the five dimensions. Qwen3.6-Plus had the highest estimate on Trend (0.687).

The paired ratings show how often Gemini-3.1-pro agreed with the human reference and how far its scores differed when they disagreed. Exact agreement was most frequent on General (79.27%). Differences of at least two points occurred in 1.04% of Magnitude ratings and 10.21% of Trend ratings. The Motif confusion matrix also shows the direction of disagreement. Among the 144 forecasts with a modal human rating of 4, Gemini-3.1-pro assigned a score of 5 to 89 (61.81%), consistent with the upward scoring tendency observed for Motif in W1.

Linearly weighted Cohen’s κ

VLM judgeTrendMotifPhaseMagnitudeGeneral
Gemini-3.1-pro0.6650.7060.7430.7320.779
Qwen3.6-Plus0.6870.6710.6850.6390.750
GPT-5.40.6250.5890.5860.5860.620
Claude-Sonnet-4.60.5260.5440.3810.5990.655
Seed-1.80.5600.5410.4420.5770.626
Llama-4-Maverick0.5040.4500.3090.5380.507

Gemini-3.1-pro–human score differences

DimensionExact agreement1-point differenceDifference of ≥2 points
Trend54.58%35.21%10.21%
Motif55.83%39.58%4.58%
Phase63.33%31.46%5.21%
Magnitude65.00%33.96%1.04%
General79.27%16.51%4.22%

Q1: Task-Specific Model Selection

In practice, we recommend defining the task objective before comparing models and reporting MSE or MAE alongside the TimeVista score for the relevant rubric. MSE and MAE quantify pointwise error, while TimeVista assesses how well a forecast preserves the temporal properties and satisfies the application requirements specified in the rubric. If the metrics favor different models, the task objective should determine which criterion takes priority. Specifically, when models have similar pointwise errors, their TimeVista scores can reveal differences in how well they capture task-relevant temporal patterns.

For example, the forecast in Figure 4(c) of submission‾\underline{\text{Figure 4(c) of submission}} captures the first congestion event but substantially underestimates the severity of the second. If the goal is accurate speed estimation at each time step, MAE can serve as the primary selection criterion. For congestion warnings, users should give greater weight to how well the model captures the timing and severity of congestion while also reporting the overall speed error.

Q2: Data Contamination and Memorization

While the raw numerical data of these datasets might exist in the VLM's pretraining corpus, our framework effectively mitigates data contamination risks through modality and task shifts. Instead of forecasting future values from historical numbers, the VLM acts as a visual judge evaluating newly generated plots against our custom rubrics. Since these visual plots and evaluation rubrics are uniquely generated for this study, it is highly unlikely that the model can rely on memorization. This design ensures that our meta-evaluation primarily assesses zero-shot visual reasoning and instruction-following capabilities, rather than historical data recall.

Thank you for your detailed response. The clarifications and additional robustness checks have fully addressed my concerns. TimeVista represents a substantive methodological advance in forecast evaluation, and I am keeping my score as a clear accept.

Reviewer_tCEqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a very interesting dataset and benchmarking study for time series forecasting models. Compared with past works mainly focus on the average MSE or average RMSE computed using numerical values, this presented timevista benchmark considering more dimensional metrics for evaluating the performance via plotting time series patterns into micro-level and macro-level image representations and using VLM to serve as a judge for checking the model performance in terms of these new checking angles.

This work extends the dimensions for more comprehensively examining the time series model performance rather than simply replying on MSE or RMSE, those traditional statistical metrics.

Critical strengths

The key strength of this work is that, via introducing this new timevista benchmark, the performance of time series forecasting models can be checked by going beyond those statistical metrics, which have some drawbacks. The VLM serving as a judge can make this checking process more autonomous.

Critical weaknesses

The critical weakness is the lack of theoretical justifications in terms of the advantages of considering these image-based checking targets. How to justify that the time series foundational model done better than other if they only do well in terms of the micro-level and macro-level image based patterns introduced in this study.

Meanwhile, the robustness of considering VLM as the judge needs a better justification.

Questions for authors

My main questions are the same as those written in the critical weakness.

4 comments

W1: TimeVista as a Complement to MSE and MAE

The relative performance of time series foundation models varies across datasets and forecast configurations [1, 2, 3]. Interpreting one forecast as better than another therefore requires specifying which aspects of forecast quality matter for the task.

MSE and MAE quantify pointwise error, but forecasts with similar errors can differ in how well they preserve temporal patterns, such as trend and phase. TimeVista complements these metrics by using plots to represent temporal structure and rubrics to specify which properties a forecast should preserve and which deviations matter most for the task. A higher TimeVista score indicates that the forecast better satisfies the task requirements specified in the rubric. Model selection should consider both how well those requirements are met and the pointwise accuracy measured by MSE and MAE.

W2: Judge Robustness to Plot and Rubric Changes

The submitted paper evaluates Gemini-3.1-pro’s robustness across repeated runs, changes in plot rendering, and rubric paraphrasing. For 94.28% of the 2,465 forecast–dimension pairs, the five scores obtained from identical inputs spanned at most one point (Table 13 of submission‾\underline{\text{Table 13 of submission}}, Appendix E.5). On 480 synthetic forecasts, Spearman’s ρ between Gemini-3.1-pro and human Magnitude ratings was 0.907 under the default settings, ranged from 0.895 to 0.917 across rendering changes (Table 4 of submission‾\underline{\text{Table 4 of submission}}), and was 0.901 after rubric paraphrasing (Table 3 of submission‾\underline{\text{Table 3 of submission}}).

To test whether the robustness generalized to other VLM judges, we evaluated Qwen3.6-Plus on the same forecasts under 11 rendering changes and a paraphrased rubric. Spearman’s ρ with human Magnitude ratings was 0.862 in a new run under the default settings, ranged from 0.850 to 0.888 across the rendering changes that preserved curve identities, and was 0.850 after rubric paraphrasing. Detailed results are reported in the response to Reviewer Lpuj (W5/Q3).

Both Gemini-3.1-pro and Qwen3.6-Plus maintained strong rank agreement with human raters across these changes, supporting their use as robust judges of time series forecasting.

References

[1] Aksu, T., Woo, G., Liu, J., Liu, X., Liu, C., Savarese, S., Xiong, C., & Sahoo, D. (2024). GIFT-Eval: A benchmark for general time series forecasting model evaluation. NeurIPS Workshop on Time Series in the Age of Large Models. Paper

[2] Das, A., Kong, W., Sen, R., & Zhou, Y. (2024). A decoder-only foundation model for time-series forecasting. Proceedings of the 41st International Conference on Machine Learning, 10148–10167. Paper

[3] Wang, Y., Chen, Z., Shi, Z., Liu, Y., Wang, S., Ye, Z., Wang, J., & Long, M. (2026). TimeAgent: A training-free LLM agent for time series foundation models. IEEE Transactions on Knowledge and Data Engineering, 38(12), 8528–8541. DOI

Reviewer_tCEqOpenReview ↗

Response to Authors

Thanks for spending efforts on addressing my concerns. Well, I can sense the change of VLM as judge has significant influence. I still have concerns on the reliability of doing so. If there exists some sensitivity, some mechanisms for selecting right VLM will be important to cover this crack. I can maintain my score.

Using a fixed evaluator gives models a common basis for comparison, as in MT-Bench [1] and AlpacaEval [2]. Meta-TimeVista serves as a benchmark for selecting the VLM judge whose ratings best align with human judgments on the forecast properties relevant to the task.

We evaluated six candidate judges on the same forecasts against the same human reference ratings. Controlled synthetic data allow us to assess agreement separately on trend, motif, phase, and magnitude. Forecasts from diverse real-world scenarios allow us to assess how well each judge applies domain-specific criteria. As shown in Table 1 of submission‾\underline{\text{Table 1 of submission}}, Gemini-3.1-pro achieved the highest rank correlation with human ratings on four of the five dimensions. We therefore selected it as the fixed judge for the full benchmark.

For a specialized task, Meta-TimeVista also supports dimension-specific judge selection. For example, GPT-5.4 achieved the highest correlation with human ratings on Phase (ρ = 0.854, compared with 0.831 for Gemini-3.1-pro). We recommend selecting the judge that best aligns with human ratings on the dimensions most relevant to the application, then using that judge consistently across all forecasting models being compared. As new VLMs become available, Meta-TimeVista can serve as a benchmark for selecting the judge whose ratings best align with human judgments on the dimensions most relevant to the task.

References

[1] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. Paper

[2] Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. Conference on Language Modeling. Paper

Reviewer_tCEqOpenReview ↗

Replying to follow-up comments by Authors

Thanks for the further explanations. I am definitely ok with this more empirical conclusion. Yet, I think the theoretical touch of this point is pretty much open and deserves further more rigorous explorations. Regarding this work, this point is not its core. I think regarding the core contributions, authors have done sufficiently and I maintain my supportive attitude.

Comments on the paper

1 comment

Dear reviewers,

The authors have now submitted their responses. Please review them and share your thoughts on whether your concerns have been addressed. I encourage you to engage with one another’s assessments, discuss any remaining disagreements, and update your reviews and scores as appropriate.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.