acceptodds
Under review as a conference paper at ICLR 2027

SoccerCanon: A challenging Benchmark and the Case for Stitching Vision Experts over VLMs

Abstract

Soccer player recognition in broadcast imagery remains challenging under blur, occlusion, small scale, and extreme viewpoints, where identity cues such as faces, appearance, and jersey numbers are often incomplete or unreliable. Existing benchmarks provide limited support for systematically evaluating recognition under such heterogeneous visual evidence. To address this limitation, we introduce SoccerCanon, a benchmark of 1,108 labeled player instances, comprising 613 validation and 495 test instances. Face recognizers that identify clean player photographs with about 96% R@1 reach only 40-47% on SoccerCanon, and accuracy collapses on the small faces that dominate broadcast imagery. We evaluate specialized vision models and general-purpose vision-language models (VLMs) under controlled, modality-aware settings. Our experiments reveal that different visual cues exhibit complementary strengths, while no single recognition pathway remains consistently reliable across challenging cases. Motivated by this observation, we propose a training-free expert-stitching framework that combines face, appearance, and number experts according to their query-specific reliability. Our framework reaches 60.94% R@1 on the face-available test observations without match metadata, whereas the strongest VLM reaches 29.24% even when given match-restricted candidates, and reliability-aware stitching improves most where face evidence is weak.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, visual recognition, player recognition, vision expert stitching, expert aggregation, vision-language models, multimodal learning, representation learning, computer vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cz9NOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper tackles the task of soccer player recognition from broadcast images. For that purpose, a dataset is collected, based on SoccerNet data, and annotated semi-automatically and semi-manually. Player identities are associated to the player crops considered. Then, various models are tested on the dataset. A classifier based on face detection attributes, jersey colors attributes, and OCR-based jersey number attributes is provided, by fusing the output of the used pretained models (no training is performed). The system reports decent performances on the dataset considered, and outperforms the baselines tested.

Critical strengths

The paper is a clean computer vision paper, with a well-defined task, a dataset built for it, a model developed for it, and evaluated against existing prior work. There seems to be no methodological flaw, and overall, the paper is pleasant to read.

On the dataset tested, the proposed solution outperforms by far the baselines considered.

Critical weaknesses

One of the claimed contributions is the dataset constructed. However, the dataset mostly relies on the SoccerNet dataset annotations: the paper subsamples some part of SoccerNet, does some processing with pretrained models, reviews the outputs and derives player names manually. Then the paper claims to provide a new dataset/benchmark under a non-related SoccerNet name. This is not really fair, and probably a coordination with the SoccerNet team (which is very active) should be made to ensure this is a legitimate way to proceed.

Besides, what is even more surprising, is that the provided annotations are very small-scale (1k instances), while SoccerNet already provides essentially the same data for 340k+ instances, just without player names. Still, 106k+ jersey numbers are provided, which can be directly associated to a player name with the match metadata (collected by the authors of the present paper). So, unless I completely missed the point, the paper could have done much more with all the data available, and I do not see why the presented approach might be substantially more interesting.

Finally, the paper seems to solve a useless, factice problem. In practice, why would anyone try to recognize a player from a broadcast stream by looking only at just one small player crop? Videos are very long and identification of a player can be done at many times, and tracking an identified player (as done in SoccerNet tracking) seems much more valuable (considering one can associate the player with its name with the metadata). Also, the proposed pipeline starts from pre-cut crops of players, but getting them, sorting those that are useful or not, is a practical resource-heavy exercise completely dismissed here, which needs to be done in production. Finally, the method will be unable to deal with new faces, new players, because of the fixed reference bank for the classification. New players appear all the time, every year, so a more flexible system would be much more valuable.

Questions for authors

What is the benefit of the proposed approach over the numerous and existing SoccerNet annotations, and that really requires substantial human effort, thus giving the value of this dataset?

What is the usefulness of the proposed work in a more global applicative context, e.g. if a new game is played, what will work, not work, and how can this be improved?

9 comments
AuthorsOpenReview ↗

Response to Reviewer cz9N (1/3): W1 / Ethics, provenance, name, consultation and release

We thank Reviewer cz9N for the careful review and the thoughtful questions. We answer W1 (ethics) in this comment, W2/Q1 in the second comment, and W3 and Q2 in the third.

[W1 / Ethics. Provenance, the name SoccerCanon, and coordination with SoccerNet]

Thank you for raising this. Clear provenance matters to us as well, and we answer the three points of the concern in turn.

(1) Does SoccerCanon rely on SoccerNet's annotations? No. SoccerCanon uses SoccerNet's images and videos, not its player annotations. From SoccerNet we take the action and replay frames released with SoccerNet-v3 [3], organized by match, and the corresponding match videos; the videos were used only by human annotators to verify identities and are never input to any model. The SoccerNet-v3 bounding boxes, jersey-number labels and player-correspondence IDs were not used at any stage, either by our pipeline or by the annotators. Player localization and the face, upper-body and number crops come from our own pipeline (Section 3.2.1, Appendix A.1). What SoccerCanon adds is a label that SoccerNet does not contain: human annotators linked each observation to a canonical real-world player, consistent across matches and seasons.

(2) Is the name fair to SoccerNet? The manuscript already names SoccerNet as the image source in the Introduction, Section 3.2, Figure 2, Appendix A.1 and the Ethics Statement, and cites Deliege et al. (2021) [2] and, in Section 3.2 and Appendix A.1, Cioppa et al. (2022) [3]. In the revision, the name SoccerCanon will denote only what is new, namely these canonical identity labels and the identification task built on them; the images remain SoccerNet's. We will state this in the abstract, the contribution list and Table 1, describing SoccerCanon as an identity annotation layer on SoccerNet-v3 frames, and we will add the original SoccerNet paper (Giancola et al., 2018) [1].

(3) Was SoccerNet consulted? Yes. Based on what we described, SoccerNet replied by email that they see no issue with the research from their side: using SoccerNet to construct and evaluate a new academic benchmark is consistent with its intended research use, provided that the original SoccerNet data and the derived benchmark are not redistributed and the relevant SoccerNet work is properly cited, and no additional permission is needed from them for the manuscript as described. SoccerNet recommended making clear in the paper that the benchmark is derived from SoccerNet and is not publicly redistributed, which we will do, and asked us to contact them before any public release of the derived benchmark. We can forward this written confirmation to the PCs if helpful.

What we release, and how. We will not redistribute any SoccerNet images or videos: no frames, crops or match videos. Third-party reference photographs and kit images are not redistributed either; they remain under their sources' terms. On publication we will release the code (construction pipeline, our method and the evaluation scripts); it contains no SoccerNet images. We do not release the benchmark annotations at this point. Should we release them, we will first contact SoccerNet, as they asked, and share only what they agree to, in this form: for each observation, a fixed observation ID, the canonical identity label, the SoccerNet game and frame identifiers, and the fixed pixel boxes of the player crop and of its face, upper-body and number crops in that frame, plus the frozen per-query rankings behind the reported results. Such files contain no image content, so users would obtain SoccerNet access themselves, under SoccerNet's terms, and rebuild each crop by cutting the frame at these boxes; no detector is re-run, so the crops do not depend on detector versions. We checked that this reproduces all 1,108 stored player crops and their face, upper-body and number crops pixel for pixel. We will revise the Ethics and Reproducibility Statements accordingly.

[1] Giancola et al. (2018), SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos, CVPR Workshops 2018.

[2] Deliege et al. (2021), SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos, CVPR Workshops 2021.

[3] Cioppa et al. (2022), Scaling up SoccerNet with multi-view spatial localization and re-identification, Scientific Data 9, 355 (SoccerNet-v3).

AuthorsOpenReview ↗

Response to Reviewer cz9N (2/3): W2/Q1 identity annotations

[W2 / Q1. What do the identity annotations add beyond numbers and match metadata?]

We appreciate the opportunity to clarify. SoccerCanon links 1,108 observations from 213 matches to 424 canonical player identities, so it can evaluate whether a difficult broadcast observation can be connected to the same real person across matches. Linking jersey numbers with line-ups, as the reviewer suggests, would give a much larger identity set, but a label of that kind exists only where a number is legible. Our audit of the 489 test labels shows that the jersey or shorts number settled only about half of the identities:

Test queriesCount
Audit: identity settled by the jersey or shorts number249 of 489 (50.9%)
Audit: identity settled by another cue (face 229, scene context 3, other 8)240 of 489 (49.1%)
Automatic reading: OCR returns no digit string204 of 489

A set built from numbers and line-ups would therefore leave out the 49.1% of test identities that were settled by the face or other evidence, as well as many observations whose number the automatic reader cannot recover from the crop (no digit string for 204 of 489). These are exactly the cases SoccerCanon is built to evaluate. We therefore see the two approaches as complementary: number-and-line-up labels scale well where the number is legible, and SoccerCanon tests the harder cases with verified identities. Human verification is used only to create the ground truth; identification at inference is automatic. We will make the distinction between the existing SoccerNet annotations and the new cross-match identity labels explicit.

AuthorsOpenReview ↗

Response to Reviewer cz9N (3/3): W3 task motivation, Q2 crops and new players

[W3. Why study a single observation when video and tracking are available?]

Thank you for raising the downstream motivation. The task is motivated by background-knowledge image question answering, and our focus is the recognition challenge posed by broadcast observations with difficult viewpoints, blur and occlusion:

SettingWhy identity grounding is needed
Background-knowledge image question answering (a related setting: SoccerBench Q4 [4])Before answering a question about a player, a system must ground the observed person to the correct identity, and a single-image question need not come with a video track
Video systems with trackingMapping a track to a real player still needs identity evidence, so the task can also be used as a component within a larger video system
Match metadata availableNarrows the candidates but does not name the player: shirt-number look-ups among the actual participants reach at most 19.6% R@1 (reply to Reviewer rXSr)

Section 3.1 will make the evaluation scope clear.

[Q2. How are crops obtained, and how can the system accommodate new players or matches?]

Thank you for this practical question:

StepHow
Test set: detection, crop generation (YOLOv8n-Pose) and candidate selectionAutomatic
Identity labelsEstablished and verified by humans (ground truth only)
New playerAdd reference photos and the roster entry; no retraining of the recognition model
Match informationCan be incorporated: the two teams' season squads or the actual participants can be placed first among the candidates
Harder caseNowNext step
Player without a reference photoCannot be namedUpdate reference photos and rosters
Team without kit references75 team-seasons coveredUpdate kit references

We will make the separation between automatic and human steps clearer in the description of the construction pipeline.

[4] Rao et al. (2025), Multi-Agent System for Comprehensive Soccer Understanding, ACM Multimedia 2025. Appendix C.1.4 (Background Knowledge Image QA) describes constructing image questions using SoccerWiki player photographs

Reviewer_ed1iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SoccerCanon, a benchmark for canonical soccer player identification from challenging broadcast imagery, and proposes a training-free expert-stitching framework combining face recognition, kit appearance, and jersey-number evidence. The framework maps heterogeneous expert outputs into a shared player identity space and achieves 60.94% R@1 on the face-available test subset.

Critical strengths

1.A meaningful benchmark for a challenging and underexplored recognition environment

SoccerCanon focuses on consistent identity recognition across matches, rather than traditional in-match tracking or re-identification. By utilizing real-world broadcast footage—characterized by small face sizes, blur, occlusions, and varying viewpoints—it creates a challenging evaluation environment. The benchmark encompasses a diverse range of matches and identities, complete with manually verified annotations. This represents a valuable contribution, as it reveals limitations that might not be apparent in conventional facial recognition benchmarks.

2.Extensive empirical evaluation and a measurable improvement over the face-only system.

This paper evaluates a variety of face recognition models, visual encoders, re-identification methods, and vision-language models (VLMs). It also includes controlled comparisons, component ablation studies, and analyses conducted under diverse visual conditions. The proposed approach improves the R@1—based solely on facial components—from 54.19% to 60.94%, with a 95% confidence interval (calculated via paired bootstrap) of [4.29, 9.20] percentage points for the magnitude of the improvement. These results provide evidence that integrating complementary information can enhance recognition performance under challenging conditions.

Critical weaknesses
  1. Compared to simpler fusion mechanisms, the aggregated contributions that consider reliability aren't really distinguished enough.

The core methodology advocates for the use of query-specific reliability estimation and conflict-aware evidence aggregation. However, the ablation results presented in Table 16 do not clearly demonstrate the necessity of these mechanisms. Starting from a baseline of fixed prompt weights, the introduction of query-specific reliability leads to a slight decline in the number of correct predictions, while conflict discounting yields no further improvement. Subsequent gains stem primarily from facial structural consistency (increasing correct predictions from 269 to 275) and reciprocal rank fusion (increasing correct predictions from 275 to 298).

This makes it difficult to determine whether the reported gains truly arise from the integration of reliability-based aggregation, or if they could largely be achieved through a simpler combination of structural consistency and rank fusion with fixed weights. Furthermore, under the same setting of 100 fixed candidates (as shown in Table 10), the proposed aggregation method improves the R@1 of the authors' own facial expert by only 1.84 percentage points (from 72.39% to 74.23%); notably, no paired significance analysis is provided for this marginal improvement.

Consequently, a controlled ablation experiment is still required—one that retains the existing consistency and rank fusion mechanisms while replacing the query-dependent reliability component with fixed weights. Without such a comparison, the benefits of the proposed reliability modeling remain insufficiently substantiated.

  1. Benchmark curation and annotation introduce potential selection bias and uncertainty about the image-only task.

Although the benchmark is intended to represent challenging broadcast observations, its construction involves substantial filtering. Only 495 of the 3,599 selector-approved test candidates (13.8%) are retained after manual identity verification. Moreover, the automatic selection process requires weakly detectable facial evidence, excluding some particularly difficult observations.

More importantly, annotators can use match metadata, lineups, and surrounding video to establish ground-truth identities, whereas evaluated methods receive only a single observation. The paper does not clearly quantify how often temporal context is necessary for annotation or whether such identities can reasonably be inferred from the evaluated image alone.

This matters because the resulting benchmark may reflect a selectively curated subset of recognizable or externally verifiable players rather than the broader distribution of broadcast identification challenges. Additional annotation statistics and a breakdown according to the evidence required for identity verification would strengthen the interpretation of the reported results.

Questions for authors
  1. Is a query-dependent reliability estimation still necessary after incorporating "consensus" and "rank fusion" mechanisms?

Table 16 suggests that the largest improvements come from consensus and reciprocal rank fusion rather than reliability estimation or conflict discounting individually. Could the authors provide a controlled comparison retaining the final consensus and RRF components while replacing query-dependent reliability with fixed cue weights? It would also be useful to report paired significance for the 1.84-point improvement over the face-only expert in the Fixed-100 setting. These results would directly affect the assessment of the method's central contribution.

  1. To what extent does the ground-truth annotation process rely on information that is inaccessible to the method being evaluated?

Could the authors provide a breakdown of the evidence used for annotations—for instance, the proportion of cases identified via jersey numbers, facial features, or video context? Specifically, how many test samples rely on temporal evidence that is absent from the "single-frame input"? Additionally, providing details on inter-annotator agreement, the process for resolving disagreements, and the primary reasons for excluding candidates already identified by the "selector" would help in assessing the reliability of the benchmark dataset and potential selection bias.

  1. To what extent can the performance gap between specialized expert models and Vision-Language Models (VLMs) be attributed to differences in their visual inputs?

In the controlled comparative experiments, the face-specific models utilize "aligned face crops," whereas the VLMs receive "crops containing the full player." Given the small face size in the SoccerCanon dataset, could the authors evaluate the best-performing VLM baseline model—keeping the candidate pool and reference images constant—by testing it on both "aligned face crops" and "images containing the full player"? This would help disentangle the limitations inherent in the VLM’s recognition capabilities from the effects stemming from different input preprocessing methods.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ed1i (1/3): Q1 per-query reliability

We thank Reviewer ed1i for the careful review and the precise questions. We answer Q1 here, and Q2 and Q3 in the next two comments.

[Q1 (W1). Is per-query reliability estimation still useful after consensus and RRF?]

Thank you for helping us sharpen this distinction; the requested control lets us separate the two roles in our framework. Cue integration produces the ranking accuracy: with consensus and RRF retained, setting all cue-reliability weights to 1 leaves the test R@1 unchanged (298/489, full candidate set; no match-specific information). Per-query reliability does a different job, one that R@1 does not measure: estimated from each expert's outputs together with the face expert's gallery-search diagnostics, it tells the system when to trust its answer, which our pre-specified test-set analyses confirm (each run once on FA-489; 95% match-cluster CIs):

What per-query reliability addsBaseline → with per-query reliabilityDifference [95% CI], p
Detecting number-cue failures (baseline: reliability inputs fixed, = the cue's own confidence)0.540 → 0.891 AUROC+0.351 [+0.287, +0.414]
Detecting system errors (baseline: the fused system's own confidence)0.9415 → 0.9676 AUROC+0.026 [+0.011, +0.043], p = 0.0006
Detecting system errors (baseline: same detector, reliability inputs fixed)0.9590 → 0.9676 AUROC+0.0086 [+0.0026, +0.0156], p = 0.006
Detecting face-cue failures (baseline: strong cross-cue-agreement detector, reliability inputs fixed)0.975 → 0.989 AUROC+0.014 [+0.006, +0.023]

Rows 2–3 use all 489 test queries; rows 1 and 4 use the 485 whose ground truth is in the candidate set.

Overall gain:

ComparisonGain of ours over the face expert, R@1
Fixed-100, test+1.84 points (72.39% → 74.23%; 11/2 discordant queries, exact McNemar p = 0.022; 95% CI [+0.46, +3.26])

We will add this analysis to Section 5.

AuthorsOpenReview ↗

Response to Reviewer ed1i (2/3): Q2 label evidence

[Q2 (W2). What evidence supports the labels, and how much of it is available to the model?]

We appreciate this important question. The evidence behind each of the 489 evaluated test labels is as follows:

Label checks (489 labels)Labels
Same player on the replay footage488 (99.8%)
Corrected to a different player1
Labelled player listed among the actual match participants (public line-ups)489 of 489
Evidence needed to settle the identity (489 test observations)Observations% [95% CI]
Query crops alone, without match information32666.7 [58.1, 74.9]
Needs information unavailable to the method without metadata (sub-rows overlap)16333.3
↳ match information such as the matchday lineup42
↳ evidence outside the crops (full frame, video or other scenes)149
↳ of these 149: another moment or scene necessary (% of all 489)14730.1 [22.0, 38.8]
Decisive cueObservations
Number249 (50.9%)
Face229 (46.8%)
Scene context3
Other8

Without any match metadata, the method still identifies many of the observations whose label needed other information, and the gain of cue integration holds in both groups (full candidate set; 95% match-cluster CIs):

R@1 without match metadataOursFace cue alone (Table 4)Ours − face cue
Settled from the query crops alone (326)66.26%58.28%+7.98 [+5.22, +10.83]
Needed other information (163)50.31%46.01%+4.29 [+1.62, +7.09]
All test queries (489)60.94%54.19%+6.75 [+4.70, +8.82]

Two research assistants cross-checked kit, number, roster and face evidence, consulting the broadcast video where needed, and one author made the final review; candidates whose evidence did not converge on a single identity were excluded.

AuthorsOpenReview ↗

Response to Reviewer ed1i (3/3): Q2 selection, Q3 VLM input

[Q2 (W2), continued. Does selection bias the evaluation?]

On selection: the relative gain of cue integration holds across the measured face-evidence strata:

Selection checkGain of ours over the face expert, R@1
Every measured face-evidence stratumNon-negative
Faces under 32 pixels (selector-measured, 88 test queries)+9.09 points
Reweighted to the distribution of the 3,047 approved candidates with a detected face+7.25 to +9.80 points

[Q3. How much of the expert–VLM gap is explained by the query input?]

Thank you for proposing this direct diagnostic. We evaluated Qwen3-VL-Reranker-8B, the strongest VLM in Table 10, with the same Fixed-100 candidate pool and reference images, replacing the player-crop query with the aligned face crop that our face expert uses:

Query given to the VLMVLM R@1Gap to the face expert (72.39%)
Player crop (paper)29.24%43.15 points
Aligned face crop (the expert's input)22.90%49.49 points

The change is −6.34 points (95% CI [−10.37, −2.38]; p = 0.0019): giving the VLM the same face input does not close the gap. We will add this comparison to Appendix B.4.

Reviewer_rXSrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of identifying soccer players from single broadcast images, where visual cues such as faces, uniforms, and jersey numbers are often difficult to recognize. The authors introduce SoccerCanon, a benchmark for evaluating player identification under these challenging conditions. They also analyze the limitations of existing face recognition, re-identification, and VLM-based approaches.

To address this problem, the paper proposes a training-free framework that combines specialized experts for face recognition, uniform appearance, and jersey-number recognition. The method uses player reference images and team-season rosters to map different visual cues to player identities and combines their predictions based on estimated reliability. The experiments show that the proposed approach improves player identification compared with individual recognition methods and simpler fusion strategies.

Critical strengths
  1. The paper provides a useful benchmark for soccer player identification under challenging visual conditions.

    SoccerCanon contains 1,108 annotated player observations covering 424 identities and 213 matches across different competitions and seasons. The experiments show that existing face recognizers perform substantially worse on broadcast images than on clean player photographs, particularly when faces are small or blurred. This provides a useful evaluation resource for studying the limitations of existing recognition models under challenging visual conditions.


  1. The paper provides useful insights into the complementary strengths of different visual cues.

    The experiments show that face recognition, uniform appearance, and jersey-number recognition provide complementary information for player identification. In particular, Table 4 shows that simply averaging these cues can reduce performance, while the proposed fusion framework improves over the face-only baseline. The additional analyses of face quality and number availability help explain when different visual cues are useful.

Critical weaknesses
  1. The practical motivation for the proposed benchmark setting is not sufficiently justified.
    SoccerCanon defines player identification from a single broadcast observation without using match-specific metadata. However, practical soccer analytics systems typically have access to participating teams, player lineups, jersey numbers, and substitution information, which can substantially reduce the candidate space. It is unclear why such readily available information is excluded from the proposed task.
    Furthermore, the VLM baselines use 100 candidates constructed from season squads and additional players rather than the actual players participating in a match. I believe that evaluating simpler identification strategies using match-specific metadata and visual cues would better establish the practical relevance of the proposed benchmark.

  1. The contribution of query-specific reliability estimation is not sufficiently validated.
    The paper emphasizes reliability-aware expert stitching as a central contribution. However, Appendix Table 16 shows that adding query-specific reliability slightly decreases R@1, while conflict discounting provides no additional improvement. Most of the performance gain comes from face–structured consensus and reciprocal rank fusion, the latter being an existing rank aggregation method.

    Although reliability estimates are also used in the consensus mechanism, it remains unclear whether the complete framework would achieve similar performance using fixed reliability weights. An additional ablation retaining consensus and rank fusion without query-specific reliability would better establish the contribution of the proposed reliability-aware design.

Questions for authors

Regarding Weakness 2: Would the proposed framework achieve comparable performance using fixed reliability weights while retaining the same consensus and rank fusion components?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rXSr (1/2): W1 match information

We thank Reviewer rXSr for the careful review and the constructive suggestions. We answer W1 here, and W2 and the question in the next comment.

[W1-(1). Why evaluate a setting without match metadata?]

Thank you for highlighting this practical setting. We fully agree that match information is valuable, and we will report it as a complementary Track B alongside the original setting:

TrackInputsQuestion it answers
A (original)Reference photos, kit references, competition-level rosters (75 team-seasons)How much identity evidence the observation itself carries when per-match metadata is unavailable
B (new)Track A + the actual match participantsHow far match information takes a practical system

Track A arises whenever a single image comes without match information, for example in background-knowledge image question answering, where a system must first identify the pictured player (a related setting: SoccerBench Q4 [1]).

[W1-(2). How does a simple metadata-based strategy compare, and are the candidates actual participants?]

Thank you for this suggestion. As you note, the Fixed-100 candidates are the two teams' season squads plus additional players from the same league, not the players who actually took part. We transcribed the actual participants (starters and substitutes who came on) of all 125 test matches from public match line-ups, cross-checked them against a second public source, and compared simple, practical strategies that use them with our method:

Strategy (FA-489, 489 test queries)Roster information usedR@1 [95% CI]
Shirt number looked up among the actual participantsActual line-ups of the match9.6% [7.3, 12.1]
Kit picks the team, then the shirt number among its participants (best simple rule we tried)Actual line-ups of the match19.6% [16.4, 22.8]
Our method (paper)Season rosters of all 75 team-seasons (no match-specific information)60.94% [56.16, 65.61]

Match information narrows the candidates but does not identify the player: simple line-up look-ups stay below 20%, whereas our method, which uses the identity evidence in the observation, reaches 60.94% without any match-specific information. We will add these rows and clearly distinguish Fixed-100 candidate lists from the actual match participants.

[1] Rao et al. (2025), Multi-Agent System for Comprehensive Soccer Understanding (SoccerBench), ACM Multimedia 2025; Appendix C.1.4, Background Knowledge Image QA.

AuthorsOpenReview ↗

Response to Reviewer rXSr (2/2): W2 and the question on reliability

[W2 / Question. With consensus and RRF retained, do fixed reliability weights perform similarly?]

Thank you for suggesting this informative control; it let us separate the two roles in our framework precisely. Cue integration produces the accuracy gain: under fixed weights, face–structured-cue consensus still adds 12 correct predictions on each of the test and validation sets, and setting all cue-reliability weights to 1 leaves the test R@1 unchanged (298/489; no match-specific information). RRF is an existing technique that we will attribute to Cormack et al. (2009) [2]. Per-query reliability does a different job, one that R@1 does not measure: estimated from each expert's outputs together with the face expert's gallery-search diagnostics, it tells the system when its answer, or an individual cue, is likely to be wrong, and this effect is clear and statistically confirmed. Each test was planned before it was run once on the test set (FA-489; 95% match-cluster CIs):

What per-query reliability addsBaseline → with per-query reliabilityDifference [95% CI], p
Detecting number-cue failures (baseline: reliability inputs fixed, = the cue's own confidence)0.540 → 0.891 AUROC+0.351 [+0.287, +0.414]
Detecting system errors (baseline: the fused system's own confidence)0.9415 → 0.9676 AUROC+0.026 [+0.011, +0.043], p = 0.0006
Detecting system errors (baseline: same detector, reliability inputs fixed)0.9590 → 0.9676 AUROC+0.0086 [+0.0026, +0.0156], p = 0.006
Detecting face-cue failures (baseline: strong cross-cue-agreement detector, reliability inputs fixed)0.975 → 0.989 AUROC+0.014 [+0.006, +0.023]

Rows 2–3 use all 489 test queries; rows 1 and 4 use the 485 whose ground truth is in the candidate set.

We will present reliability as setting the relative weight of the cues and flagging likely failures, and include the requested control in the paper.

[2] Cormack et al. (2009), Reciprocal rank fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009.

AuthorsOpenReview ↗

A brief follow-up

Dear Reviewer rXSr,

We sincerely thank you once again for the time and effort you have devoted to reviewing our paper.

As the author–reviewer discussion period approaches its conclusion, we hope that our responses have helped clarify the points you raised. If any questions or concerns remain, we would be more than happy to provide further clarification.

We truly appreciate your thoughtful feedback and consideration throughout the review process.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the three reviewers for their careful reading and constructive suggestions. Each new analysis is described in the thread of the reviewer who raised it, together with what will be added to the revised paper.

  • Reviewer rXSr. A fixed-weight control (ρ = 1, consensus and RRF kept): test R@1 unchanged (298/489, full candidate set), so the accuracy gain comes from cue integration, while per-query reliability detects cue failures (number-cue AUROC 0.540 → 0.891); simple line-up look-ups over the actual participants of all 125 test matches (best rule we tried 19.6% R@1, vs. 60.94% for ours with the season rosters of all 75 team-seasons and no match-specific information).
  • Reviewer ed1i. The same control and reliability analyses; a re-check of all 489 test labels against the replay footage and public line-ups (488 of 489 confirmed, every labelled player among the actual participants, 66.7% settled from the query crops alone); the strongest VLM of Table 10 given our face expert's aligned face crop (R@1 29.24% → 22.90% on Fixed-100, vs. 72.39% for the face expert, so the gap does not close).
  • Reviewer cz9N. Written confirmation from SoccerNet that building a new academic benchmark from SoccerNet is consistent with its intended research use and needs no additional permission from SoccerNet for the manuscript as described, provided neither the SoccerNet data nor the derived benchmark is redistributed and SoccerNet is cited; in the revision, SoccerNet attribution will be more prominent in the abstract, contributions and Table 1.

We hope these results address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.