- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SED, a structured prediction interface built on frozen CLIP image representations. It separates shared, evidence, and concept pathways, uses concept-conditioned gating to modify the evidence representation, and combines these pathways for fact and concept prediction through text-prototype heads. The evaluation uses Flickr30k-derived paired annotations with 603 evidence labels and 2,487 concept labels. It compares SED with alternative heads using the same frozen features and examines source removal, gate perturbation, concept substitution, and representation probes. Predictive performance is reported as close to that of a shared MLP, and the intended contribution is an inspectable and intervenable representation structure.
Critical strengths
The shared, evidence, and concept pathways expose representations that can be probed and modified separately. The paper avoids equating architectural separation with complete semantic disentanglement or causal mediation. This makes the proposed interface potentially useful even when its aggregate predictive accuracy is similar to a conventional shared head. Holding the frozen visual features and label space fixed is a sensible way to compare downstream structures. Source removal, gate perturbation, concept substitution, and probes address different aspects of the architecture. The authors also acknowledge the small predictive gap to the shared MLP rather than treating that gap alone as decisive evidence.
Critical weaknesses
-
The seed-level comparison also needs reconciliation. Table 1 reports concept F1 of for SED and for the shared MLP, using the mean and sample standard deviation over three seeds. Table 2 reports a seed-42 difference of points. Ordinary seed variation therefore does not explain the reported discrepancy.
-
Table 3 introduces another full-model result, whose relationship to Tables 1–2 is insufficiently specified.
-
The four refinement losses in Eq. 16 are named without enough information, and the provenance and construction of the paired fact–concept annotations are unclear.
-
The final dataset counts after filtering, split construction, and the data used for vocabulary filtering also need specification. Since the contribution depends on the supervised semantic categories and the refinement procedure, the omissions materially limit reproducibility and the interpretation of the diagnostic findings.
Questions for authors
-
Are the point estimates in Table 2 original test-set differences or means of bootstrap estimates? How the reported seed-42 difference relates to the three-seed statistics.
-
Could you provide the referenced appendices, full definitions of the refinement objectives, and the annotation-generation procedure? How were evidence and concept labels produced and checked, what are the final split sizes, and were frequency-based vocabulary decisions made using training data only?
-
The fixed/shuffled-gate interventions modify a gate that depends on both image-side and concept-side inputs, while substituting also affects direct fusion and prediction routes. What happens when the concept contribution to the gate is disabled while retaining the image-side gate, or when is substituted only inside the gate while its direct routes remain fixed?