- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
When one wants to train a model to detect compositional video events, what is usually available are the final labels. The intermediate concept labels for every frame and location are typically not available. To address this, this paper proposes a neuro-symbolic model that combines the VideoMamba backbone with Scallop. Experimental results show that the proposed model recovers expensive intermediate concepts from cheap final labels alone.
Critical strengths
The studied problem is interesting, i.e., how to recover expensive intermediate concepts from cheap final labels alone?
Critical weaknesses
The overall writing needs improvement. It is hard to follow the paper. The performance in some cases is not convincing.
Questions for authors
-
For Section 3, I would suggest giving more detailed descriptions of the selected setting/task, illustrating it with an example. After reading Section 3, all I know is that there might be two agents handing over the instrument, and I do not know what the goal is or what the actual example is.
-
The method is based on VideoMamba and Scallop, but there is no preliminary information about them.
-
“Perception module”: The descriptions of the processing procedure are not clear enough; for example, what does the “selective scan” in L139 mean, and what does the notation f in L141 mean? It would also be appreciated if the authors could give more explicit descriptions of how to get the neural output, what each probability in the map means, and what the focal loss is like.
-
“Symbolic reasoning”: The paper only describes six concepts. Does the proposed pipeline only allow six predefined, intermediate concepts? Or can it be extended to more concepts? If not, it seems the proposed pipeline only works under a very restrictive scenario.
-
“Supervision control”: The loss notations, e.g., L_{total}, L_{scallop}, L_{structural}, L_{color}, L_{visual}, L_{perception}, are never defined in the paper.
-
I have a question about the intuition of the paper: Why do you expect that the proposed pipeline—training a neuro-symbolic model in an end-to-end manner with final labels—can recover intermediate concept labels? Intuitively, this would lead to the shortcut issue, and the intermediate concepts may learn incorrect labels.
-
Experimental results: Could the authors explain why the handover-clean-0% case has extremely low performance, i.e., 0.093?
4 comments
2/2
Q5. Loss terms are never defined. A new appendix section defines each term.
L_scallopis a class-rebalanced binary cross-entropy between the frame-level rule outputs and the cheap label (positive frames weighted 4×), summed over the rule hierarchy with annealed weights.L_perceptionis the focal loss on cells, active only on batches with expensive labels.L_coloris a cross-entropy between each role channel and a pseudo-label computed from the fraction of pixels in a cell that match the known agent color, plus an exclusion term.L_structuralis a sum of penalties for scene properties (one cell per agent, possession located at an agent, roles adjacent in handover frames, and others, each listed).L_visualis the prototype-based consistency loss on possession.Q6. Why should training with final labels recover the intended concepts rather than shortcuts? This is the central question, and the reviewer's skepticism is supported by our own data in two ways.
First, the final label passed through the rules alone does produce a shortcut. In the Scallop-only ablation, now run with three seeds and scored against ground truth, the role channels are uncorrelated with the agents (CC −0.08 and 0.14) and the possession map never overlaps the holder (IoU at most 0.01), on every seed (Table 4). This agrees with the analysis of reasoning shortcuts by Marconato et al. (2023): constraints alone leave concept semantics underdetermined.
Second, even with the additional knowledge, recovery is only partly achieved. The intended solution is singled out by three constraints: the rules force the decision through a possession transfer between two differently-roled, adjacent agents; the color prior ties the role channels to the agents' appearance; and the structural constraints remove most remaining assignments (one cell per agent, possession located at an agent). With these, the roles are recovered on every seed (CC 0.93 and 0.91). Possession, the one concept that has no prior of its own, is recovered in three of six runs and remains diffuse in the others. The submitted paper claimed more than this (CC 0.999); the revision reports the per-seed values and states that purely abductive recovery of possession is unreliable in our setup. The argument and its limit are now in Sections 3 and 5.1.
Q7. Why is "handover-clean-0%" only 0.093? That value is not a detection score. It is the correlation between the predicted
handoverprobability map and the ground-truth handover cells (0.207 as a three-seed mean in the corrected table). The detection F1 of the same models on clean data is 0.973. Thehandovermap correlates poorly because its ground truth covers several adjacent cells during the transfer, and without cell-level labels nothing tells the model which of them to mark; the prediction is centered near the right place (normalized localization error 0.037, against 0.009 at 50% annotation) but spread over neighboring cells. Detection is not affected, because decisions are read from theholds_transferrule, which does not use this channel. The table caption and text now make the distinction explicit."The reported performance is not convincing." We agree in part, and the corrected results are more modest than the submitted ones. With three seeds for every model, our approach is clearly better than all neural baselines only on the most complex test set, comparable in distribution, and worse under occlusion and recoloring. The revision states this in the abstract, with per-seed results for every model in the appendix.
I thank the authors for their responses.
Q1 and Q2: The authors' responses have addressed my concerns.
Q4: I understand that, in principle, the proposed method can be applied to a larger number of concepts. However, it seems that the experiments only consider six concepts. Is that correct? More broadly, has the proposed method been evaluated only in the constructed synthetic setting? If so, I remain concerned about its generalizability and practical value.
Q6: I think the authors' response actually reinforces my concern. As the authors stated, "the final label passed through the rules alone does produce a shortcut and even with the additional knowledge, recovery is only partly achieved." Given this observation, I would appreciate a clearer intuitive explanation of why the proposed method is expected to work.
Q7: The (updated) empirical results do not seem sufficiently convincing, even in the synthetic setting.
Overall, these remaining concerns, together with the substantial revisions that appear necessary (including clarifying the notation for all loss terms and updating the experimental results), lead me to maintain my original score.
We thank the reviewer for the follow-up and are glad that Q1 and Q2 are resolved.
Q4. Yes. The experiments use six perception channels (two roles, possession, handover, and two activity states), from which the Scallop program derives further intermediate relations such as green_holds and holds_transfer. All experiments are on the synthetic testbed. This is deliberate, not incidental: our question is whether the intermediate concepts are recovered correctly, which requires per-frame, per-cell ground truth for every concept. Real video of compositional events does not provide this ground truth, and it does not allow controlled variation of single factors such as the number of agents, occlusion, or movement speed. We state this scope in the introduction and the limitations, and we agree that practical value on real video is not demonstrated in this paper. We see it as a first step towards modeling spatio-temporal interactions with abductive learning, to be extended to real video in future work.
Q6. Why should it work? The cheap label says when a handover happens. The rules say what it is: possession moves from a green-agent cell to an adjacent blue-agent cell between consecutive frames, or vice versa. If the perception model’s possession signal does not switch at a handover, the rule output gives a false negative; if it switches where there is no handover, a false positive. Both contradict the cheap label, and the decision loss corrects them, so the backbone is driven to produce a possession signal that switches at exactly the right frames, across thousands of clips in which the agents are at different positions. The decision loss cannot, however, tell exactly where the signal sits: a signal with the right timing near, but not on, the holder still yields a correct decision.
In the complete model, the auxiliary losses narrow down this location. The color prior ties the role channels to the agents, and the structural constraints force possession to sit at an agent, with one cell per agent. What remains consistent with the rules and all constraints is close to the intended solution. This explains both outcomes we measure. The roles, which are pinned down directly, are recovered on every seed (CC 0.93/0.91). Possession is pinned down only indirectly, through its co-location with an agent and its timing. It is recovered in three of six runs. In the other runs it peaks near the holder (normalized error 0.058, about one cell) without being concentrated on it, while decisions remain correct (F1 of at least 0.94 on clean data). The ablation confirms the role of the auxiliary losses: with the same backbone trained only through the rules, the timing is still learned but the location is arbitrary (IoU at most 0.01).
Q7. We agree that the corrected results do not show a general accuracy advantage, and the revised abstract does not claim one. What they show, with three seeds and a paired bootstrap, is narrower. First, without any intermediate labels the model matches three neural baselines with the same backbone in distribution and is better than all of them on the most complex unseen scenes (0.735 vs. 0.471/0.293/0.524); it is worse under occlusion and recoloring, for a reason the concept metrics identify (the color prior). Second, at the decision level, removing all expensive labels costs little: the 0% and 100% models are within test-set noise on seven of nine sets, and the fully supervised model is clearly better only under occlusion and recoloring. Expensive labels still matter for the concepts themselves (handover-channel CC 0.21 at 0% vs. 0.82 at 100%).
The annotation rate is not a binary choice: we also trained models at 5% and 50%. At 5% the model behaves like the 0% model. At 50%, which in our single-epoch schedule labels about a third of the distinct training clips, it recovers nearly all of the concept quality of the 100% model (possession CC 0.78 vs. 0.83, handover 0.79 vs. 0.82, stable across seeds) and most of its robustness (occlusion F1 0.889 vs. 0.941). A practitioner can therefore choose a rate between these points according to whether decisions alone or also interpretable concepts are needed
Revisions. These revisions are already in the uploaded PDF, not pending. For convenience: preliminaries on VideoMamba and Scallop, p. 3; the task example with both label types, Figure 1, p. 5; concept metrics for the ablation, Table 4, p. 8; definitions of every loss term, Appendix A2 (“Loss Definitions”, p. 14); per-seed results and the bootstrap comparisons, p. 21. All tables are regenerated from the three-seed runs, and the general response summarizes the corrections. We thank the reviewer again for the questions, which led to the concept-level ablation and a more careful statement of our results.
1/2