acceptodds
Under review as a conference paper at ICLR 2027

Abductive learning of intermediate video concepts in a neuro-symbolic AI Architecture

Abstract

A network trained to detect a compositional video event must discover the intermediate concepts that define it from examples alone; when the training data is narrow, it latches onto surface patterns of the training distribution rather than the event itself, and fails on unseen scenarios. Supervising those concepts directly fixes this but demands expensive per-frame, per-region annotation. We take the neuro-symbolic route instead: domain knowledge enters the model as differentiable logic rules and auxiliary symbolic induction losses. We couple a grid-adapted VideoMamba perception backbone with Scallop, a differentiable logic programming framework, and evaluate on a purpose-built synthetic testbed of role-based agents delivering instruments on a discrete grid, in which every intermediate concept has exact per-frame ground truth. With zero intermediate labels, the model reaches Pearson correlation above on all three core channels on the training distribution; of these, only instrument possession is recovered without an appearance prior. On unseen scenario complexity its decisions far exceed purely neural baselines sharing the same backbone ( against on the extreme set), while remaining comparable or worse under pure perceptual degradation such as occlusion; models trained with less expensive supervision produce fewer false positives while recovering the same events. A component ablation separates the symbolic machinery into structural gradient routing from the logic rules and visual grounding from the auxiliary losses. Either alone yields comparable decisions, but only grounding yields interpretable intermediate concepts, and the two come apart under recoloring in both directions: only their combination transfers on the clean recolored set ( against and ), while under combined recoloring and extreme complexity all three degrade and routing alone is best ( against ). Decision accuracy is therefore a poor proxy for concept quality in neuro-symbolic systems.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic AI, Abductive Learning, VideoMamba

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SSdoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

When one wants to train a model to detect compositional video events, what is usually available are the final labels. The intermediate concept labels for every frame and location are typically not available. To address this, this paper proposes a neuro-symbolic model that combines the VideoMamba backbone with Scallop. Experimental results show that the proposed model recovers expensive intermediate concepts from cheap final labels alone.

Critical strengths

The studied problem is interesting, i.e., how to recover expensive intermediate concepts from cheap final labels alone?

Critical weaknesses

The overall writing needs improvement. It is hard to follow the paper. The performance in some cases is not convincing.

Questions for authors
  1. For Section 3, I would suggest giving more detailed descriptions of the selected setting/task, illustrating it with an example. After reading Section 3, all I know is that there might be two agents handing over the instrument, and I do not know what the goal is or what the actual example is.

  2. The method is based on VideoMamba and Scallop, but there is no preliminary information about them.

  3. “Perception module”: The descriptions of the processing procedure are not clear enough; for example, what does the “selective scan” in L139 mean, and what does the notation f in L141 mean? It would also be appreciated if the authors could give more explicit descriptions of how to get the neural output, what each probability in the map means, and what the focal loss is like.

  4. “Symbolic reasoning”: The paper only describes six concepts. Does the proposed pipeline only allow six predefined, intermediate concepts? Or can it be extended to more concepts? If not, it seems the proposed pipeline only works under a very restrictive scenario.

  5. “Supervision control”: The loss notations, e.g., L_{total}, L_{scallop}, L_{structural}, L_{color}, L_{visual}, L_{perception}, are never defined in the paper.

  6. I have a question about the intuition of the paper: Why do you expect that the proposed pipeline—training a neuro-symbolic model in an end-to-end manner with final labels—can recover intermediate concept labels? Intuitively, this would lead to the shortcut issue, and the intermediate concepts may learn incorrect labels.

  7. Experimental results: Could the authors explain why the handover-clean-0% case has extremely low performance, i.e., 0.093?

4 comments

We thank the reviewer for the assessment. We revised Sections 3 and 4 to make the paper self-contained and answer each question below. Please also see the general response: our reruns corrected several results, and some of the reviewer's doubts about the performance were justified.

Q1. What is the task? Please give an example. The task is to detect, in a video clip, the frames in which one agent hands an instrument to an agent of the other role, and in which direction. In our synthetic top-down scenes, a green agent picks up an instrument, walks to a blue agent and hands it over; the blue agent works with it and later returns it. Some pickups end without a handover and some approaches are empty-handed, so the event cannot be detected from motion or proximity alone. There are two kinds of labels. The cheap label is one flag per frame ("a handover happens here") plus its direction. The expensive labels are per-frame, per-cell maps: where each agent is, which cell holds the instrument, and where the handover takes place. The model is trained with the cheap label only and evaluated on whether it nevertheless predicts the expensive maps.

The submission illustrated this only in the appendix (the figures "Example Scenes" and "Architecture Overview"), which was too far from where the task is defined. The example scenes are now Figure 1 in Section 4, with a caption that states the task and shows both label tiers, and the task description in the text is rewritten.

Q2. Preliminaries on VideoMamba and Scallop. Added at the start of Section 3. In brief: VideoMamba splits the clip into spatio-temporal patches and processes the token sequence with Mamba blocks, state-space layers whose state update depends on the current token, at a cost linear in sequence length. Scallop is a Datalog-based logic programming language whose input facts carry probabilities. A rule such as green_holds(x,y,t) = holds(x,y,t) and green(x,y,t) derives new facts; the probability of a derived fact is computed from its proofs with a provenance semiring (we use differentiable top-k proofs). Because this is differentiable, a loss on a derived fact sends gradient to exactly the input facts used in its proofs, and through them to the network.

Q3. "Selective scan", notation f, neural outputs, focal loss. Selective scan is the recurrence inside a Mamba block: tokens are processed in order and the parameters of the state update are computed from the current token. Because every output token mixes information from the whole sequence, a patch feature does not describe only its patch, which is why we concatenate a locally projected copy of the raw patch embedding. We had used f for two things; the per-cell feature is now z_{h,w,t} and the conjunction in Eq. 1 is ⊗. Neural output: for each concept k, a small MLP maps z_{h,w,t} to one logit; its sigmoid p_k(h,w,t) is the predicted probability that concept k holds at cell (h,w) in frame t, and these probabilities enter Scallop as probabilistic facts. Focal loss: FL(p) = −α(1−p)^γ log p for positive cells and −(1−α)p^γ log(1−p) for negative cells, with α = 0.75 and γ = 2; it is used only where expensive labels exist.

Q4. Is the pipeline restricted to six predefined concepts? No. The concepts are defined by the rule program, not by the architecture: each concept is one sigmoid head on the shared backbone plus one input relation in the Scallop program, and adding a concept means adding a head and the rules that use it. The real restrictions are different, and the revision states them: the rules have to be written by someone with domain knowledge, inference cost grows with the number of proofs per window, and, as our reruns show, a concept is only as robust as the prior that grounds it.

Q5. Loss terms are never defined. A new appendix section defines each term. L_scallop is a class-rebalanced binary cross-entropy between the frame-level rule outputs and the cheap label (positive frames weighted 4×), summed over the rule hierarchy with annealed weights. L_perception is the focal loss on cells, active only on batches with expensive labels. L_color is a cross-entropy between each role channel and a pseudo-label computed from the fraction of pixels in a cell that match the known agent color, plus an exclusion term. L_structural is a sum of penalties for scene properties (one cell per agent, possession located at an agent, roles adjacent in handover frames, and others, each listed). L_visual is the prototype-based consistency loss on possession.

Q6. Why should training with final labels recover the intended concepts rather than shortcuts? This is the central question, and the reviewer's skepticism is supported by our own data in two ways.

First, the final label passed through the rules alone does produce a shortcut. In the Scallop-only ablation, now run with three seeds and scored against ground truth, the role channels are uncorrelated with the agents (CC −0.08 and 0.14) and the possession map never overlaps the holder (IoU at most 0.01), on every seed (Table 4). This agrees with the analysis of reasoning shortcuts by Marconato et al. (2023): constraints alone leave concept semantics underdetermined.

Second, even with the additional knowledge, recovery is only partly achieved. The intended solution is singled out by three constraints: the rules force the decision through a possession transfer between two differently-roled, adjacent agents; the color prior ties the role channels to the agents' appearance; and the structural constraints remove most remaining assignments (one cell per agent, possession located at an agent). With these, the roles are recovered on every seed (CC 0.93 and 0.91). Possession, the one concept that has no prior of its own, is recovered in three of six runs and remains diffuse in the others. The submitted paper claimed more than this (CC 0.999); the revision reports the per-seed values and states that purely abductive recovery of possession is unreliable in our setup. The argument and its limit are now in Sections 3 and 5.1.

Q7. Why is "handover-clean-0%" only 0.093? That value is not a detection score. It is the correlation between the predicted handover probability map and the ground-truth handover cells (0.207 as a three-seed mean in the corrected table). The detection F1 of the same models on clean data is 0.973. The handover map correlates poorly because its ground truth covers several adjacent cells during the transfer, and without cell-level labels nothing tells the model which of them to mark; the prediction is centered near the right place (normalized localization error 0.037, against 0.009 at 50% annotation) but spread over neighboring cells. Detection is not affected, because decisions are read from the holds_transfer rule, which does not use this channel. The table caption and text now make the distinction explicit.

"The reported performance is not convincing." We agree in part, and the corrected results are more modest than the submitted ones. With three seeds for every model, our approach is clearly better than all neural baselines only on the most complex test set, comparable in distribution, and worse under occlusion and recoloring. The revision states this in the abstract, with per-seed results for every model in the appendix.

I thank the authors for their responses.

Q1 and Q2: The authors' responses have addressed my concerns.

Q4: I understand that, in principle, the proposed method can be applied to a larger number of concepts. However, it seems that the experiments only consider six concepts. Is that correct? More broadly, has the proposed method been evaluated only in the constructed synthetic setting? If so, I remain concerned about its generalizability and practical value.

Q6: I think the authors' response actually reinforces my concern. As the authors stated, "the final label passed through the rules alone does produce a shortcut and even with the additional knowledge, recovery is only partly achieved." Given this observation, I would appreciate a clearer intuitive explanation of why the proposed method is expected to work.

Q7: The (updated) empirical results do not seem sufficiently convincing, even in the synthetic setting.

Overall, these remaining concerns, together with the substantial revisions that appear necessary (including clarifying the notation for all loss terms and updating the experimental results), lead me to maintain my original score.

We thank the reviewer for the follow-up and are glad that Q1 and Q2 are resolved.

Q4. Yes. The experiments use six perception channels (two roles, possession, handover, and two activity states), from which the Scallop program derives further intermediate relations such as green_holds and holds_transfer. All experiments are on the synthetic testbed. This is deliberate, not incidental: our question is whether the intermediate concepts are recovered correctly, which requires per-frame, per-cell ground truth for every concept. Real video of compositional events does not provide this ground truth, and it does not allow controlled variation of single factors such as the number of agents, occlusion, or movement speed. We state this scope in the introduction and the limitations, and we agree that practical value on real video is not demonstrated in this paper. We see it as a first step towards modeling spatio-temporal interactions with abductive learning, to be extended to real video in future work.

Q6. Why should it work? The cheap label says when a handover happens. The rules say what it is: possession moves from a green-agent cell to an adjacent blue-agent cell between consecutive frames, or vice versa. If the perception model’s possession signal does not switch at a handover, the rule output gives a false negative; if it switches where there is no handover, a false positive. Both contradict the cheap label, and the decision loss corrects them, so the backbone is driven to produce a possession signal that switches at exactly the right frames, across thousands of clips in which the agents are at different positions. The decision loss cannot, however, tell exactly where the signal sits: a signal with the right timing near, but not on, the holder still yields a correct decision.

In the complete model, the auxiliary losses narrow down this location. The color prior ties the role channels to the agents, and the structural constraints force possession to sit at an agent, with one cell per agent. What remains consistent with the rules and all constraints is close to the intended solution. This explains both outcomes we measure. The roles, which are pinned down directly, are recovered on every seed (CC 0.93/0.91). Possession is pinned down only indirectly, through its co-location with an agent and its timing. It is recovered in three of six runs. In the other runs it peaks near the holder (normalized error 0.058, about one cell) without being concentrated on it, while decisions remain correct (F1 of at least 0.94 on clean data). The ablation confirms the role of the auxiliary losses: with the same backbone trained only through the rules, the timing is still learned but the location is arbitrary (IoU at most 0.01).

Q7. We agree that the corrected results do not show a general accuracy advantage, and the revised abstract does not claim one. What they show, with three seeds and a paired bootstrap, is narrower. First, without any intermediate labels the model matches three neural baselines with the same backbone in distribution and is better than all of them on the most complex unseen scenes (0.735 vs. 0.471/0.293/0.524); it is worse under occlusion and recoloring, for a reason the concept metrics identify (the color prior). Second, at the decision level, removing all expensive labels costs little: the 0% and 100% models are within test-set noise on seven of nine sets, and the fully supervised model is clearly better only under occlusion and recoloring. Expensive labels still matter for the concepts themselves (handover-channel CC 0.21 at 0% vs. 0.82 at 100%).

The annotation rate is not a binary choice: we also trained models at 5% and 50%. At 5% the model behaves like the 0% model. At 50%, which in our single-epoch schedule labels about a third of the distinct training clips, it recovers nearly all of the concept quality of the 100% model (possession CC 0.78 vs. 0.83, handover 0.79 vs. 0.82, stable across seeds) and most of its robustness (occlusion F1 0.889 vs. 0.941). A practitioner can therefore choose a rate between these points according to whether decisions alone or also interpretable concepts are needed

Revisions. These revisions are already in the uploaded PDF, not pending. For convenience: preliminaries on VideoMamba and Scallop, p. 3; the task example with both label types, Figure 1, p. 5; concept metrics for the ablation, Table 4, p. 8; definitions of every loss term, Appendix A2 (“Loss Definitions”, p. 14); per-seed results and the bootstrap comparisons, p. 21. All tables are regenerated from the three-seed runs, and the general response summarizes the corrections. We thank the reviewer again for the questions, which led to the concept-level ablation and a more careful statement of our results.

Reviewer_AXjEOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines VideoMamba with Scallop rules to learn intermediate concepts for tool handover detection. It uses frame-level event labels, a color prior and auxiliary constraints instead of dense intermediate labels. Experiments on a synthetic grid environment measure both concept quality and event detection. The main finding is that good event accuracy does not necessarily mean that the intermediate concepts have correct spatial meaning.

Critical strengths
  1. The paper directly evaluates intermediate concepts instead of only reporting final accuracy. The difference between correlation and IoU also shows why one metric is not enough to judge concept quality.

  2. The method has a large advantage on complex scenes.

Critical weaknesses
  1. The mechanism analysis needs more direct concept-level evidence. The paper shows that correct decisions do not imply correct concepts, these ablations should also compare concept quality over the full test set.
Questions for authors

Table 3 compares the impact of different components on the accuracy of handover detection. Would it be possible to further compare how well these configurations—in terms of their predicted role positions and tool-holding status—align with the ground-truth annotations? This would help clarify which components actually improve the intermediate concepts, rather than just the final prediction.

1 comment

We thank the reviewer for the clear summary and the specific question. We agree with its premise: if correct decisions do not imply correct concepts, the ablation has to measure the concepts directly. The submission did not, and the revision does.

Q: How well do the configurations in Table 3 align with the ground-truth role positions and tool-holding status? We retrained all three configurations with three seeds and scored them against the ground-truth grids on every test set (new Table 4; full results in the appendix). On clean data, mean over seeds with [min, max]:

VariantRole positions: CC green / blueTool holding: CCTool holding: [email protected]Decision F1
Full induction0.94 / 0.890.56 [0.18, 0.77]0.39 [0.12, 0.84]0.952 ± 0.062
Scallop only−0.08 / 0.140.05 [0.02, 0.08]0.000.648 ± 0.561
Visual constraints1.00 / 1.000.49 [0.23, 1.00]0.48 [0.21, 0.99]0.292 ± 0.505 (rule), 0.786 ± 0.162 (neural)

For the Scallop-only model, which has no color prior, each seed is scored under both role assignments and the better one is shown.

Three findings follow.

  1. Role positions are recovered whenever the auxiliary losses are present, and not at all without them: with the rules alone, the role channels are uncorrelated with the agents on every seed.
  2. Tool holding is the harder concept. With the rules alone it is never localized (IoU at most 0.01 on every seed and test set). With the auxiliary losses it is localized in some runs and not in others, with or without the rules. This is a weaker result than the submission claimed (CC 0.999), and the paper now reports it per seed.
  3. Decisions and concepts dissociate in both directions. The visual-constraints model has the best role maps and the worst rule-based decisions. Conversely, full-induction runs with a poorly localized possession map (CC about 0.2) still reach F1 of 0.94 or more on clean data.

One result differs from our submitted description. With single runs we had written that rules alone or auxiliary losses alone are each sufficient for good decisions. With three seeds this does not hold: Full induction has the best mean F1 on seven of nine test sets, one of three Scallop-only seeds produces no detection at all, and the transfer rule attached to the visual-constraints model rarely fires. We revised Section 5.3, the abstract and the conclusion accordingly. Our general response lists the other corrections that came out of the reruns.

Reviewer_t9KhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a video model can learn concepts such as agent roles and instrument possession without detailed annotations for every frame. It combines a grid-based VideoMamba model with Scallop logic rules, color priors, and auxiliary training losses. Its main contributions are this learning pipeline and a synthetic video testbed with known intermediate states. Experiments compare different annotation levels, neural baselines, and model components under additional agents, failed transfers, occlusion, and recoloring. The results show improved generalization on several complex scenarios, while the ablations demonstrate that accurate event detection does not necessarily mean the model has learned the intended intermediate concepts.

Critical strengths
  1. The synthetic testbed provides exact ground truth for intermediate concepts, allowing the authors to check whether agent roles and instrument possession are correctly learned. This is valuable because high event-detection accuracy alone can hide incorrect intermediate predictions.

  2. The method reports a substantial improvement on complex unseen scenes without expensive intermediate annotations. On the Extreme set, its F1 reaches 0.803, compared with 0.263 for the best VM baseline in Table 2. This provides promising evidence that symbolic rules and auxiliary priors can improve generalization when detailed labels are limited.

Critical weaknesses
  1. The performance conclusions are stronger than the experimental evidence. Main models use three seeds, while baselines and ablations use only one. On Extreme, the 0% model scores 0.803 ± 0.166 versus 0.738 ± 0.055 at full supervision, leaving the supervision-related gain uncertain. Changing annotation rate also changes how frequently the fixed-weight perception loss is applied, so annotation availability is confounded with training-loss strength. The paper should also report uniquely annotated examples across epochs, since supervising 5% of training steps need not mean annotating only 5% of the dataset. Several written claims conflict with the tables. Section 5.2 says VM-grid performs worse than both neural baselines, although Table 2 shows it outperforming both on Extreme (0.263 versus 0.103 and 0.231) and Extr.+col (0.424 versus 0.000 and 0.232). The broad claim that only the combined method survives recoloring also needs qualification: Tables 3 and 10 show that the visual-constraints-only model achieves 0.763 on Colors through its neural output, while Scallop-only outperforms the full model on Extr.+col, 0.526 versus 0.440. Together, these issues weaken the claimed explanation of which components improve robustness and why less supervision helps.

  2. The central concept-grounding claim lacks sufficient quantitative validation, and important implementation details remain unclear. The component ablations rely mainly on example predictions and attribution plots to distinguish meaningful concepts from spatially meaningless representations. They do not report concept-level correlation, IoU, or localization error for each ablation across seeds and test conditions. Decision F1 and selected visual examples therefore do not establish the paper’s main claim about intermediate representations. Reproduction is also difficult: Section 3 specifies 3×3×7 reasoning windows, whereas Listing 1 defines eight temporal indices, 0–7, without explaining their relationship to the 16-frame clips. The training configuration omits batch size and total epochs, dataset sizes are unspecified, and the repository link remains a camera-ready placeholder. Because all experiments depend on a custom synthetic generator, these omissions prevent independent verification of both the implementation and the reported findings.

Questions for authors
  1. Can you provide matched multi-seed results for the neural baselines and ablations, with paired uncertainty estimates for the claim that reducing expensive supervision improves robustness?

  2. How do the full model, Scallop-only, and visual-constraints variants compare on concept-level correlation, IoU, and localization error across test conditions? Do these measurements support the grounding differences illustrated in the figures?

  3. Does a 5% annotation rate mean a fixed subset of uniquely annotated examples or supervision on 5% of training steps? How many unique examples receive annotations across training, and does the robustness advantage remain when the effective perception-loss weight is controlled?

  4. How should the claim that both components are necessary for recoloring robustness be interpreted, given VC neural’s 0.763 F1 on Colors and Scallop-only’s 0.526 versus the full model’s 0.440 on Extr.+col?

  5. Which temporal window was actually used: seven positions, as stated in Section 3, or eight, as defined in Listing 1? How does it relate to the 16-frame clips, and can you provide the complete configuration, dataset split sizes, and an anonymous implementation?

6 comments

We thank the reviewer for a precise review. Each inconsistency the reviewer identified was real, and following them up led to the corrections in our general response.

W1 / Q1. Seeds and uncertainty. All baselines and ablation variants are now trained with three seeds, like the main models, and every table reports mean ± standard deviation with per-seed values in the appendix. We also add a paired block bootstrap over test events (10,000 resamples). We present it as indicative only: each test set contributes seven blocks from two videos.

The reviewer's doubt about the supervision claim was justified. After correction, the 0% and 100% models differ reliably on two of nine sets, and both favor full supervision: occlusion (0.745 against 0.941) and recoloring (0.315 against 0.663). On the complex sets there is no reliable difference (+0.116 on 4a+FO, +0.016 on Extreme). We removed the claim that less supervision improves robustness from the abstract, Section 5.2 and the conclusion.

Against the baselines the picture is also more mixed than we reported. The retrained baselines are stronger than the submitted single runs, and they vary strongly between seeds (VM-bin on Extreme: 0.190, 0.222, 1.000). Our 0% model is ahead of all three on Extreme (0.735 against 0.471, 0.293, 0.524) and ahead of VM-dir on the four-agent sets, level in distribution, and behind under occlusion, hidden instruments and recoloring. On every set the best single baseline run is at least as good as our worst seed, and we say so.

W1 / Q3. Annotation rate confounded with loss strength; unique annotated examples. The reviewer is right. The perception loss has a fixed weight (λ = 20) and is applied on every ⌊1/r⌋-th batch, so its time-averaged weight is 20·r. We trained two weight-matched controls with three seeds each: 100% annotation with λ = 1 (same average weight as 5%) and 5% annotation with λ = 400 (same as 100%). Neither reproduces the model whose weight it matches. The 100%/λ=1 model is below the 5% model on 4a (0.685 against 0.788) and Extreme (0.572 against 0.759) and above it on Colors; the 5%/λ=400 model differs from the 100% model on six of nine sets and stays weak on Colors (0.148). So the loss weight alone does not explain the differences between rates. With three seeds and 4–18 events per set we do not claim more than that; in particular we do not claim to have isolated the effect of the rate. One caveat is stated in the paper: the control rescales the perception loss only, not a second cell-level loss on the rule outputs that is also active on annotated batches.

On unique examples: all neuro-symbolic models train for a single epoch (6,382 clips drawn with replacement by a class-balanced sampler from 7,776 training clips, batch size 4), so labels do not accumulate over epochs. Simulating the sampler 1,000 times, the 5%, 50% and 100% settings annotate on average 313, 2,609 and 4,328 distinct clips, that is 4.0%, 33.6% and 55.7% of the training clips. The exact draws of the trained runs cannot be replayed, so these are expectations with ranges (appendix table "Distinct training clips").

W1 / Q4. Text–table inconsistencies. Both are resolved by the corrected tables rather than by rewording. (a) VM-grid: the reviewer's reading was right, and it holds after retraining on the same data as the other models. VM-grid is below both sequence baselines on Clean, Irand, Occl. and h30, and above both on 4a+FO, Extreme (0.524 against 0.471 and 0.293) and Extr.+col. The sentence that it is worse than both baselines is removed. (b) Recoloring: the claim that only the combined model survives recoloring does not hold with three seeds. On Colors, Full induction reaches 0.323 ± 0.288, Scallop only 0.488 ± 0.432, and the neural channel of the visual-constraints model 0.455 ± 0.130; no ranking is supported, and the abstract, Section 5.3 and the conclusion have been revised accordingly. The concept metrics show why the full model fails there: its role channels drop from CC 0.93/0.91 on clean data to 0.34/0.55 on the recolored set.

W2 / Q2. Concept-level metrics for the ablation. This was the main missing piece, and it is now Table 4 (all test sets in the appendix). For every variant and seed we report CC and [email protected] against the ground-truth grids, and a temporal score that ignores location (the frame-wise correlation between the most confident role-holds cell and the ground-truth holder). Because the Scallop-only model has no color prior, each of its seeds is scored under both role assignments and the better one is reported. On clean data (mean over three seeds, [min, max]):

VariantCC greenCC blueCC possessionIoU possessiontemporal
Full induction0.94 [0.87, 0.98]0.89 [0.82, 0.97]0.56 [0.18, 0.77]0.39 [0.12, 0.84]0.30 [−0.02, 0.80]
Scallop only−0.08 [−0.15, 0.01]0.14 [−0.03, 0.32]0.05 [0.02, 0.08]0.000.36 [0.29, 0.41]
Visual constraints1.001.000.49 [0.23, 1.00]0.48 [0.21, 0.99]0.14 [−0.44, 0.78]

The Scallop-only channels have no spatial meaning on any seed, yet they carry a weak, consistently positive temporal signal. This supports, in a weaker form than we wrote, that the rules shape when the channels activate but not where. The visual-constraints model has the best role maps but its rule-based decision fails (Table 3). Decision accuracy and concept quality therefore dissociate in both directions. The example and attribution figures remain as single-run illustrations and are labeled as such.

W2 / Q5. Reproducibility. Window: the reasoning window is 3 × 3 cells × 7 frames. The rule file declares eight time indices, but facts and temporal lookups are supplied only for t ∈ {0,…,6}; index 7 is inert, which the appendix now states. Relation to the 16-frame clip: windows are placed at four temporal positions (frames 3, 7, 11, 15), centered on the k highest-scoring cells of an agent-centric priority map (k = 2 in training, 4 at inference); window outputs are written back to the full grid and the frame decision is the maximum over cells. Configuration and data: batch size 4, one epoch, clip stride 2, focal loss α = 0.75, γ = 2; with one epoch no scheduler step or checkpoint selection takes effect, and we removed the sentence on model selection. Each dataset has 40 videos of 500 frames, split 32/8 by video (the submission said 10%; it is 20%). Evaluation protocol: while preparing this answer we found that the text misstated it. Matching tolerance is 12 frames, not four; peak thresholds are fixed per model family, not shared; and metrics are computed on the first 200 clips of each test set (two of eight videos, 4–18 events). All of this is now in Section 4, a dataset table and the limitations. Code: the supplementary archive now contains the model, baseline and evaluation code, the data generator, all configurations, the Scallop program and the raw per-seed results. Datasets and checkpoints exceed its size limit and will be released with the paper.

Thank you for the reply. The extra runs, concept scores, and window details answer several of my questions. I also appreciate the changes to claims about labels and colors.

My main concern is still the small test set. Each condition uses only two videos and 4 to 18 events. Why were only the first 200 clips used? Were the thresholds chosen using development data?

The new scores help. However, the model's ability to track who holds the instrument still varies widely across runs. Role detection seems more stable. The controls do not match both loss weights. So we cannot separate the effect of fewer labels.

These changes improve the paper. Still, the test set is small. The results do not show reliable possession tracking across all runs.

Comments on the paper

2 comments
AuthorsOpenReview ↗

general response

We thank the three reviewers for careful and constructive reviews. To answer them we retrained every baseline and every ablation variant with three seeds, added two control experiments, and re-evaluated all models with concept-level metrics. This changed the paper more than we expected, and we report it plainly.

Corrections to the submitted results. (a) The neuro-symbolic rows of Table 2 mixed training runs: their first-seed entries came from an earlier run, not from the first seed of the three-seed sweep. The revised Table 2 uses the three runs of one sweep. (b) The concept metrics of Table 1 were computed on ten clips with an accumulator that was not reset between test sets; they are recomputed on the full evaluation window with three seeds. (c) The baselines were single runs, and VM-grid had been trained on an older configuration of the data generator; all baselines are retrained with three seeds on the same training set as our model. The corrected numbers are lower for our model and higher for the baselines, and we have revised the claims accordingly.

What the evidence now shows.

  1. Concepts without expensive labels (Table 1). The role channels are recovered reliably (CC 0.93 and 0.91 on clean data at 0% annotation). Instrument possession, the only concept learned without an appearance prior, is recovered in three of six training runs (CC 0.72–0.81) and stays diffuse in the others (about 0.2). The revised text reports this in place of the single value of 0.999.
  2. Decisions (Table 2). Without expensive labels the model matches the three baselines in distribution and is better than all of them on the most complex set (F1 0.735 against 0.471, 0.293 and 0.524; each difference exceeds test-set sampling noise in a paired bootstrap). It is worse than the best baseline under occlusion (0.745 against 0.966), and it fails under recoloring (0.315 against 0.591–0.757), because its role channels are grounded by a color prior tied to the training palette. The revised paper therefore reports recoloring as a limitation of the color prior, not as a case of transfer.
  3. Annotation rate. The 0% and 100% models differ reliably on two of nine sets, occlusion and recoloring, both in favor of 100%. The revised text no longer frames this as "less supervision, more robustness".
  4. Ablation, now with three seeds and concept metrics (Tables 3 and 4; raised by all three reviewers). Rules alone leave the channels without spatial meaning (role CC −0.08 and 0.14, possession IoU at most 0.01 on every seed) and give unstable decisions. Auxiliary losses alone ground the roles (CC 1.00) but do not yield a usable rule-based decision (F1 at most 0.31). Only the combination gives the best decisions, on seven of nine sets. The submitted statement that either function alone suffices for decisions was an artifact of single runs and is reversed.

Other changes. An example figure of the task with both label tiers in the main text (Figure 1); preliminaries on VideoMamba and Scallop; definitions of every loss term; the evaluation protocol as implemented (12-frame matching tolerance, thresholds per model family, 200-clip evaluation window with 4–18 events per test set); dataset sizes, training budget, seeds and compute; per-seed results for every model; and a limitations paragraph on the small evaluation window, seed variance and the color prior. The code, all configurations and the raw per-seed results are in the supplementary archive.

We believe the paper is more modest and more reliable after these changes. Its supported contributions are the testbed with exact concept ground truth, the measured dissociation between decision accuracy and concept quality, and the quantified roles of rules and auxiliary losses.

Update for all reviewers: full validation splits. Following reviewer t9Kh's concern about the small evaluation window, we evaluated all models on the full validation split of every test set (eight videos per set, 633 events in total, against 128 in the 200-clip window), with the same thresholds and matching; the paired bootstrap now resamples whole videos.

Decisions (point 2). The pattern holds. On the extreme set the label-free model reaches 0.70 ± 0.20 against 0.44 (VM-bin), 0.40 (VM-dir) and 0.56 (VM-grid), all differences with 95% intervals above zero; the same holds on 4a+FO (0.80 against 0.49–0.64). It remains worse under occlusion (0.63 against 0.94 for VM-bin) and under recoloring (0.38 against 0.55–0.68). In distribution it matches VM-dir and VM-grid but is slightly below VM-bin (0.92 against 0.99).

Annotation rate (point 3). With more events, more differences become measurable: 100% annotation is better on five sets (clean, random identities, occlusion, h30, recoloring; +0.04 to +0.15), 0% is better on 4a+FO (+0.14), and three sets show no difference (4a, extreme, extreme + recoloring).

The full-split results will replace the window results in the camera-ready version; the conclusions of the revised paper are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.