- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TRACE, a concept bottleneck framework for video activity forecasting that recursively predicts named concept states. It derives concept activations from a frozen vision–language encoder, refines them using sparse spatio-temporal graphs, and forecasts their evolution through activity-conditioned transitions and a shared activity classifier. Its main contribution is an explicit concept-space forecasting interface that supports inference-time edits to concept activations, activity beliefs, and graph relations, with effects propagated to subsequent predictions. Evaluation on BARISTA, Breakfast, and MPII Cooking 2 includes comparisons with concept-based and black-box baselines, component ablations, dependency analyses, and intervention experiments. The results demonstrate competitive forecasting performance and localized dependencies, while showing that interventions can correct or steer predictions, with the strongest correction results relying on oracle information.
Critical strengths
-
A meaningful extension of CBMs from static explanations to editable forecasting. The paper addresses a clear research question: how to maintain an explicit concept interface while predicting future states and propagating interventions over time. The architecture is well aligned with this objective, combining named concept coordinates, sparse relational updates, and a shared activity classifier. This extends the role of concepts beyond explaining a final prediction and provides a coherent mechanism for inspecting and modifying the forecasting process.
-
Extensive and informative empirical validation. The evaluation covers multiple datasets, predictive baselines, component ablations, matched-capacity controls, and intervention analyses. The Jacobian analysis provides concrete evidence of more localized dependencies than dense rollout. Importantly, the authors also report negative results and limitations, making the evaluation useful for understanding the method’s capabilities and trade-offs rather than only its best-performing settings.
Critical weaknesses
-
The added intervention value of the proposed dynamics is not sufficiently isolated. The forecasting gains from recursive transitions and activity feedback are modest in Table 2, while the principal intervention experiments focus on TRACE itself. Although Table 21 establishes more localized dependencies, comparable intervention evaluations against simpler concept-space models are missing. Since those models can also expose editable concept coordinates, the current evidence does not clearly establish whether TRACE’s graph-based dynamics yield more effective or reliable interventions. The relative intervention benefits of the proposed dynamics therefore remain incompletely characterized.
-
The demonstrated correction capability remains strongly dependent on privileged information. The strongest correction results use future concept targets, true future activity labels, or candidate intervention effects. Without such information, Blind Qwen decreases the average true-class probability across all three datasets (Table 4). Moreover, the steering experiment combines concept edits with activity-belief edits, making their respective contributions difficult to distinguish. Oracle evaluation is valid, but these results provide limited evidence for the motivating practical benefit of reliably correcting forecasts from observed information alone.
Questions for authors
-
Normalizer training and gradient flow. Section 3.1 describes the normalizer parameters as learned, whereas Section 3.4 states that they do not update. Are these parameters jointly optimized through the observed-input branch, with stop-gradient applied only to future targets, or are they trained separately and subsequently frozen? Please specify the actual optimization procedure and gradient paths.
-
Evidence supporting the architectural contribution. Beyond the localized dependencies demonstrated in Table 21, what evidence establishes an intervention advantage over simpler editable concept-space models? Are matched intervention results available for the sparse-linear or graph-free baselines under the same information access and editing budget? This comparison would help determine whether the proposed dynamics improve intervention effectiveness or reliability beyond providing editable coordinates.
-
Steering results. In Table 5, interventions include both concept-node edits and activity-belief edits involving the requested target activity. How much of the steering success comes from each intervention type? Can the reported results or a targeted breakdown establish whether concept edits provide additional benefit over activity-belief edits alone?
-
Section 4.2 states that node interventions use future H3 concept targets, whereas Appendix E.1.1 describes clamping the H1 state using concept targets from the next video window. Which horizon supplies the replacement concept values, and is the H3 ground-truth label used only for ranking candidate edits?
Response to Reviewer h498
We thank the reviewer for the positive assessment. We are glad that the paper is seen as a meaningful extension of CBMs from static explanations to editable forecasting, and that the empirical validation, including its reported limitations, is found extensive and informative.
Below we address each point (changes are marked in orange in the updated PDF):
The normalizer parameters are learned: in all nine main checkpoints they are trainable, appear in the optimizer and moved from their initialization, with gradients arriving through the observed-concept branch. We clarified the Sec. 3.4 sentence that only refers to the future-concept target, which is computed under a stop-gradient.
Appendix E.1.1 (E.2.1 in the revised PDF) is correct: the code clamps the H1 state to the concept target of the next window and ranks candidate coordinates by their effect on the true H3 label. We corrected Sec. 4.2.
We ran the same oracle concept-edit protocol on Sparse Linear (128 concepts, same features) and on two dense rollout controls, one structural and one with exactly TRACE's parameter count, on the same 60 cases per seed. Budget 5, H3 (%):
The same five edits correct 2–16× more wrong forecasts in TRACE, never harm a correct one, and stay local. The exact-capacity dense model, which is slightly more accurate than TRACE on BARISTA, is the least intervenable.
These results are for concept edits only. For belief edits, the exact-capacity dense model with the same feedback interface is on par with TRACE (7.7 / 65.6 / 40.1% vs. 8.0 / 63.3 / 46.0% of wrong forecasts corrected). The sparse graph therefore improves the concept interface, not the belief interface.
Belief edits provide the strongest steering signal, achieving 85–90% success on Breakfast and MPII. Concept edits offer complementary control: when selected by their measured effect on the requested activity in the model, concept-only edits reach 58–64%, and combining concept and belief edits increases success to 94% on both datasets. In contrast, concept edits chosen by the heuristic or the Qwen agent reach only 4–12%, the level of random selection. This highlights the importance of selecting relevant concepts rather than a limitation of the intervention mechanism itself. We added this point to the limitations.
Correction also does not require future information: a user who only confirms the current activity already corrects 35/23% of wrong forecasts on Breakfast/MPII with one belief edit, and 63/46% when also supplying the next two activities (point 4 of our response to Reviewer fLCp).
Taken together, the two interfaces are complementary, and which one helps depends on how persistent the activities are. On Breakfast and MPII, the current activity is informative about the future, and belief edits are the effective interface. On BARISTA the activity changes quickly: among the wrong forecasts we edit, the current activity equals the H3 activity in only 8% of cases (58% / 37% on Breakfast / MPII). There, belief edits have little effect (8% corrected) and concept edits are the stronger interface (20%).
We hope this clarifies the addressed points and are happy to answer further questions for the remainder of the discussion period.