acceptodds
Under review as a conference paper at ICLR 2027

Intervenable Concept Dynamics for Video Activity Forecasting

Abstract

Concept bottleneck models route predictions through human-understandable concepts, making their intermediate states inspectable and editable. Extending this model family to forecasting requires modeling how concept activations evolve over time as objects, actions, and their interactions change, yet existing CBMs ground concepts only in observed inputs and do not forecast their future evolution. We study this problem using videos of ongoing activities and forecast future activities through a named-concept state. We propose TRACE-CBM (Temporal Relational Activity Concept Evolution), which first refines observed concept activations with sparse spatio-temporal graphs. It then recursively predicts future concept states, conditioning each transition on the prior activity distribution, and decodes both observed and predicted states with a shared activity classifier. At inference time, the model supports editing concept values, activity activations, and learned edges without parameter updates, exposing the relational spatio-temporal dependencies used for forecasting. Across multiple datasets, TRACE improves observed-activity Top-1 accuracy over concept-based methods by 1.2–8.5 percentage points and attains H3 forecast Top-1 accuracy within 1.9–2.2 points of matched black-box forecasters.

Primary area:
interpretability and explainable AI
Keywords:
Concept bottleneck models, video activity forecasting, model interventions.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_h498OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TRACE, a concept bottleneck framework for video activity forecasting that recursively predicts named concept states. It derives concept activations from a frozen vision–language encoder, refines them using sparse spatio-temporal graphs, and forecasts their evolution through activity-conditioned transitions and a shared activity classifier. Its main contribution is an explicit concept-space forecasting interface that supports inference-time edits to concept activations, activity beliefs, and graph relations, with effects propagated to subsequent predictions. Evaluation on BARISTA, Breakfast, and MPII Cooking 2 includes comparisons with concept-based and black-box baselines, component ablations, dependency analyses, and intervention experiments. The results demonstrate competitive forecasting performance and localized dependencies, while showing that interventions can correct or steer predictions, with the strongest correction results relying on oracle information.

Critical strengths
  • A meaningful extension of CBMs from static explanations to editable forecasting. The paper addresses a clear research question: how to maintain an explicit concept interface while predicting future states and propagating interventions over time. The architecture is well aligned with this objective, combining named concept coordinates, sparse relational updates, and a shared activity classifier. This extends the role of concepts beyond explaining a final prediction and provides a coherent mechanism for inspecting and modifying the forecasting process.

  • Extensive and informative empirical validation. The evaluation covers multiple datasets, predictive baselines, component ablations, matched-capacity controls, and intervention analyses. The Jacobian analysis provides concrete evidence of more localized dependencies than dense rollout. Importantly, the authors also report negative results and limitations, making the evaluation useful for understanding the method’s capabilities and trade-offs rather than only its best-performing settings.

Critical weaknesses
  • The added intervention value of the proposed dynamics is not sufficiently isolated. The forecasting gains from recursive transitions and activity feedback are modest in Table 2, while the principal intervention experiments focus on TRACE itself. Although Table 21 establishes more localized dependencies, comparable intervention evaluations against simpler concept-space models are missing. Since those models can also expose editable concept coordinates, the current evidence does not clearly establish whether TRACE’s graph-based dynamics yield more effective or reliable interventions. The relative intervention benefits of the proposed dynamics therefore remain incompletely characterized.

  • The demonstrated correction capability remains strongly dependent on privileged information. The strongest correction results use future concept targets, true future activity labels, or candidate intervention effects. Without such information, Blind Qwen decreases the average true-class probability across all three datasets (Table 4). Moreover, the steering experiment combines concept edits with activity-belief edits, making their respective contributions difficult to distinguish. Oracle evaluation is valid, but these results provide limited evidence for the motivating practical benefit of reliably correcting forecasts from observed information alone.

Questions for authors
  • Normalizer training and gradient flow. Section 3.1 describes the normalizer parameters as learned, whereas Section 3.4 states that they do not update. Are these parameters jointly optimized through the observed-input branch, with stop-gradient applied only to future targets, or are they trained separately and subsequently frozen? Please specify the actual optimization procedure and gradient paths.

  • Evidence supporting the architectural contribution. Beyond the localized dependencies demonstrated in Table 21, what evidence establishes an intervention advantage over simpler editable concept-space models? Are matched intervention results available for the sparse-linear or graph-free baselines under the same information access and editing budget? This comparison would help determine whether the proposed dynamics improve intervention effectiveness or reliability beyond providing editable coordinates.

  • Steering results. In Table 5, interventions include both concept-node edits and activity-belief edits involving the requested target activity. How much of the steering success comes from each intervention type? Can the reported results or a targeted breakdown establish whether concept edits provide additional benefit over activity-belief edits alone?

  • Section 4.2 states that node interventions use future H3 concept targets, whereas Appendix E.1.1 describes clamping the H1 state using concept targets from the next video window. Which horizon supplies the replacement concept values, and is the H3 ground-truth label used only for ranking candidate edits?

1 comment
AuthorsOpenReview ↗

Response to Reviewer h498

We thank the reviewer for the positive assessment. We are glad that the paper is seen as a meaningful extension of CBMs from static explanations to editable forecasting, and that the empirical validation, including its reported limitations, is found extensive and informative.

Below we address each point (changes are marked in orange in the updated PDF):


  1. Normalizer training and gradient flow, Sec. 3.1 vs 3.4 (Q1).

The normalizer parameters are learned: in all nine main checkpoints they are trainable, appear in the optimizer and moved from their initialization, with gradients arriving through the observed-concept branch. We clarified the Sec. 3.4 sentence that only refers to the future-concept target, which is computed under a stop-gradient.

  1. H1 vs H3 node targets, Sec. 4.2 vs Appendix E.1.1 (Q4).

Appendix E.1.1 (E.2.1 in the revised PDF) is correct: the code clamps the H1 state to the concept target of the next window and ranks candidate coordinates by their effect on the true H3 label. We corrected Sec. 4.2.

  1. Intervention advantage over simpler editable concept models (W1, Q2).

We ran the same oracle concept-edit protocol on Sparse Linear (128 concepts, same features) and on two dense rollout controls, one structural and one with exactly TRACE's parameter count, on the same 60 cases per seed. Budget 5, H3 (%):

ModelFinal W→C (BARISTA / Breakfast / MPII)Final C→WMean shift of non-edited concepts
TRACE20.0 / 30.7 / 28.00 / 0 / 00.003 / 0.010 / 0.007
Sparse Linear10.1 / 13.7 / 3.20.6 / 1.4 / 0.00.042 / 0.039 / 0.026
Dense rollout, structural8.1 / 6.5 / 5.22.2 / 4.5 / 2.80.022 / 0.020 / 0.023
Dense rollout, exact capacity6.9 / 4.3 / 1.72.2 / 3.1 / 2.80.022 / 0.017 / 0.020

The same five edits correct 2–16× more wrong forecasts in TRACE, never harm a correct one, and stay local. The exact-capacity dense model, which is slightly more accurate than TRACE on BARISTA, is the least intervenable.

These results are for concept edits only. For belief edits, the exact-capacity dense model with the same feedback interface is on par with TRACE (7.7 / 65.6 / 40.1% vs. 8.0 / 63.3 / 46.0% of wrong forecasts corrected). The sparse graph therefore improves the concept interface, not the belief interface.

  1. Share of steering success from concept vs belief edits (W2, Q3).

Belief edits provide the strongest steering signal, achieving 85–90% success on Breakfast and MPII. Concept edits offer complementary control: when selected by their measured effect on the requested activity in the model, concept-only edits reach 58–64%, and combining concept and belief edits increases success to 94% on both datasets. In contrast, concept edits chosen by the heuristic or the Qwen agent reach only 4–12%, the level of random selection. This highlights the importance of selecting relevant concepts rather than a limitation of the intervention mechanism itself. We added this point to the limitations.

Correction also does not require future information: a user who only confirms the current activity already corrects 35/23% of wrong forecasts on Breakfast/MPII with one belief edit, and 63/46% when also supplying the next two activities (point 4 of our response to Reviewer fLCp).

Taken together, the two interfaces are complementary, and which one helps depends on how persistent the activities are. On Breakfast and MPII, the current activity is informative about the future, and belief edits are the effective interface. On BARISTA the activity changes quickly: among the wrong forecasts we edit, the current activity equals the H3 activity in only 8% of cases (58% / 37% on Breakfast / MPII). There, belief edits have little effect (8% corrected) and concept edits are the stronger interface (20%).


We hope this clarifies the addressed points and are happy to answer further questions for the remainder of the discussion period.

Reviewer_uQJFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TRACE-CBM (Temporal Relational Activity Concept Evolution), an autoregressive Concept Bottleneck Model (CBM) designed for activity recognition and anticipation. While standard CBMs typically focus on extracting concepts from the observed visual input, TRACE-CBM models the evolution of the concepts over a longer temporal horizon, maintaining human-readable concepts traces. Users can edit and intervene at inference time on the predicted concepts and actions. A single shared decoder maps both observed and forecasted concept states into activity labels. The model is evaluated on MPII Cooking 2, Breakfast and BARISTA, outperforming prior concept-based baselines in activity recognition and anticipation. Intervention experiments through oracle and VLM-guided approaches demonstrate that test-time edits can effectively correct initial mistakes and steer them toward the target activities.

Critical strengths
  • The problem is well-motivated and addresses a key limitation in the recent works on Concept Bottleneck Models by learning to predict concepts' evolution over time. The architecture is clean and well presented.

  • The paper presents an extensive evaluation, showing the effectiveness of edits and interventions on the predicted concept representations, and how such edits affect the model predictions on the downstream task. The paper also discusses how VLMs can be used to perform edits automatically, which is very effective for measuring how TRACE-CBM responds to such interventions.

Critical weaknesses
  • The formulation as a spatio-temporal graph is somewhat misleading. The edges connect concepts from the same window or across windows. Visual embeddings are extracted from temporal windows, lacking any form of spatial or fine-grained temporal grounding, and concepts across windows are connected based only on their temporal distance. Conceptually, the choice of a graph is therefore questionable. Furthermore, Table 6 (appendix) shows very limited improvements when different configurations for the message passing network and the graph topology are compared, further supporting the argument against the message-passing network.

  • There is no direct evaluation of the quality and interpretability of the refined concepts predicted by the model. The concepts are evaluated only through downstream tasks and intervention analysis. A user study (or a more affordable LLM-as-a-judge analysis) could offer a better understanding of the quality of the predicted concepts.

  • TRACE-CBM produces a C-dimensional vector indexed by the concept names, but the message-passing modules and the action-to-concepts feedback approach introduce significant concept mixing at every layer, which impacts the interpretability of the concept predictions and how they can be uniquely identified. How does TRACE-CBM address the correctness of the predicted concepts?

  • How robust is TRACE-CBM with respect to the choice of the concept vocabulary? Sec. C.1 describes the procedure to produce a concepts list through Qwen. Would it be possible to produce a more task-agnostic concepts list, like a general list that can be reused across datasets and tasks?

  • [Minor] The setting and experimental validation described in the abstract are vague. Specifically, the abstract reports numeric improvements without introducing the evaluation datasets and the evaluation metrics, such as H3, which makes it somewhat confusing.

Questions for authors
  • How does TRACE-CBM address the correctness of the predicted concepts?
  • How robust is TRACE-CBM with respect to the choice of the concept vocabulary?
  • Would it be possible to produce a more task-agnostic concepts list, like a general list that can be reused across datasets and tasks?
1 comment
AuthorsOpenReview ↗

Response to Reviewer uQJF

We thank the reviewer for the clear assessment. We are glad that the problem is considered well-motivated and a step beyond static CBMs, that the architecture is found clear, and that the evaluation of edits and interventions is seen as extensive. We also thank the reviewer for the question about a task-agnostic concept list, which we tested directly.

Below we address each point (changes are marked in orange in the updated PDF):


  1. Spatio-temporal graph, small Table 6 gains (W1).

We agree with the wording: nodes are concept scores of pre-extracted temporal windows and edges are learned sparse dependencies within and across windows, without spatial structure. We clarified this in the manuscript. Message passing barely changes accuracy and we do not claim it does. Instead, the role of the sparse graph is to keep interventions local. Compared to baselines without a graph (point 3 of our response to Reviewer h498), TRACE's edits move the non-edited concepts 2–16× less than Sparse Linear or a parameter-matched dense model and correct 2–16× more wrong forecasts. We now state the role of the graph more clearly in the paper.

  1. Correctness of the predicted concepts (W2, W3, Q1).

During training, TRACE supervises predicted concepts against actual future-window scores using a smooth L1 loss (Sec. 3.4). The errors on the predicted concept scores are low (0.016 / 0.014 / 0.016 for BARISTA / Breakfast / MPII). Because that supervision comes from the same extractor, we evaluated the predicted states with two independent checks. First, Gemma-3-27B, a VLM from a different family than our vocabulary generator, judged concept presence in the actual observed, H1 and H3 frames of 300 windows (4,500 judgments), without access to labels and TRACE scores. Second, one author hand-labelled a random subset of 240 of these judgments. Each entry is the AUROC of TRACE's concept scores against presence labels obtained from the real frames of that window; for H1 and H3 these are future frames the model never saw.

Labels fromObservedTRACE H1TRACE H3
Gemma judge (BARISTA / Breakfast s1 / MPII)0.53 / 0.64 / 0.580.61 / 0.61 / 0.580.54 / 0.65 / 0.59
Human judge (pooled)0.750.710.77

This comparison also addresses the concern that message passing and activity-to-concept feedback could change what a coordinate means (W3): the observed column contains the extractor scores before either is applied, and the forecast columns contain the states after both. The forecast keeps the grounding of its input. Against the human labels the forecast states reach 0.74 on the pooled H1 and H3 items, the same as the observed scores and above the judge itself, which turns out to be conservative (90% precision, 62% recall against the author). After edits, more than 99% of unedited coordinates keep their label. The forecast is thus as grounded as the extractor scores it is trained on, and that extractor remains the main limiting factor.

  1. Vocabulary robustness and a task-agnostic vocabulary (W4, Q2, Q3).

To test whether TRACE also works with a task-agnostic vocabulary, we built one generic 128-concept vocabulary shared over all datasets (body parts, generic objects, materials, container states, spatial relations, hand-object interactions; no activity names) and retrained TRACE. Recognition and H3 accuracy stay within 2.5 points of the dataset-specific vocabularies on every dataset, with no consistent direction, and the correction rates stay in the same range (18–38% vs. 26–34%). This indicates that TRACE is robust with respect to the specific concept vocabulary and can also work with more general vocabularies. We added the new vocabulary and the results in Appendix A.7.

  1. Abstract without dataset or metric names (W5, minor).

We revised the abstract: it now describes the findings without isolated numbers, which are reported with their dataset and metric in Sec. 4.


We hope this clarifies the addressed points and would be glad if the reviewer considered raising the score. We are happy to answer any further questions for the remainder of the discussion period.

Reviewer_fLCpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies video activity forecasting through editable concepts. TRACE-CBM extracts concept scores with a frozen vision-language model, refines them with sparse temporal graphs, and predicts future concept states step by step. Previous activity predictions feed back into these states, and a shared classifier predicts activities from them. The model supports edits to concepts, activity beliefs, and graph edges without retraining. Experiments examine prediction accuracy, correction of forecast errors, and steering toward a requested activity. The main contribution is extending the concept-editing interface to future states.

Critical strengths
  • The problem is well-motivated. A user may want to inspect or correct the intermediate state behind a forecast. Keeping an explicit concept state throughout prediction provides a concrete interface for doing this.

  • The evaluation includes common frozen features, multiple seeds, component and loss ablations, and a control with the same parameter count. The appendix also reports negative results, background-class effects, and unstable long rollouts. These details help assess the method fairly.

  • Table 1 shows improvements of 1.2-3.1 percentage points in current-activity accuracy over MoTIF. Table 21 also shows that predicted concept states have more concentrated dependencies than the dense control on the sampled windows. This supports the value of a more inspectable representation.

Critical weaknesses
  • Novelty is mainly incremental on common practices [1, 2]. Preserving an editable state during recursive video forecasting is a useful extension. However, the paper needs clearer evidence of what its particular combination adds. Its Sparse Linear baseline already predicts successive named concept states through a sparse matrix and shared classifier, so concept and connection edits are possible there too.

  • The extra forecasting benefit is small. In Table 2, the full model exceeds concept persistence by only 1.4, 0.0, and 0.5 points on BARISTA, Breakfast s1, and MPII. Activity feedback adds only 0.4, 0.4, and 0.0 points. The control with the same parameter count outperforms TRACE on BARISTA H3 forecasting, 70.37% versus 67.36% (Table 11). TRACE also trails Sparse Linear on BARISTA, 67.6% versus 68.5% (Table 1). These results support modest predictive gains from the distinctive components.

  • Practical forecasting quality needs stronger evidence. On BARISTA, 64.68% of H3 targets are background, while TRACE achieves about 67.4% overall accuracy. Its accuracy on actual activities is only 25.48%, with 5.65% macro-F1 (Table 13). The paper should show performance specifically when the activity changes. Table 14's strong persistence result uses the observed ground-truth activity label, which gives it extra information; a fair practical comparison should also repeat the model's predicted current label.

  • The correction results are upper-bound tests with privileged information. The concept-edit experiment uses future concept values and the true H3 label to choose edits. This follows the oracle intervention setup discussed in the paper, but leaves practical correction unproven. Without future labels or edit outcomes, blind Qwen lowers true-label probability by 2.9, 13.0, and 9.0 points across the three datasets (Table 4). Also, the supplied evaluator counts a case as cumulatively corrected once it becomes correct at any edit step, even if later edits make it wrong again. Thus, the reported correction rate does not necessarily equal the fraction still correct after five edits. The abstract should state these limits.

  • The steering headline needs qualification. The 90.0% result comes from the simple target heuristic on Breakfast (Table 5). Its allowed edits include directly increasing belief in the requested activity, as well as changing concepts. This makes it unclear how much success comes from concept dynamics. The heuristic also beats Qwen on Breakfast and MPII. Separate concept-only and activity-belief-only results are needed to support the claim about concept steering.

  • Named states are not enough to establish that concept meanings remain reliable. Table 27 audits the vocabulary using automated checks, while future targets come from the same vision-language scoring and learned normalization pipeline. Neither establishes that a forecast value still means what its concept name says after graph updates and activity feedback. The paper itself acknowledges that concept states can carry information beyond their intended meanings. Independent concept annotations at H1-H3 would provide stronger evidence. The concentrated dependencies in Table 21 support inspectability, but much of their influence is the concept's own retained state.

  • Computational efficiency is not demonstrated. The trainable models are small, about 0.27-0.30 million parameters (Table 11), and frozen features can be reused. However, Appendix D.3 explicitly states that sparse masks are applied to dense matrices, so they do not provide sparse-computation savings. With one graph layer, rollout costs O(HN(C^2 + KC)), where H is the forecast length, N the context length, C the number of concepts, and K the number of activity classes. No measured latency or memory comparison is reported. The cost of feature extraction, vocabulary construction, and intervention planning also needs to be quantified.

[1] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural Relational Inference for Interacting Systems. ICML, PMLR 80:2688-2697, 2018.

[2] Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. CoPhy: Counterfactual Learning of Physical Dynamics. ICLR, 2020.

Questions for authors
  • How well does TRACE predict actual activity changes? Please report results on changed-label windows, non-background activities, and a baseline that repeats the model's current prediction. Keep the ground-truth-label persistence diagnostic separate.

  • What are the final correction and harm rates after five edits, rather than the rates of ever becoming correct or incorrect during editing? Can a policy using only observed information improve net accuracy, and how would it decide when to stop?

  • Do predicted and edited concepts retain their intended meanings? Please measure concept accuracy against independent annotations across forecast steps and check whether the unchanged concepts remain sensible after an edit.

  • How much of Table 5's steering success remains with concept edits alone? Please compare concept-only, activity-belief-only, and combined edits, and assess whether the resulting concept sequences are plausible.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fLCp (1/2)

We thank the reviewer for the constructive review, and are happy that the problem is seen as well-motivated, the concept state as a concrete interface for inspecting and correcting forecasts, and the breadth of the evaluation (seeds, ablations, matched control, negative results) as a strength.

Below we address each point (changes are marked in orange in the updated PDF):


  1. Novelty (W1).

TRACE is a concept bottleneck whose forecast state remains a named, editable concept vector at every horizon; neither NRI nor CoPhy exposes a human-readable state editable mid-rollout. The Sparse Linear baseline already builds on TRACE's construction of concept-evidence windows, so Table 2 only compares different forecasting heads on the same concept interface. Thus, Sparse Linear already relies on parts of TRACE and can also be seen as a simpler, less intervenable alternative contribution.

  1. Forecasting gains (W2).

We partly agree that, on accuracy, TRACE has only minor gains over concept persistence and Sparse Linear, and we now state this more clearly in the paper. However, our claim is intervenability at matched accuracy. We compared TRACE and the two baselines regarding intervenability, which shows that in the same samples and intervention settings, TRACE corrects between 2 and 16 times more wrong forecasts with substantially smaller side effects (details in point 3 of our h498 response).

  1. Forecasting of actual changes (W3, Q1).

We evaluated all test windows on label changes and non-SIL targets. The table shows H3 change top-1 / non-SIL top-1 / non-SIL macro-F1 (%):

BARISTABreakfast s1MPII
TRACE forecast21.0 / 25.0 / 5.516.1 / 44.3 / 23.716.3 / 36.4 / 11.1
Sparse Linear23.7 / 27.4 / 5.715.2 / 43.9 / 20.716.9 / 35.8 / 9.6
Feature Transformer29.1 / 32.6 / 5.817.5 / 46.0 / 18.818.6 / 38.0 / 9.8
SlowFast-TCN30.7 / 32.1 / 7.118.2 / 45.6 / 19.019.3 / 37.7 / 10.1

Black-box models are stronger on changed windows on BARISTA and MPII; on non-SIL macro-F1 TRACE is best on Breakfast and MPII. TRACE and Sparse Linear are within about two points in every column, while TRACE is more intervenable.

  1. Final rates and correction without privileged information (W4, Q2).

We now report whether a forecast is correct after the final edit. For the concept oracle, this changes the correction rate by at most 0.7 percentage points; no initially correct forecasts become wrong.

The interventions are meant to let a user add knowledge the model lacks, so we now state what information each policy uses and report the final wrong-to-correct rates (%):

Policy (information available)BARISTABreakfast s1MPII
Observed concepts only, 5 edits2.71.34.0
Blind Qwen agent (Table 4)10.013.312.2
Current activity only, 1 belief edit3.335.323.3
Current and next two activities, 3 belief edits8.063.346.0
Concept oracle, future targets, 5 edits20.030.728.0

Correction does not require future information: a user who only confirms the current activity, which is visible in the observed window, corrects 35.3% / 23.3% of wrong forecasts on Breakfast / MPII with a single belief edit (harm 3.3 / 2.7%). Also supplying the next two activities raises this to 63.3% / 46.0% (harm 4.7 / 3.3%). The last row additionally assumes future concept values and is an upper bound for concept edits. These work best on BARISTA, where belief edits have little effect. The two automatic policies in rows one and two do not yet select helpful edits: the blind Qwen agent is no better than the random-order control (Table 33 in the revised PDF). This is a limitation of how edits are chosen, not of the interface.

AuthorsOpenReview ↗

Response to Reviewer fLCp (2/2)

  1. Concept-Only and Activity-Belief-Only Steering (W5, Q4).

To compare steering effectiveness via different changes in the model, we ran the same experiment with separate concept and activity-belief updates. Target top-1 rate (%) on BARISTA / Breakfast s1 / MPII:

PolicyConcept edits only (5)Belief edits only (3)Combined (5 edits)
Target heuristic (Table 5)0.7 / 10.3 / 4.04.7 / 90.0 / 85.34.0 / 88.0 / 83.7
Qwen agent0.7 / 12.3 / 5.34.7 / 90.0 / 85.37.7 / 82.7 / 78.0
Model-guided search (edits ranked by measured effect on the target)10.7 / 64.0 / 58.38.0 / 90.3 / 86.320.0 / 93.7 / 94.0

The first two rows choose edits without querying the model. The last row ranks each candidate edit by its measured effect on the requested activity in the model and applies the top five; it uses no future information. Belief edits act on the requested activity directly and explain the 90% in Table 5. Concept edits alone steer the forecast when selected by this search (58–64% on Breakfast and MPII), but not when selected by the heuristic or the Qwen agent, which stay at the level of random selection (0.7 / 7.3 / 7.7%). Combining both raises the search result to 94% on Breakfast and MPII. The limiting factor is therefore the selection of concept edits, not the mechanism. We added this table to the paper. There are example sequences with cocept edits in the appendix, e.g. Breakfast with target fry_egg where the heuristic sets the belief and then switches on oil bottle.

  1. Do predicted and edited concepts keep their meaning? (W6, Q3).

During training, TRACE supervises predicted concepts against actual future-window scores using a smooth L1 loss (Sec. 3.4). The errors on the predicted concept scores are low (0.016 / 0.014 / 0.016 for BARISTA / Breakfast / MPII). To independently assess semantic grounding, we conducted two additional evaluations.

VLM evaluation. Gemma-3-27B independently labeled concept presence in 300 windows (4,500 judgments), without access to TRACE scores or activity labels. Each entry is the AUROC of TRACE's concept scores against presence labels obtained from the real frames of that window; for H1 and H3 these are future frames the model never saw.

Labels fromObservedTRACE H1TRACE H3
Gemma judge (BARISTA / Breakfast s1 / MPII)0.53 / 0.64 / 0.580.61 / 0.61 / 0.580.54 / 0.65 / 0.59
Human judge (pooled)0.750.710.77

Human validation. One author annotated a blinded subset of 240 judgments. Against these labels, TRACE's forecast states reach an AUROC of 0.74, compared to 0.75 for the observed states. The Gemma judge itself reaches 0.69 against the same labels.

Together, these results suggest that predicted concepts maintain semantic grounding comparable to observed concepts. The main limitation for concept grounding in the current state remains the frozen extractor used for concept supervision. We include these results in the paper and discuss the quality and limitations of the concept grounding.

  1. Computational cost (W7).

We benchmarked 100 windows across five seeds on an H200 GPU, using GPU synchronization. Forecasting heads were measured in float32, feature extraction in fp16.

StepLatency per window
PE-L14 extraction (8 frames)19 ms (24 ms incl. preprocessing)
TRACE H1 / H3 / H102.7 / 3.7 / 7.5 ms
Sparse Linear / Transformer / SlowFast-TCN (H3)0.10 / 0.34 / 0.23 ms
GPU memory (all heads)34–40 MB
Concept intervention (5 edits + re-forecast)~25 ms
Belief intervention3.7 ms
Blind Qwen agent (optional)~2 s
Vocabulary construction (once per dataset)~13 GPU-min

Although TRACE is slower than baseline heads, its 3.7 ms H3 forecast is only ~20% of feature extraction time and remains well within real-time requirements. Latency is dominated by GPU dispatch overhead (826 kernel launches, ~2 ms GPU execution), not computation or model size (<0.3M parameters). Kernel counts remain identical across datasets and GPU architectures, suggesting that CUDA graphs or compilation could substantially reduce this overhead.


We hope this clarifies the addressed points and would be glad if the reviewer considers raising their score. In the meantime, we are happy to answer any further questions for the remainder of the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to All Reviewers and AC

We thank all reviewers for their thorough and constructive feedback. We are glad that the problem is considered well-motivated and the explicit, editable concept state a concrete interface for inspecting and correcting forecasts (fLCp, uQJF), that our work is seen as a meaningful extension of CBMs (h498), the positive remarks on the clarity of the architecture and presentation (uQJF, h498), the breadth of the evaluation and the honest reporting of negative results (fLCp, h498).

The reviews contain four main concerns, which we address below briefly, with details in the responses below and in the revised PDF (colored orange). We are happy to go into further depth on any point if requested.


  1. Practical forecasting quality (fLCp).

To test forecasting beyond activity persistence, we additionally evaluated windows where the activity label changes. At H3, TRACE achieves 21.0%, 16.1%, and 16.3% top-1 accuracy on BARISTA, Breakfast, and MPII, respectively. These results are comparable to Sparse Linear, while TRACE additionally provides an interpretable and intervenable forecast state. While the strongest black-box baselines achieve higher accuracy (30.7%, 18.2%, and 19.3%, respectively), TRACE demonstrates competitive performance on activity transitions while retaining its concept-based interpretability and intervention capabilities. We now make this comparison explicit.

  1. Interventions with privileged information and steering (fLCp, h498).

TRACE’s editable state lets users incorporate their knowledge of the activity into a forecast. Confirming only the current activity with a single activity-belief edit corrects 35.3% and 23.3% of initially wrong H3 forecasts on Breakfast and MPII, respectively. Additionally supplying the next two activities raises this to 63.3% and 46.0%. We now report correction rates after the final edit and clarify what information each policy receives. We also add an appendix table separating the two steering interfaces: concept edits alone can steer forecasts toward a requested activity, while activity-belief edits drive most of the steering success.

  1. Added value of the graph dynamics (h498, uQJF).

The graph dynamics provide a more responsive and localized concept-editing interface. In a matched intervention comparison, we apply the same oracle concept edits and budget to TRACE, Sparse Linear, and parameter-matched dense alternatives on the same test cases. TRACE corrects 2–16× more initially wrong forecasts, leaves initially correct forecasts unchanged, and changes the unedited concept state 2–16× less.

  1. Meaning of predicted concepts (fLCp, uQJF).

We added independent checks of whether predicted concepts retain their intended meanings. Gemma-3-27B, from a different model family than the vocabulary generator, judged concept presence from the actual video frames. One author also annotated a blinded subset of 240 judgments. Against these human labels, TRACE’s forecast concept states achieve 0.74 AUROC, close to the observed concept scores (0.75) and above the VLM judge with access to the actual frames (0.69). These results support the semantic grounding of the forecast state, while its quality remains dependent on the frozen concept extractor used for supervision.

In addition, we measured latency and memory (fLCp), tested a generic vocabulary shared across datasets (uQJF), better described the normalizer training and the H1/H3 node-target description (h498), and slightly polished the wording of the editable forecast state (fLCp).


We hope that we have addressed all your concerns meaningfully and concisely, and we would be happy to clarify any further questions that may arise during the remainder of the rebuttal period.

TRACE authors.

Dear reviewers,

Thank you again for your reviews. We have responded to your comments and uploaded a revised version of the paper. We would be grateful if you could let us know before the rebuttal period ends whether our responses address your concerns, or if any questions remain.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.