acceptodds
Under review as a conference paper at ICLR 2027

TFoC: A Feature Gimbal for Radar Semantic Segmentation

Abstract

The proper utilization of spatio-temporal information is essential for Radar Semantic Segmentation (RSS). Classical 3D convolutions or spatio-temporal attention mechanisms can help an RSS model preserve spatio-temporal information to a certain extent. However, since the objects of interest in radar applications are dynamically moving, these "" computational modules are not the optimal choice in the absence of a spatio-temporal alignment mechanism. To address this, we propose TFoC (Temporal Focusing Convolution), motivated by the working principle of a camera gimbal, , dynamically adjusting the lens pose to steadily focus on the region of interest. Inspired by this, the receptive field design of TFoC aims to make spatio-temporal convolution operations focused rather than blind, and the sampling efficiency in the spatio-temporal domain dense rather than sparse. Therefore, TFoC is fundamentally a novel learnable module inspired by physical intuition. To better elucidate the learning mechanism of TFoC, we start from the classical Maximum A Posteriori (MAP) estimation and incorporate Bayesian theory to provide an approximate interpretive perspective for the forward computation logic and loss function design of TFoC.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Spatio-temporal representation, Radar Semantic Segmentation, Temporal Focusing Convolution

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_NFqrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Focusing Convolution (TFoC) for radar semantic segmentation. The method introduces Semantic Alignment-based Focusing (SAF) to identify relevant features from historical radar frames and Trajectory Consistency-based Aggregation (TCA) to incorporate temporal information. An auxiliary Track Loss is proposed to encourage temporal consistency. The authors evaluate the approach on CARRADA, CARRADA-RAC, and KuRALS datasets.

Critical strengths
  1. The paper presents an interesting approach to temporal feature aggregation for radar semantic segmentation. Rather than aggregating information from fixed spatial locations across frames, TFoC learns to focus on relevant historical features. This is particularly meaningful for radar data, where moving targets may occupy different locations across consecutive frames.
  2. The experimental evaluation covers different radar technologies and application scenarios, which is a notable strength. The authors evaluate TFoC on automotive FMCW radar data (CARRADA and CARRADA-RAC) and Ku-band surveillance radar data (KuRALS-CW and KuRALS-PD). The results in Tables 2 and 3 are promising, with state-of-the-art performance on CARRADA-RAC and substantial improvements on KuRALS-PD. This demonstrates the applicability of the proposed approach across different radar sensing systems.
  3. The authors provide useful analyses of the proposed focusing mechanism and its sensitivity to temporal continuity. Table 1 compares different spatial focusing strategies, while Figure 4 evaluates the contribution of Track Loss. Appendix D.1 further investigates temporal continuity across datasets and helps explain the differences in performance.
Critical weaknesses
  1. Although the paper compares TFoC with several existing methods, a controlled comparison with simpler temporal aggregation mechanisms would help establish the contribution of the proposed focusing design. For example, comparisons with local temporal attention or 3D convolution using the same backbone, temporal window, and training settings would make it easier to understand the advantages of TFoC.

  2. The computational evaluation is limited. Tables 2 and 3 report model parameter counts, but do not provide FLOPs, inference latency, or memory consumption. Since TFoC performs feature matching across historical frames and spatial neighborhoods, its computational requirements may increase with the temporal window and search size. Quantifying this overhead would help assess its practical efficiency.

Questions for authors
  1. Table 3 reports a substantial improvement on KuRALS-PD. Could the authors clarify the temporal input configurations used by the competing methods? A comparison with existing temporal aggregation approaches using the same number of frames would help isolate the contribution of TFoC's focusing mechanism.

  2. Could the authors provide FLOPs, inference latency, and memory consumption for TFoC and representative temporal baselines? It would also be useful to understand how the computational cost changes with the number of historical frames and the spatial search window.

  3. The proposed Track Loss is based on semantic labels aggregated across historical frames rather than explicit object trajectories. Could the authors clarify to what extent this supervision captures trajectory consistency, particularly when multiple objects of the same class appear within the same spatial neighborhood?

5 comments
AuthorsOpenReview ↗

Response to NFqr (part I)

NFqr-R1:Controlled comparison with simpler temporal aggregation (Weakness1)

Tab. NFqr-R1-T1 presents additional ablation studies on temporal aggregation based on the TFoC backbone: temporal averaging (including average / sum / max pooling), 3D convolution and matched local attention. The comparison that holds the backbone, the context length, the search support, and the training protocol fixed. The experimental results further verify the effectiveness of TFoC.

We further provide the results of “GT-guided TFoC” , which replaces the learned correspondence with the ground-truth class (historical labels) match. This model represents the performance ceiling of TFoC. The relevant results have validated the effectiveness of the core idea (spatio-temporal alignment mechanism) behind TFoC.

Table NFqr-R1-T1. Ablation studies on temporal aggregation. Bold: best. Italic: second best.

DatasetMethod#Param.RD mIoU (%)RD mDice (%)RA mIoU (%)RA mDice (%)Global Avg.
CARRADA-RACLocal Attention5.06 M61.373.549.661.761.5
CARRADA-RAC3D Conv5.08 M55.567.344.455.355.6
CARRADA-RACAvg Pool5.11 M53.865.347.259.056.3
CARRADA-RACMax Pool5.11 M58.170.245.055.957.3
CARRADA-RACSum Pool5.11 M60.272.348.860.660.5
CARRADA-RACGT-Guided5.09 M70.380.958.470.370.0
CARRADA-RACTFoC (ours)5.1 M64.276.251.063.463.7
KuRALS-PDLocal Attention1.85 M71.882.5––77.2
KuRALS-PD3D Conv1.91 M75.083.7––79.4
KuRALS-PDAvg Pool1.86 M68.578.5––73.5
KuRALS-PDMax Pool1.86 M74.983.8––79.4
KuRALS-PDSum Pool1.86 M73.282.4––77.8
KuRALS-PDGT-Guided1.86 M78.786.9––82.8
KuRALS-PDTFoC (ours)1.8 M77.185.7––81.4
KuRALS-CWLocal Attention1.85 M58.167.0––62.6
KuRALS-CW3D Conv1.83 M64.864.8––64.8
KuRALS-CWAvg Pool1.86 M57.364.1––60.7
KuRALS-CWMax Pool1.86 M58.866.8––62.8
KuRALS-CWSum Pool1.86 M58.165.0––61.6
KuRALS-CWGT-Guided1.85 M60.968.6––64.8
KuRALS-CWTFoC (ours)1.8 M58.466.8––62.6

NFqr-R2:Temporal input of the baselines (Question1)

We fully agree with your comment. The number of input frames for each network in Table 3 is not uniform, which is constrained by practical conditions. Specifically, FCN, U-Net, DeepLabv3+, HRNet, Swin-T, SegFormer, RSSNet, and KuRALS-Net take 1 input frame, i.e., only the target frame. TMVA-Net, PKCIn-Net, and TARSS-Net use 5 input frames. All these results are cited from published papers. Among the aforementioned methods, except TMVA-Net and TARSS-Net, the network designs do not focus on temporal modeling. Therefore, we cannot introduce multi-frame temporal relations into these methods, as that would constitute secondary innovation on top of them. Since TARSS-Net outperforms TMVA-Net under the same input frame setting, it is sufficient to compare TFoC with TARSS-Net with the same number of input frames. We retrained TARSS-Net following the temporal parameter settings of TFoC, i.e. t=12 on KuRALS-PD, t=8 on KuRALS-CW and t= 9 on CARRADA and CARRADA-RAC, the results are shown in Tab.NFqr-R2-T1.

Table NFqr-R2-T1.

DatasetModelSettingParam.RD DiceRD MIoURA DiceRA MIoUGlobal avg.
CARRADATARSS-Nett=96.373.361.345.837.554.5
CARRADATFoCt=95.174.362.252.542.357.8
CARRADA-RACTARSS-Nett=96.374.362.460.448.761.5
CARRADA-RACTFoCt=95.176.264.263.451.063.7
KuRALS-CWTARSS-Nett=81.352.042.4--47.2
KuRALS-CWTFoCt=81.866.858.4--62.6
KuRALS-PDTARSS-Nett=121.350.941.6--46.3
KuRALS-PDTFoCt=121.885.777.1--81.4
AuthorsOpenReview ↗

Response to NFqr (part II)

NFqr-R3:Trajectory consistency supervision (Question3)

Thank you for raising this valuable comment.

  1. The current TCA implements tracking-guided temporal occupancy supervision rather than rigorous trajectory events. If there is only one entity per target class, trajectory consistency among targets of the same class is roughly equivalent to class consistency. When there exist multiple instances for targets of the same class, the current version of TCA only imposes supervision on class consistency, and the supervision over trajectory consistency is weakened. We will supplement this in revised manuscript. Nevertheless, existing RSS datasets (CARRADA, CARRADA-RAC, KuRALS-CW, and KuRALS-PD, all included in the experiments of this manuscript) lack instance-level labels for multiple targets of the same class, which prevents us from realizing effective supervision of instance-level trajectory consistency within TCA.
  2. However, this does not undermine the effectiveness of our proposed TCA (Track Loss) module in boosting RSS performance. The core role of TCA in TFoC is to fully leverage the trajectory consistency (i.e., continuity) of targets across historical frames, which has been validated by the ablation experiments on Track Loss in Section 4.2. Furthermore, we present a detailed analysis of the continuity of all datasets in Appendix D1. Since KuRALS-PD and CARRADA-RAC exhibit superior continuity compared to KuRALS-CW, Track Loss yields a more pronounced performance improvement for RSS on these two datasets. All these findings sufficiently demonstrate that TFoC effectively utilizes trajectory continuity and could improve RSS results.
Reviewer_NFqrOpenReview ↗

Response to Authors and Final Assessment

Thank you for the detailed responses and additional experiments. The controlled comparisons with other temporal aggregation methods address my main concern and strengthen the paper. The clarification regarding Track Loss is also helpful.

Overall, I appreciate the additional experiments and will maintain my current rating.

Reviewer_cR6cOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TFoC for radar semantic segmentation, addressing the difficulty of aggregating sparse target features across frames under motion. The method aligns historical features with the current observation and combines them through learned temporal weighting. An auxiliary Track Loss provides supervision from historical semantic masks.

Critical strengths
  1. Problem formulation. Paper addresses the misalignment of sparse, moving radar targets, which limits the usefulness of temporal aggregation at fixed spatial locations.

  2. Practical design. Local historical search and learned temporal weighting provide a flexible mechanism for incorporating past observations without requiring exploit object tracks or instance-level trajectory annotations.

  3. Segmentation results and ablations. Proposed method achieves competitive performance across the evaluated radar settings, also the paper examines search size, aggregation temperatures, sequence length, and Track Loss.

Critical weaknesses
  1. Trajectory consistency is not demonstrated. Track Loss supervises historical class presence at fixed coordinates (Eq 7), without preserving object identity or temporal ordering. This does not establish the same object trajectory consistency Eq 18 and the reported segmentation metrics provide no independent verification.

  2. Challenging scenarios are insufficiently evaluated. Paper does not separately assess same-class crossings, occlusion, reappearance, or object entry and exit. These cases are particularly relevant because semantic agreement alone cannot distinguish different objects of the same category, as acknowledged in Appendix A.2.

  3. Contribution of temporal methods is unclear. Tables 2-3 compare complete segmentation systems, but controlled comparisons with temporal averaging, matched local attention, or motion-guided alignment using the same backbone are missing. Those comparisons would better establish the value of this work.

Questions for authors
  1. What evidence supports same-object trajectory consistency? (beyond semantic class agreement)
  2. How does the class-presence supervision in Eq. 7 enforce the trajectory consistency definition in Eq. 18?
  3. How does TFoC perform under same class crossings, occlusions, and object entry/exit?
  4. Can you provide controlled temporal aggregation baselines and more challenging qualitative examples?
2 comments
AuthorsOpenReview ↗

Response to cR6c (part I)

cR6c-R1: Trajectory consistency ( Weakness1,Question1 and Question2)

We understand your concern and appreciate you pointing this out. If there is only one entity per target class, trajectory consistency among targets of the same class is roughly equivalent to class consistency. When there exist multiple instances for targets of the same class, the current version of TCA only imposes supervision on class consistency, and the supervision over trajectory consistency is weakened. We will supplement this in revised manuscript. Nevertheless, existing RSS datasets (CARRADA, CARRADA-RAC, KuRALS-CW, and KuRALS-PD, all included in the experiments of this manuscript) lack instance-level labels for multiple targets of the same class, which prevents us from realizing effective supervision of instance-level trajectory consistency within TCA. This limitation is imposed by the drawbacks of available datasets. However, this does not undermine the effectiveness of our proposed TCA (Track Loss) module in boosting RSS performance. The core role of TCA in TFoC is to fully leverage the trajectory consistency (i.e., continuity) of targets across historical frames, which has been validated by the ablation experiments on Track Loss in Section 4.2. Furthermore, we present a detailed analysis of the continuity of all datasets in Appendix D1. Since KuRALS-PD and CARRADA-RAC exhibit superior continuity compared to KuRALS-CW, Track Loss yields a more pronounced performance improvement for RSS on these two datasets. All these findings sufficiently demonstrate that TFoC effectively utilizes trajectory continuity and could improve RSS results.

In summary, the core essence of TFoC lies in exploiting trajectory consistency to improve RSS performance, and its effectiveness has been thoroughly substantiated in the manuscript. Constrained by the annotation limitations of existing datasets, TFoC is currently unable to separately supervise the trajectory consistency of individual instances across different target classes. Nevertheless, this is not a fatal flaw of the proposed method; instead, it represents a direction for further upgrade and improvement of TFoC in future work.

cR6c-R2: Challenging scenarios ( Weakness2 and Question3)

Among existing RSS datasets, CARRADA (CARRADA-RAC) is designed for autonomous driving scenarios, covering targets including vehicles, pedestrians, and bicycles across diverse weather conditions and environments. The KuRALS dataset encompasses a broader range of radar surveillance scenarios, spanning air, ground, and maritime surveillance, with target categories such as UAVs, vehicles, and boats. Both datasets are constructed to thoroughly validate the core challenge in the RSS field: multi-scenario detection of low-slow-small (LSS) targets.

The scenarios you raised, such as crossings and occlusion, are primarily a central focus in the computer vision domain. This challenge also needs to be addressed in the radar domain, albeit not as a primary priority. Addressing such scenarios in the radar field requires dedicated construction of group target scenarios and corresponding data collection. As a result, data samples involving crossings and occlusion are scarce in both CARRADA (CARRADA-RAC) and KuRALS, rendering them inadequate for rigorous validation.

In comparison, all samples in CARRADA(CARRADA-RAC) contain target entry/exit events, while reappearance scenarios occur frequently in KuRALS-CW. Accordingly, our current experiments are sufficient to fully verify the effectiveness of TFoC in reappearance and entry/exit scenarios. Our team is actively developing an RSS dataset for group targets with instance-level annotations, and we expect that the effectiveness of TFoC in crossings and occlusion scenarios can be further validated in future work. Taken together, TFoC makes a positive contribution in the radar surveillance domain.

AuthorsOpenReview ↗

Response to cR6c (part II)

cR6c-R3: Temporal aggregation baselines (Weakness3 and Question4)

Tab. cR6c-R1-T1 presents additional ablation studies on temporal aggregation based on the TFoC backbone: temporal averaging (including average / sum / max pooling), 3D convolution and matched local attention. The comparison that holds the backbone, the context length, the search size, and the training protocol fixed. The experimental results further verify the effectiveness of TFoC.

A motion-guided alignment control is not in this table, as exited datasets only provide semantic masks. They do not provide tracks, optical flow, or a motion field that could guide a warp. However, we further provide the results of “GT-guided TFoC” , which replaces the learned correspondence with the ground-truth class match. This corresponds to the upper bound of motion compensation and represents the performance ceiling of TFoC. The relevant results validate the effectiveness of the core idea (spatio-temporal alignment mechanism) behind TFoC.

Table cR6c-R1-T1. Ablation studies on temporal aggregation. Bold: best. Italic: second best.

DatasetMethod#Param.RD mIoU (%)RD mDice (%)RA mIoU (%)RA mDice (%)Global Avg.
CARRADA-RACLocal Attention5.06 M61.373.549.661.761.5
CARRADA-RAC3D Conv5.08 M55.567.344.455.355.6
CARRADA-RACAvg Pool5.11 M53.865.347.259.056.3
CARRADA-RACMax Pool5.11 M58.170.245.055.957.3
CARRADA-RACSum Pool5.11 M60.272.348.860.660.5
CARRADA-RACGT-Guided5.09 M70.380.958.470.370.0
CARRADA-RACTFoC (ours)5.1 M64.276.251.063.463.7
KuRALS-PDLocal Attention1.85 M71.882.5––77.2
KuRALS-PD3D Conv1.91 M75.083.7––79.4
KuRALS-PDAvg Pool1.86 M68.578.5––73.5
KuRALS-PDMax Pool1.86 M74.983.8––79.4
KuRALS-PDSum Pool1.86 M73.282.4––77.8
KuRALS-PDGT-Guided1.86 M78.786.9––82.8
KuRALS-PDTFoC (ours)1.8 M77.185.7––81.4
KuRALS-CWLocal Attention1.85 M58.167.0––62.6
KuRALS-CW3D Conv1.83 M64.864.8––64.8
KuRALS-CWAvg Pool1.86 M57.364.1––60.7
KuRALS-CWMax Pool1.86 M58.866.8––62.8
KuRALS-CWSum Pool1.86 M58.165.0––61.6
KuRALS-CWGT-Guided1.85 M60.968.6––64.8
KuRALS-CWTFoC (ours)1.8 M58.466.8––62.6

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to BjCX (part I)

BjCX-R1: The temporal-temperature explanation (Weakness1 and Question1)

We apologize for the typographical error: our statement about σtemp\sigma_{\mathrm{temp}} in Section 3.2.2 is reversed. The discussion of Figure 3(b) already says that σtemp≈0.95\sigma_{\mathrm{temp}}\approx 0.95 is the longer path memory on RD, where a continuous trail benefits from keeping every frame, and that the RA peak is at 0.10.1, where angular smear makes a single frame the better choice. The experimental results match Equation (4).

The typo in Section 3.2.2 will be fixed promptly. Thank you again for your careful review.

BjCX-R2: Track Loss, background, and the trajectory event (Weakness2 and Question2)

  1. It is true that current TCA implements tracking-guided temporal occupancy supervision rather than rigorous trajectory events. If there is only one entity per target class, trajectory consistency among targets of the same class is roughly equivalent to class consistency. When there exist multiple instances for targets of the same class, the current version of TCA only imposes supervision on class consistency, and the supervision over trajectory consistency is weakened. We will supplement this in revised manuscript. Nevertheless, existing RSS datasets (CARRADA, CARRADA-RAC, KuRALS-CW, and KuRALS-PD, all included in the experiments of this manuscript) lack instance-level labels for multiple targets of the same class, which prevents us from realizing effective supervision of instance-level trajectory consistency within TCA. This limitation is imposed by the drawbacks of available datasets. However, this does not undermine the effectiveness of our proposed TCA (Track Loss) module in boosting RSS performance. The core role of TCA in TFoC is to fully leverage the trajectory consistency (i.e., continuity) of targets across historical frames, which has been validated by the ablation experiments on Track Loss in Section 4.2. Furthermore, we present a detailed analysis of the continuity of all datasets in Appendix D1. Since KuRALS-PD and CARRADA-RAC exhibit superior continuity compared to KuRALS-CW, Track Loss yields a more pronounced performance improvement for RSS on these two datasets. All these findings sufficiently demonstrate that TFoC effectively utilizes trajectory consistency (continuity) and could improve RSS results.

  2. We did not warp historical labels to SAF-selected locations. The shape of T∗\mathbf{T}^* in Equation (7) is C×H×WC\times H \times W and it retains the class channel, where the background is one of these classes. Therefore, cases where a moving object generates both foreground and background labels at the same coordinate can be distinguished along the class channel dimension.

  3. The background is also treated as one class in semantic segmentation tasks. Accordingly, Eq. (6) distributes probabilities over the background and all target classes, i.e., it predicts the probability of each location belonging to a specific class. Eq. (7) produces a binary trajectory label, which provides trajectory-filter-like supervision for the predicted class outputs. As stated above, this loss is not equivalent to strict trajectory consistency (especially when multi-entity belong to same class ), yet it still delivers supervision for trajectory consistency. Hence, Track Loss is correlated with trajectory consistency.

AuthorsOpenReview ↗

Response to BjCX (part II)

BjCX-R3: Additional ablation studies on close alternatives based on the TFoC backbone (Weakness3 and Question4)

Tab. BjCX-R3-T1 presents additional ablation studies on temporal aggregation based on the TFoC backbone: temporal averaging (including average / sum / max pooling), 3D convolution and matched local attention. The comparison that holds the backbone, the context length, the search support, and the training protocol fixed. The experimental results further verify the effectiveness of TFoC.

We further provide the results of “GT-guided TFoC” , which replaces the learned correspondence with the ground-truth class (historical labels) match. This model represents the performance ceiling of TFoC. The relevant results have validated the effectiveness of the core idea (spatio-temporal alignment mechanism) behind TFoC.

Table BjCX-R3-T1. Ablation studies on close alternatives. Bold: best. Italic: second best.

DatasetMethod#Param.RD mIoU (%)RD mDice (%)RA mIoU (%)RA mDice (%)Global Avg.
CARRADA-RACLocal Attention5.06 M61.373.549.661.761.5
CARRADA-RAC3D Conv5.08 M55.567.344.455.355.6
CARRADA-RACAvg Pool5.11 M53.865.347.259.056.3
CARRADA-RACMax Pool5.11 M58.170.245.055.957.3
CARRADA-RACSum Pool5.11 M60.272.348.860.660.5
CARRADA-RACGT-Guided5.09 M70.380.958.470.370.0
CARRADA-RACTFoC (ours)5.1 M64.276.251.063.463.7
KuRALS-PDLocal Attention1.85 M71.882.5––77.2
KuRALS-PD3D Conv1.91 M75.083.7––79.4
KuRALS-PDAvg Pool1.86 M68.578.5––73.5
KuRALS-PDMax Pool1.86 M74.983.8––79.4
KuRALS-PDSum Pool1.86 M73.282.4––77.8
KuRALS-PDGT-Guided1.86 M78.786.9––82.8
KuRALS-PDTFoC (ours)1.8 M77.185.7––81.4
KuRALS-CWLocal Attention1.85 M58.167.0––62.6
KuRALS-CW3D Conv1.83 M64.864.8––64.8
KuRALS-CWAvg Pool1.86 M57.364.1––60.7
KuRALS-CWMax Pool1.86 M58.866.8––62.8
KuRALS-CWSum Pool1.86 M58.165.0––61.6
KuRALS-CWGT-Guided1.85 M60.968.6––64.8
KuRALS-CWTFoC (ours)1.8 M58.466.8––62.6
AuthorsOpenReview ↗

Response to BjCX (part III)

BjCX-R4: Protocol (Weakness4 and Question3)

  1. We apologize for any ambiguity in our manuscript that has led to this misunderstanding. Appendix B.2 is the protocol Section 4 points to, and it already states the window, the temperature, and the context length of each TFoC experiment. The sentence “unless otherwise specified, we use a unified set of hyperparameters for all models” refers to models related to TFoC. The same appendix states that "every other model’s results is the results in its own paper". Here, “other models” include FCN, U-Net, DeepLabv3+, HRNet, Swin-T, SegFormer, RSSNet, KuRALS-Net, TMVA-Net, PKCIn-Net, and TARSS-Net. We will emphasize this point in revised manuscript to avoid misunderstanding. Thank you for pointing this out.
  2. The test-subset sentence does not say that the window, the temperature, or the context length were chosen on the test set. Appendix B.2 says that precision, recall, IoU, and Dice are computed during training and validation, and then that the reported results were obtained from the test subset. The checkpoint is selected on the validation split. The table entry is the test metric of that checkpoint. Publishing the test result is the evaluation step. Notably, kk, σtemp\sigma_{\mathrm{temp}}, or tt were not searched on the test split. The neighboring values of those three factors are also already in the paper: search size in Figure 2, temperature in Figure 3, and context length in Appendix D. The operating point of each table is the one printed in Appendix B.2.
  3. Furthermore, the model is trained with the AdamW optimizer with initial learning rate 0.000096, training epoch is 300, λTrk\lambda_{Trk} is set to 0.65 and 10 for the CARRADA (CARRADA-RAC) and KuRALs datasets, respectively and batch size is set to 6 and 10 for CARRADA (CARRADA-RAC) and KuRALs datasets. We have uploaded the original code as an attachment. The code will also be released upon paper acceptance, so these configurations can also be found in the json file.
  4. Among the compared baseline models, except TMVA-Net and TARSS-Net, the network designs do not focus on temporal modeling. Therefore, we cannot introduce multi-frame temporal relations into these methods, as that would constitute secondary innovation on top of them. Since TARSS-Net outperforms TMVA-Net under the same input frame setting, it is sufficient to compare TFoC with TARSS-Net with the same number of input frames. We retrained TARSS-Net following the temporal parameter settings of TFoC, i.e. t=12 on KuRALS-PD, t=8 on KuRALS-CW and t= 9 on CARRADA and CARRADA-RAC, the results are shown in Tab.BjCX-R4-T1.

Tab.BjCX-R4-T1

DatasetModelSettingParam.RD DiceRD MIoURA DiceRA MIoUGlobal avg.
CARRADATARSS-Nett=96.373.361.345.837.554.5
CARRADATFoCt=95.174.362.252.542.357.8
CARRADA-RACTARSS-Nett=96.374.362.460.448.761.5
CARRADA-RACTFoCt=95.176.264.263.451.063.7
KuRALS-CWTARSS-Nett=81.352.042.4--47.2
KuRALS-CWTFoCt=81.866.858.4--62.6
KuRALS-PDTARSS-Nett=121.350.941.6--46.3
KuRALS-PDTFoCt=121.885.777.1--81.4
AuthorsOpenReview ↗

Response to BjCX (part IV)

BjCX-R5: Train-Infer Profiling (Weakness4 and Question4)

We test the GMACs, peak memory, and latency of the TFoC model under different search-size (k) and temporal window lengths (t) on CARRADA-RAC, KuRALs-PD, and KuRALs-CW, respectively.

KuRALS-PD

SettingGPU×Batch size per GPUParams (M)GMACsGMACs per SampleInference latency (ms)Inference peak memory(MB)Train FWD (ms)Train BWD (ms)Train peak memory (MB)
t=7, k=31GPU×b101.853378.837.946.78±0.37304156.43196.6013062
t=8, k=31GPU×b101.853431.943.256.02±0.04398969.63231.9315158
t=9, k=31GPU×b101.853485.148.565.84±0.04493983.10267.7917262
t=10, k=31GPU×b101.853538.253.875.64±0.04589096.41303.6919361
t=11, k=31GPU×b101.853591.359.185.11±0.046947109.79338.9921464
t=12, k=31GPU×b101.853644.464.495.20±0.098049123.29374.5223562
#GPUs x BatchSizeParams (M)GMACsGMACs per SampleInference latency (ms)Inference Peak Mem. (MB)Train FWD (ms)Train BWD (ms)Train Peak Mem.(MB)
t=12, k=11GPUxb101.852624.862.567.52±0.04547875.98233.8117394
t=12, k=31GPUxb101.853644.464.495.20±0.098049123.29374.5223562
t=12, k=51GPUxb101.855771.977.2191.14±0.0519566257.76763.7936483
t=12, k=71GPUxb101.8581183.2118.3444.68±0.2736840566.133843.9362523
t=12, k=92GPUs×b51.8621071.3214.3497.89±1.6229975746.924184.0148673

KuRALS-CW

#GPUs x BatchSizeParams (M)GMACsGMACs per SampleInference latency (ms)Inference Peak Mem. (MB)Train FWD (ms)Train BWD (ms)Train Peak Mem.(MB)
t=7, k=52GPUs×b51.8551029.2205.8202.37±0.5714222326.83903.1440476
t=8, k=52GPUs×b51.8551205.9241.2259.17±0.0620740443.921482.3949647
t=9, k=52GPUs×b51.8551382.6276.5320.53±0.0527267559.031846.1958831
t=10, k=52GPUs×b51.8551559.3311.9383.26±0.0833783675.562209.0967999
t=11, k=52GPUs×b51.8551736.0347.2445.29±0.0640310791.882575.0677179
t=12, k=52GPUs×b51.855765.1382.5244.74±0.3618803404.331399.9134907
#GPUs x BatchSizeParams (M)GMACsGMACs per SampleInference latency (ms)Inference Peak Mem. (MB)Train FWD (ms)Train BWD (ms)Train Peak Mem. (MB)
t=8, k=12GPUs × b51.8521050.1210.0116.31±0.176723153.34532.1330680
t=8, k=32GPUs × b51.8531070.8214.2150.60±0.059449238.59735.1236953
t=8, k=52GPUs × b51.8551205.9241.2259.17±0.0620740443.921482.3949647
t=8, k=72GPUs × b51.8581642.5328.5543.50±0.8238583870.234433.3768303
t=8, k=95GPUs × b21.8621064.6532.3495.48±0.0725295710.593964.0840881

Carrada RAC

#GPUs x BatchSizeParams (M)GMACsGMACs per SampleInference latency (ms)Inference Peak Mem. (MB)Train FWD (ms)Train BWD (ms)Train Peak Mem. (MB)
t=7, k=54GPUs x b35.088205.368.443.52±0.24264899.64177.689897
t=8, k=54GPUs x b35.088246.382.161.01±0.073824125.81238.7112953
t=9, k=54GPUs x b35.088287.495.880.12±0.144997159.66294.8615997
t=10, k=54GPUs x b35.088328.4109.598.68±0.116174188.50355.3119052
t=11, k=54GPUs x b35.088369.5123.2116.92±0.087346221.84410.9622093
t=12, k=54GPUs x b35.088410.5136.8135.00±0.158521258.66470.0325148
#GPUs x BatchSizeParams (M)GMACsGMACs per SampleInference latency (ms)Inference Peak Mem. (MB)Train FWD (ms)Train BWD (ms)Train Peak Mem. (MB)
t=9, k=14GPUs x b35.078239.179.723.77±0.04138962.3599.506812
t=9, k=34GPUs x b35.081245.681.936.41±0.07219186.41161.199874
t=9, k=54GPUs x b35.088287.495.880.12±0.144997159.66294.8615997
t=9, k=74GPUs x b35.097422.6140.9179.82±0.079317308.25928.8725179
t=9, k=94GPUs x b35.109738.2246.1375.20±0.0915078562.442436.4237418

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.