- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: the paper performs a joint search over neural architectures and sensor settings, and shows that their search method can produce effective architecture-sensor setting combinations for movement recognition tasks.
Critical strengths
-
The method accounts for the geometry of the discrete ordinal space over which the search is conducted.
-
The specific problem of co-adapting sensor settings and architecture hasn’t been studied in the literature before to my knowledge, and the architecture-sensor location combination found does seem to outperform only tuning the architecture.
-
I didn’t spot any errors in the mathematical claims
Critical weaknesses
-
The problem is not naturally motivated. It is true that nobody has tried to co-search over architecture and sensors, but this is perhaps because the problem doesn’t arise naturally, rather than because the problem is important and unsolved due to its difficulty.
-
The Rao-blackwellized score estimation is, as far as I can tell, just finite differences. Finite differences is a very reasonable strategy for estimating derivatives of discrete points on the line, and the authors should be up front about the method rather than trying to obfuscate the approach behind unnecessary statistical machinery.
-
The ‘co-design’ framing feels a bit inaccurate, since the method uses a two-phase process to first select a good set of sensors and then runs NAS with respect to those fixed sensors.
Questions for authors
Q1. How specific is the method to the data modalities considered in this paper? Could it be used for other types of feature selection?
Q2. Is W2 correct, or is there some advantage of the Rao-blackwellization that I missed which yields different properties from finite differences?
6 comments
Response to Reviewer 1YMh (2/3)
Response to Weakness 2 and Q2. Thank you for this careful reading. We agree with the reviewer: the selector update is a finite difference. For each setting axis, the update is the difference between the losses measured at the two neighbouring levels (Eq. 6). Rao–Blackwellization gives this finite difference no property that a finite difference does not have; it is only the route by which we derive it. The selector rounds each axis at random to one of two neighbouring levels. The usual single-sample (score-function) gradient of this random rounding is noisy, and averaging that gradient over the rounding in closed form leaves exactly the difference of the two neighbouring losses. The derivation tells us two things about the finite difference: it is an unbiased gradient of the expected loss, and its variance is no larger than that of the single-sample gradient (Proposition 1). The row "w/o finite difference" of Table 4 replaces the finite difference with the single-sample gradient:
| Variant (Table 4) | Radar HV | IMU HV |
|---|---|---|
| Full (SPANAS), finite difference | 0.325 | 0.263 |
| w/o finite difference (single-sample gradient) | 0.325 | 0.190 |
We agree that the submission should have said this up front. The paper now calls the update a finite difference throughout: in the abstract, §1, §3.2, the ablation of §4.4 (the row is now named "w/o finite difference") and the conclusion. Rao–Blackwellization is named only in the related work (§2) and in the derivation (Appendix D.1). What is new is the use of this finite difference: it gives the loss of the architecture a gradient with respect to sensor settings that are separate measurements, which, to our knowledge, no earlier NAS method obtains.
Where in the revised paper: Abstract; §1; §2; §3.2; §4.4 (Table 4); §5.
Response to Weakness 3. Thank you for this comment. We understand that the word 'co-design' can be read as optimising the two at exactly the same time and in one procedure, whereas the method has two phases. To avoid this reading, we removed 'co-design' and 'joint' as descriptions of SPANAS from the revised paper (abstract, introduction, contribution list, section title, Algorithm 1 and the tables).
The contribution of the paper is that the sensor pre-acquisition parameters and the architecture parameters are optimised in a coupled way, each with the current state of the other. In Phase A the two are updated in the same loop: for the current sensor setting a network proposes an architecture, the data acquired at that setting pass through this architecture, and the measured loss updates both the selector of the setting and the network that proposes the architecture (§3.2, Eqs. 4 to 6). Thus the loss of the architecture drives the selection of the sensor pre-acquisition parameters, and the architecture follows the selected setting. Phase B then refines the architecture at the selected setting. To our knowledge, this coupled optimisation is the first of its kind: a gradient that runs end to end from the architecture search to the setting of the sensor pre-acquisition parameters has not been proposed so far for sensors without a differentiable acquisition model, and existing methods cannot obtain it, because each setting is a separate measurement. SPANAS builds this gradient from measured losses.
This makes three things possible that existing methods do not offer. (i) The sensor setting is selected with a gradient although the sensor is not differentiable; differentiable NAS can only relax choices inside one recorded input. (ii) One shared network is trained once; black-box search trains every candidate pair of setting and architecture separately. (iii) The architecture is generated for the setting that is selected, so the two fit each other.
The ablation supports this. If the network that proposes the architecture is not told the setting, Phase A selects a different and less accurate sensor placement on IMU (Table 4, "w/o conditioning": IMU HV 0.263 → 0.162). On radar, with 27 settings, both variants select the same setting, so the HV is the same.
Where in the revised paper: Abstract; §1 (text and contribution list); §3.2 (new sentence); §3.4 (title, Algorithm 1); §4.3 (Table 2); §5; Fig. 1. The evidence is in Table 4 (§4.4).
Response to Reviewer 1YMh (3/3)
Response to Q1. Thank you for this question; it led us to add a third modality, speech: the 35 spoken words of the public Speech Commands v2 benchmark (Warden, 2018). The same recordings are re-acquired under 81 settings. Sampling rate, ADC bits and recording window are ordered settings, and the ADC encoding curve (linear, μ-law, A-law) is an unordered setting. The same selector handles both kinds: on an ordered setting it compares the two neighbouring levels, and on the unordered setting it compares all options. Results (Appendix I):
(1) SPANAS selects 16 kHz, 1 s, μ-law in all three seeds. Its selections rank 2nd, 2nd and 3rd of the 81 settings, within 0.004 accuracy of the best setting (Fig. 11). The rank comes from a separate reference run: 12 reference architectures were trained at each of the 81 settings, and the settings are ordered by the best test accuracy reached there.
(2) The HV of SPANAS is 0.402 ± 0.003, at the same level as the four black-box baselines (0.397 to 0.403; Table 17). The whole SPANAS pipeline takes less time on the same machine (one NVIDIA RTX 5090 GPU, every run alone): 20 minutes on average, against 31 to 39 minutes on average for a black-box baseline, counting its search and the verification of its front (Fig. 4c). The reason is that a black-box baseline trains 100 candidate models one after another, whereas SPANAS trains one shared network and then retrains only the few cells of its shortlist.
| Method | Verified HV, seed 0 | Verified HV, seed 1 | Verified HV, seed 2 | Mean ± std | Best acc. | Best acc. ≤ 401 KB | Front points (seeds 0/1/2) |
|---|---|---|---|---|---|---|---|
| Random search | .395 | .405 | .409 | .403 ± .007 | .916 | .789 | 6/4/5 |
| NSGA-II | .385 | .410 | .401 | .399 ± .012 | .920 | .793 | 5/5/5 |
| qNEHVI | .392 | .401 | .398 | .397 ± .004 | .904 | .862 | 4/5/3 |
| qNParEGO | .401 | .400 | .409 | .403 ± .005 | .915 | .849 | 5/3/4 |
| SPANAS (six weights) | .403 | .404 | .399 | .402 ± .003 | .916 | .863 | 2/4/3 |
Table 17 of the revised paper: speech, verified HV per seed and mean ± std, best accuracy, best accuracy among cells of at most 401 KB, and the number of points on each verified front.
Where in the revised paper: Abstract; §1; §4.1; §4.2 (Table 1, Fig. 4c); §5; Appendix I.
Response to Reviewer 1YMh (1/3)
Response to Weakness 1. Thank you for this comment. We agree that the submission did not explain the motivation sufficiently. To strengthen it, we revised the paper: the introduction now states which existing studies the motivation rests on, which measurements support it, and what the method contributes (the changed sentences carry the tag [1YMh W1]), and Appendix A has a new heat map of the 27 radar settings (Fig. 5).
Every sensing field already studies this problem, but one sensor at a time. Sensor pre-acquisition parameters must be set before any data are recorded, and they change the recognition performance. For this reason, published studies in each sensing field tune them, in applications from autonomous driving (LiDAR placement) to wearable health monitoring (IMU, PPG, EEG). Table 7 of the revised paper lists 28 such parameter families; some examples are below.
Examples from Table 7 of the revised paper (Appendix B).
These studies show that the problem is important in each field. However, each of them handles one sensor with its own procedure, and none of them searches the architecture. The suitable setting, in turn, depends on the architecture (Enokibori et al., 2025, for IMU placement; our radar measurements are in (ii) below), so the setting should not be tuned without regard to the architecture.
In short, our core contribution is a search in which the sensor pre-acquisition parameters are optimised under the influence of the model architecture: no gradient passes through the sensor, yet the loss of the architecture drives the selection of these parameters. To our knowledge, SPANAS is the first gradient-based search that optimises sensor pre-acquisition parameters in this way, and the first method that does so across sensing domains with one procedure: the same method selects radar chirp parameters, IMU placements and speech acquisition parameters (Table 1). Three measurements show that ignoring this dependence is costly.
(i) The accuracy that can be reached changes a lot with the sensor setting: from 0.43 to 0.86 across the 27 radar settings (Fig. 5).
(ii) Which architecture is suitable differs from one sensor setting to another: different architectures reach their best accuracy at different settings. Table 6 lists ten architectures whose best setting is not the overall best setting, 160/255/30. Fixing the setting to 160/255/30 in advance costs these architectures up to 0.26 accuracy. If the two choices were independent, this loss would be zero; because they are not, the setting cannot be chosen without regard to the architecture.
(iii) SPANAS, which selects the setting with the loss of the architecture, reaches a comparable accuracy with 5.7× less storage than the strongest IMU model from the literature at its recommended placement: 0.857 ± 0.007 at 232 KB against 0.860 ± 0.036 at 1,325 KB (Table 2).
Searching the setting as well as the architecture does not mean searching the product of the two spaces. The selector moves along each ordered setting axis by comparing neighbouring settings, and the generator supplies an architecture suited to each setting, so the cost grows with the number of axes.
Where in the revised paper: §1 (new sentences, tagged [1YMh W1]: the existing studies of each sensing field, our measurements, and the missing method; contribution list); Appendix A (Fig. 5, new; Table 6); Appendix B (Table 7, now referenced from §1); Table 2 (§4.3).