- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds classifiers with a weighted-rule interpretation and extends them to deeper networks with separate chains for each class. It evaluates accuracy, program size, and inference time on 18 datasets.
Critical strengths
I like that the rules belong to the model making the prediction, rather than being fitted afterward to imitate it. The experiments cover a good range of datasets, and the size and runtime results help show what using these models would cost.
Critical weaknesses
-
The checkpoint selection uses validation data, and Table 4 gives both variants separately. My concern is with Table 1 taking the better shallow/deep test result for each dataset. What would the result be if that choice were made on validation data? The pruning results have a similar limitation: they are clearly labeled as oracle results, but do not show how much compression a validation-selected pruning level would give.
-
Tables 8–9 do compare shared and separate chains, but I could not find a comparison with matched parameter counts. Separate chains also give the model more capacity. Without controlling for that, I am not sure how much of the gain comes from reducing gradient interference.
Questions for authors
- What results do you get when both the shallow/deep choice and pruning level are selected on validation data?
- Do class-specific chains still help against a shared model with a comparable parameter count?
3 comments
W2 / Q2. Do class-specific chains still help against a parameter-matched shared model?
Added (Table 9: a shared chain with clause counts scaled by to match the parameter count). With matched parameters, they help on chess and letRecog, and tie on car. Claims in Sec. 3.4 and 5 are revised accordingly.
| Dataset | C | Shallow | Shared | Shared (param.-matched) | Class-cond. |
|---|---|---|---|---|---|
| car | 4 | 94.81 ± 5.50 | 98.55 ± 1.75 | 99.11 ± 1.25 | 98.71 ± 1.17 |
| chess | 18 | 83.47 ± 0.94 | 67.09 ± 0.72 | 63.04 ± 1.02 | 86.54 ± 0.87 |
| letRecog | 26 | 94.86 ± 0.32 | 91.70 ± 0.84 | 91.07 ± 0.43 | 92.87 ± 0.40 |
| #params (car/chess/letRecog) | --- | 35K / 160K / 579K | 139K / 2.75M / 14.6M | 138K / 2.75M / 14.6M |
Thanks for new experiments which address some of my concerns. I raised my score accordingly.
W1 / Q1. Results when the Shallow/Deep choice and the pruning level are selected on validation?
Both are now validation-selected.
Shallow/Deep: Table 1 changes on three datasets (magic 84.80→84.41, wine 96.68→96.14, dota2 59.43→59.36); average rank 4.50→4.56, still 4th of 12.
Pruning (Table 5: Before / After (val) / After (oracle)): Shallow shrinks by 19--68% on 12 of 18 datasets (banknote 254.6→82.4; adult 198.6→160.0 vs. 110.4 oracle), Deep by 9--81%. Test F1 drops by ≤0.4pp on 17/18 datasets (Shallow) and 15/18 (Deep); the exceptions have tiny validation splits (wine, 15 samples: −5.6 / −2.6; tic-tac-toe Deep −1.5; car Deep −0.9).