- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a quantization method that reweighs the Hessian of GPTQ, giving each token a heuristic weight and combine it with a clipping search per (expert, modality) to improve quantization accuracy.
Critical strengths
- the evaluation matrix is thorough and broad
- SOME_JQ achieves good empirical performance compared to established baselines
Critical weaknesses
- Having quickly skimmed VEQ I don't see substantial difference in the formulation - for example the per expert weight changes from S_i to P_i^2, and the XCX formulation appears in 3.3 of VEQ. It seems to be building upon that work and the phrasing should be tuned down.
- The choice of P_i^2 doesn't seem to affect optimization result? What is its role there in the equation 8? It doesn't seem to appear in many terms but rather looks like a constant there because each expert is optimized separately?
- While maybe standard in this literature, some of the assumptions are not spelled out: e.g., (1) treats each layer independently and L is the final accuracy, so the rollout is much more than how it is decomposed.
Questions for authors
Please see weakness, additionally:
- SOME_JQ's eval numbers seem to be always a ~multiple of 0.11 - is there a reason for this? The baseline rows seem to be not in this pattern.
- The baseline for VEQ is reused from VEQ - did we ensure VEQ and SOME-JQ share the exact same setup? Giving a PTQ + MEJH with u=0 would be helpful.
Response to Reviewer NuUw
Thanks for the valuable feedback and for pointing out the similarity to VEQ, the role of pi2, and the evaluation-pattern concern. We address each point with new ablations and a unified re-evaluation. If the reviewer finds our response adequate, we would really appreciate it if the reviewer considers raising the score. Please see our responses below one by one.
Q1: The formulation appears very close to VEQ; the phrasing should be tuned down.
A1: We agree and will tune down the "second-order" claim. VEQ already provides routing-affinity and modality-sensitivity weighting; our algorithmic additions are the cross-modal fusion factor (1+μϕj) and per-expert, per-modality clipping. SOEA contributes the error-decomposition framework and the pi2 priority, not a new optimizer. Table 10 quantifies the incremental contributions.
Q2: The choice of pi2 does not seem to affect the optimization result because it looks like a constant when each expert is optimized separately.
A2: Under fully independent optimization, this is correct. In SOME-JQ, pi2 also scales the clipping-candidate quantile range and reweights the cross-expert update direction during joint calibration. Table 11 shows the full mechanism is 0.69 points above constant scaling alone.
We will make this implementation detail explicit in Algorithm 1.
Q3: Some assumptions are not spelled out, e.g., Eq. (1) treats each layer independently while the task loss is global.
A3: We agree. SOEA is a layer-wise, second-order motivation rather than a rigorous global upper bound. We will state in the limitations that the decomposition motivates layer-wise error weighting, and that the empirical effectiveness under global rollout is supported by the unified re-evaluation rather than by a global theoretical guarantee.
Q4: Evaluation numbers are always about multiples of 0.11, and the baselines do not follow this pattern.
A4: The SOME-JQ rows were produced on 900-question or divisor-of-900 sets, while some baseline rows came from VEQ's own evaluation set, causing the pattern. Under the unified harness (Supp. H.4) all rows share one evaluation set, and the systematic pattern disappears.
Q5: Did we ensure VEQ and SOME-JQ share the exact same setup? A PTQ + MEJH with μ=0 would be helpful.
A5: We re-ran both under one harness. The requested μ=0 comparison is in Table 10: VEQ + MEJH with μ=0 gains only +0.03, confirming that the main MEJH gain comes from the fusion term, not from reparameterizing VEQ's weights.
Finally, we thank the reviewer again for the careful comparison with VEQ and for the evaluation-pattern observation. We believe the incremental ablation and the unified re-evaluation make the contribution and the empirical basis clearer, and we hope these updates improve the overall evaluation of our work.