- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies downstream structured query composition for intent-to-knowledge-graph retrieval. Given grounded action, object, and target slots with confidence scores supplied by an upstream system, the model retrieves a complete triple from a fixed candidate catalog. The proposed CCSC method combines MLP-based slot content scores with log-confidence biases, applies softmax to obtain slot weights, and uses role-aligned concatenation and a shared projection to construct retrieval representations. Training uses a contrastive retrieval objective with entropy regularization.
Experiments cover two BINS-derived panels and three known-fact catalogs constructed from existing knowledge graphs. Grounding errors are simulated by replacing or removing slots. CCSC outperforms the evaluated confidence-free composition baselines, although its additional gains over GateOnly, which receives the same confidence signal, are small and condition-dependent. The paper also examines confidence use during training and inference, temperature sensitivity, and confidence transformations.
Critical strengths
-
Clear task formulation and method specification. The paper explicitly separates upstream grounding from downstream retrieval and defines the correspondence between semantic roles and triple positions. The method is relatively simple to implement.
-
Informative controlled comparisons. GateOnly, NoConf, and the training/inference confidence grid help distinguish the benefit of supplied confidence from the additional contribution of learned content weighting.
-
Candid reporting of limitations and unfavorable results. The authors disclose the gold-dependent construction of confidence and the use of candidate facts seen during training. They also report conditions under which GateOnly or separate projections perform better.
Critical weaknesses
-
Limited methodological novelty and insufficient added value beyond a combination of simple components. The method primarily combines per-slot MLP scoring, confidence weighting, softmax, ordered concatenation, shared projection, and contrastive learning. Suppressing unreliable inputs based on their confidence is a well-established idea, and the log-confidence bias is equivalent to normalized multiplicative weighting. Simplicity is not itself a weakness, but the paper does not establish a sufficiently novel mechanism, explanation, or substantial practical benefit to justify this combination as a research contribution. In particular, under the same confidence signal, the MRR differences between CCSC and GateOnly range from only −0.0002 to +0.0052, leaving the necessity of the full learned architecture insufficiently established.
-
The comparisons do not adequately establish competitiveness. Table 1 mainly compares elementary composition strategies, including role-wise cosine matching, average pooling, SumMLP, and global scalar weighting, rather than establishing competitiveness against relevant existing retrieval approaches. Moreover, these baselines do not use the external confidence supplied to CCSC, so the reported improvements partly reflect access to additional reliability information. GateOnly is an important control, but strong simple alternatives directly matched to the setting—such as confidence-weighted role-wise matching and candidate filtering using high-confidence slots—are still missing.
-
The evaluation is highly controlled and remains substantially disconnected from the practical motivation. Correct slots always receive confidence 1, whereas incorrect slots receive values in [0.3, 0.6], perfectly separating correct from incorrect inputs. The paper does not evaluate actual grounder outputs or systematically examine partially informative confidence, overlapping confidence distributions, or confidently incorrect slots. The evaluation primarily measures recovery from artificially corrupted fields within known candidate catalogs. Although this is consistent with the stated experimental task, it provides insufficient evidence that the method addresses reliability challenges in practical intent-to-KG retrieval.
Questions for authors
What is the main methodological novelty of CCSC beyond applying existing confidence weighting and gating mechanisms to intent-to-knowledge-graph retrieval? Since GateOnly achieves most of the performance gains, what evidence shows the added value of the full CCSC design?
2 comments
Thank you for the additional experiments. They show conditional gains, but do not resolve my concerns about methodological novelty or establish sufficient added value over simpler alternatives. Several broader weaknesses in my review also remain insufficiently addressed. I therefore maintain my score of 2.
What CCSC brings to the table
Thank you for asking us to clarify the value of the complete design.
Q1. What is the main methodological novelty of CCSC beyond applying existing confidence weighting and gating mechanisms to intent-to-knowledge-graph retrieval?
CCSC combines learned content and supplied reliability in a role-aligned representation before a shared query–candidate projection. We compare it with GateOnly, which uses confidence-only weights and a learned projection, and with direct matching on the same inputs. NoConf retains the content scorer and shared projection but is trained without confidence; ON/OFF supplies/omits it at evaluation.
On CoDEx-M with 50% of present positions selected for within-role confidence reassignment, CCSC exceeds GateOnly by 0.0064±0.0014 MRR and NoConf receiving the same evaluation confidence by 0.0040±0.0004 (paired mean ± SD over three backbone means). Both differences are positive in all three means. Under submitted ties, it also exceeds squared-confidence cosine by 0.0190 and match-priority by 0.0117. These matched-input comparisons show additional retrieval utility of the fitted CCSC procedure in this condition.
At this level, one CoDEx-M relation-slot realization has correctness AUROC 0.7502. CCSC's own MRR decreases from 0.7303 to 0.7195. Reassignment changes only evaluation confidence, retaining the observations, masks, candidates and fitted models on previously evaluated panels.
Appendix B.1 gives the normalized-product equivalence. In Figure 3, squaring confidence preserves its ordering and improves synthetic Brier/ECE by construction, yet lowers fixed-CCSC mean MRR on all five panels. We propose revising the abstract's attribution of gains to match the comparisons of fitted procedures under common evaluation signals in Section 3.3.
Q2. Since GateOnly achieves most of the performance gains, what evidence shows the added value of the full CCSC design?
The submitted mean CCSC−GateOnly differences range from -0.0002 to +0.0052 MRR; Table 7 details their dependence on input correctness, including cases favoring GateOnly.
Under the original confidence and submitted tie rules, the table reports CCSC minus comparator (paired mean ± SD over three graph backbone means or five BINS repeats conditional on one backbone):
Squared cosine weights role-wise similarity by squared confidence. Match-priority places candidates matching all confidence-1 roles first, then uses squared cosine within each group. These full-catalog measurements are target-rank-equivalent to filtering with correct confidence-1 roles, identical scores/ties and full-catalog fallback for no selected role. Match-priority also leads on all three graphs under expected tie credit (CCSC differences -0.0321/-0.0156/-0.0412).
With naturally predicted roles on narrow BINS using frozen RoBERTa, the AP gains are 0.0191±0.0120 over GateOnly and 0.0063±0.0210 over NoConf-ON. GateOnly differences are positive in all five fitted states; NoConf differences have three positive and two negative states. NoConf-ON leads in the other three curated/narrow × TF–IDF/RoBERTa conditions. Curated/RoBERTa also favors linear matching by 0.0284 AP. These evaluations use 512 all-role-present development queries per condition from the historical grounder-training pool, with states sharing predictor outputs and queries; they are not independent generalization tests.
At 50% reassignment, mean MRR favors CCSC over GateOnly on the three graph panels and GateOnly on both BINS panels. In mean MRR, NoConf-ON leads by 0.0133/0.0215 on curated/narrow BINS and about 0.0005 on WN18RR/FB15k-237; CoDEx-M favors CCSC. These differences limit a general preference for fitting with confidence. At 75%, all five panels favor NoConf-OFF over CCSC-ON; at 100%, all favor NoConf-ON. Even the natural query-averaged gain can depend on aggregation: curated/TF–IDF CCSC−GateOnly changes from +0.0140 AP to -0.0019 under equal group weights.
In a separate single-state synthetic curated-BINS test with the object role missing, GateOnly had higher MRR@10 and lower latency. The comparisons do not establish every component's necessity or deployment superiority.
At 50% reassignment on CoDEx-M, the fitted CCSC procedure achieves higher mean MRR than GateOnly and NoConf-ON under the same evaluation confidence. We hope this clarifies the added value of the complete design in this tested setting.