- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper provides a profound investigation on influential factor of Prompt-Optimized Logic Translator. It has defined 4 factors: supervision, channel, notation / adaptor and translator unit. The motivations are clear that different paper has their own judger. And the experiments are valid to prove the claims.
Critical strengths
- The topic, and the claims provide insights to future research.
- The experiments settings are controllable and valid to support the claims.
- Clear failures and prompts are shown to provide reproducibility.
Critical weaknesses
- The presentation is a bit hard to follow. The narratives needs to be fixed for example the academic writing is hard to follow - some of them are too long and hard to comprehend such as line20-23 and may terms need to be introduced such as P=H、arity conflict、name variant- but it is minor.
- Section 4 lacks one comparison: you miss the reward only group
- (minor and maybe future work) 4 influential factors are separatedly investigated but what about their joint effects?
- Whether the judger is biased? - It is now Rocq + nltac while it has similar notation from silver settings.
Questions for authors
Please see above. I found this paper up to the standard of the acceptance threshold. I may reconsider the overall assessment during the discussion period once the author clarify my concerns.
Response to Reviewer 5Lsn
Thank you for the review. We answer your points in order. A short overview of the paper (pipeline, the feedback each condition receives, one example) is in our general response at the top of the forum.
1. Training reward vs. evaluation scoring. You are right. On a SICK pair labeled contradiction, the label reward used during optimization gives full credit to an unknown verdict as well as to a proved contradiction (lenient); the evaluation scoring gives credit only to the proved contradiction (strict). Why the reward was lenient: SICK's contradiction labels assume that the two sentences describe one scene, so the set contains pairs whose negation cannot be proved, and on those pairs an unknown verdict is not an error. Why the evaluation is strict: we chose the reading that does not credit silence, so that the SICK numbers are not inflated by contradictions nobody proved. Scored by the lenient score, the gap between S and the control on the matched pool shrinks from .080 to .046 but does not close (.127 to .074 on the full pool of §3). So the mismatch explains part of the gap, as you suggest, and not all of it. We will keep strict as the headline reading, report the lenient reading for SICK beside it, and give the gap under both.
2. Evaluation coverage (W1). The SICK list drops the neutral pairs, and the HANS list holds only entailment pairs, and you ask whether the results hold on balanced data and on the full test sets. HANS: non-entailment is evaluated on a separate list of 570 pairs whose gold is unknown (§2; table in the appendix). On a rebuilt version of that list, balanced over its fifteen constructions, the share of pairs wrongly proved is .24 for the control (.24 with the adapter), .18 for silver-view, .18 for S, and .16 for notation. On average, silver-view, S, and notation prove fewer wrong pairs than the control. The adapter gains seven points on the entailment list, while wrong proofs on this list increase by only two to four out of 570. The complete table will go next to Table 3. SICK: the neutral pairs are not evaluated, because the neutral class is not clean. For example, "A boy is standing outside the water" / "A boy is standing in the water" is labeled neutral, although its label in the B-to-A order is contradiction. In SICK's own directional labels, 319 of the 2,790 neutral test pairs (11%) carry a contradiction judgment in one direction, so a proof on a neutral pair is not always an error. We will state this as a limitation.
3. Translation quality (W2). We agree on both points. The silver formulas come from a CCG parser with hand-written templates. We keep a formula only where that pipeline proves the gold label. This does not guarantee that the formula preserves the meaning. What we can report is that the pipeline rarely proves something wrong (0 false decisive verdicts on SICK and 2 on MED, Appendix B). The merge rate can also flag valid paraphrases, and §5 says so: it is a diagnostic, not a direct measure of unfaithfulness. On pairs whose gold label is entailment, we cannot separate the two automatically. Where the gold label allows a check, the same rewriting produces wrong proofs: on the HANS non-entailment list no proof can be valid, and refinement still turns 59 pairs into proofs; on MED its wrong decisive verdicts rise from 67 pairs to 146 pairs (§5). We will check a sample of 100 merged pairs by hand and report the result.
4. Clarity (W3). We agree. We will add a pipeline figure with one worked example, and a list of what each condition receives as feedback, as in the general response.
5. Training data and presentation (§3). §3 compared L and S on different training pools, so the gap between them mixes the feedback with the data. To optimize against silver formulas, we first run the symbolic parser on every pair and keep only the pairs where it proves the dataset's own label. Every pair has a label, so L used all 16,734 pairs; 3,117 pairs remained for S. We included this comparison to ask whether label-only optimization can learn on the pairs the pipeline cannot handle. But it makes the paper harder to read. The updated paper will start from the matched-pool comparison, which you name as a strength; the full-pool table will move to the appendix, with no number removed.
We will post the updated paper with a list of changes.