- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper combines a probing/supervised fine-tuning study of temporal relations with a "selective-repair" intervention. The former takes frozen language models and trains linear probes against temporal relation ground truths for the objective of improving downstream classification (their research question 1 (RQ1)). This is essentially supervised finetuning a new classifier which uses an LM as a frozen backbone. They then see if these probes/classifiers transfer across corpora? (RQ2), and do they work for solving a synthetic graph-ordering task? (RQ3) They then consider how to combine these with the original backbone model's output as a hard-selection mixture of models(RQ4) (though it seems like RQ4 used differently trained probes for this "The preceding tests use independent logistic readouts. For Q4, we instead couple the models through a shared event-timeline decoder."). They present mixed results to these questions.
Critical strengths
None
Critical weaknesses
-
This is clearly AI slop from start to finish. The paper has many undefined terms. It begins with no motivation to the problem and instead jumps into technical explanations about what they did, in both the abstract and introduction.
-
The paper misses many critical details, making evaluating their experimental setup and, thus their results impossible.
-
Even if this paper were well written and had these missing details, and taking the paper's results at face value, I do not think the paper provided a significant or novel research contribution. Training a supervised classifier should outperform the baseline model (RQ1), so these results are not surprising. RQ2 and 3 showed either null results or minimal results, and again RQ4 just shows that a more complex model outperforms baselines. Basically, no ablation was done on the stronger predictive methods used.
-
The prompting procedure was insufficiently specified. Section 4.1 stated that only the first two answer options reversed their event arguments, while document markers remained unchanged, but it supplied neither prompt templates nor a worked forward/reverse example.
-
Appendices A, B, C, D, E, F, G, and J were explicitly referenced but absent from the paper. This includes references to missing figures.
Note that I have indicated a confidence of 4. This is not based on knowledge of this specific domain, but on a broader knowledge of what constitutes an acceptable research paper for this conference.
Also note that disclosing AI use does not alleviate these problems. The problem is that AI was improperly used in writing this paper, and I expect in the experimentation as well, not that AI was used at all.
Questions for authors
In your own words, can you please explain:
Q1) The problem statement and motivation for the paper?
Q2) Why was such a motivation not provided in either the abstract or the introduction, outside of `a hidden-state readout can recover a temporal relation that a language model answers incorrectly. The practical question is what to do with that signal'? In your experience reading other papers, does that provide a sufficient motivation for readers?
Q3) Why did you introduce technical terms without definition? For example, what is the difference between a `readout' and a linear probe?
Q4) What is the relationship between your four research questions? Is there a cohesive connection between these? What do you mean by `success at one interface does not waive the next test' and why did you phrase it that way? (line 107).
Q5) Can you explain where and how the experimental environment changes for these questions? For example, is research question 4 established in the same environment as 1-3?
Q6) Could you clarify how “reverse-order recall” was defined and calculated, and how it differed from the “reverse-direction questions” used to construct the readout features? A worked example showing the document’s event order, gold relation, forward and reverse prompts, and prediction scored for recall would help resolve this ambiguity.
Q7) 'Native decoding uses a cononcial four-option next-token decision' (line 214). What does 'cononical' refer to here, and why did you feel the need to specify it?
Q8) The AI use statement disclosed assistance with constructing controlled graph instances and preprocessing data. Please specify which operations involved AI? Also, please describe the human-level checks used to establish that the constructed instances preserved the source temporal relations and semantics.