- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the "update overflow" problem when the basic model updates the persistent structure state: even if the model correctly identifies the target field and its change direction, it may also modify the irrelevant field incorrectly. The author divides the status update into update range recognition, target action prediction and non target field maintenance, and proposes evaluation indicators such as conditional maintenance rate. This paper analyzes 205920 outputs of 13 models, and then examines the role of the update range and output interface through the "prediction/real update range × full/local output" experiment on ten models. The results show that correct prediction of the target does not mean that the remaining states can be maintained, and sparse output itself is not sufficient to ensure local updates.
Critical strengths
-
The thesis keeps a clear distinction between the correct target modification and the rest of the fields to avoid a single accuracy rate masking incidental modifications. Condition retention rate, overflow quantity and accurate transfer accuracy rate jointly describe different error sources, which is helpful to evaluate the model of maintaining external state.
-
The author cross controls whether the update range is provided, whether the output is localized, and compares the four protocols on the same sample and model. The experiment shows that providing the real range and limiting the execution range play a role respectively, while directly switching to sparse output may reduce the performance. This result is instructive in practice.
-
The paper covers multiple models, and adds experiments to clarify field evidence, maintain instruction strength, field serialization and interpretation stability. Self help sampling based on basic state clustering also considers the dependence between repeated observations.
Critical weaknesses
-
Security, activity, boredom and pedestrian facilities are semantically related, and the evidence describing the change of one attribute may support the change of other attributes at the same time. The generation process only specifies one target, which is not enough to prove that other changes are not supported by evidence. It is suggested to manually verify the unique update range of semantic evidence, and repeat the experiment on mutually independent fields to distinguish between true spillovers and reasonable association inference.
-
The complete output protocol also requires returning the main target field, while the sparse protocol requires generating a field list, and the local real range protocol further restricts the fields and value fields. Therefore, protocol differences involve not only the output range, but also field selection, format burden and task difficulty. It is suggested to supplement the comparison between unified semantic tasks and output constraints, and separately evaluate the prediction range of the model before implementing the two-stage scheme locally.
-
The experiment focused on the text tasks with eleven city attributes, single target and mainly extreme direction changes, and the output was also an increase or decrease action rather than a complete new value; All models use low bit quantization version. There is still a lack of verification of multi-objective, no update, continuous multiple rounds of accumulation and real state maintenance tasks.
Questions for authors
-
Have you checked the semantic evidence manually to support only one field change? For the associated fields such as safety and pedestrian facilities, activity and boredom, how consistent are the labels with the update range?
-
After all protocols unify the main target prediction requirements, field representation and decoding constraints, will sparse output still be weaker than full output on the nine models? What is the result of local update after updating the range of independent forecast?
3.Can the separation of target correctness and retention rate be reproduced in multiple targets, no update or continuous multiple rounds of status updates? Does the validation of a small number of non quantitative models get the same trend?