1. Lack of engineering details in the paper (both main text and appendix)
In my opinion, this paper is an engineering-oriented dataset/environment work, so the engineering details are very important. However, some important information is missing; in particular, the construction process and the key parameters of the dataset are not reported:
- Simulators: the paper does not state who wrote the simulators for the four domains, whether open-source projects were used, or whether their dynamics were checked against real data.
- Generating and selecting candidate configurations: the paper does not state how many candidates are generated per template (4 in the code), which LLM is used (Kimi K3 is mentioned only in the AI use statement), what the prompt is, or what the sampling temperature is (0.7 in the code), nor does it explain why an LLM makes the choice.
- Rewriting the task description: the paper states neither the scope nor the purpose of the rewriting.
- Reference controllers: the paper only states that they consist of parameter estimation, state filtering, finite-step planning, and information acquisition; however, it gives no concrete decision rule for each domain and does not state who wrote them (a human or an LLM).
- Admission thresholds and test results: the paper does not give the thresholds ρ and δ or the confidence level α.
- The paper does not state the agent harness used for data collection; it gives neither the inference parameters (temperature, top-p, whether thinking mode is on) nor the collection budget, and it does not state how many times each instance was run.
- Filtering pass rate: Eq. 4 gives the threshold 0.2, but the values of the normalizing scale gψ are not given.
- (Optional) Release of example data: if some examples of the final SFT data and of the collected model trajectories were provided, they would give readers a solid and straightforward understanding.
As I believe the authors have a much clearer understanding of their own work, this weakness can be easily resolved as long as they provide this information during the rebuttal.
2. The external benchmark results are severely inconsistent with the original benchmarks
Some of the results look abnormal. For example, the CEO score that this paper reports for a small model (Qwen 27B) is higher than the best runs of large models (Opus 4.7, GPT-5.5) reported in the CEO-Bench paper. I believe the authors are responsible for explaining this phenomenon in the paper. Without a reasonable explanation, it is a major weakness.
The inconsistent/abnormal results are as follows:
-
The horizons differ from those of the original benchmarks: the paper's reference durations are 365 days for CEO and 30 days for Merchant, whereas the original benchmarks use 500 days for CEO-Bench (the original paper only defines a 50-day variant, not a 365-day one) and 365 days for MerchantBench (there is no 30-day simulation setting).
-
Merchant: the MerchantBench paper states "Each run starts with RMB 2,000 in cash, a RMB 1,000 security deposit" (RMB = CNY); the metric is final net assets over 365 days; all LLMs score between 6.56 and 59.46 thousand RMB (best: Qwen3.7-Max with Hermes, 59.46), humans score 217.61 thousand, and the rule-based policy scores 24.48 thousand; the original paper has no other starting-capital setting. In Table 11 of the paper, with only 30 days, 27B Base reaches 1,299,153.51 RMB, 35B Base 2,179,983.58 RMB, and 27B SFT 4,679,074.09 RMB; that is, Base is about 22 and 37 times the best LLM in the original benchmark, and about 6 and 10 times the human result. In Table 10, across 8 single-domain training conditions and 2,048 episodes, the lowest single-episode Merchant score is 1,291.2 thousand RMB (27B Network); even the worst episode is about 22 times the best original LLM and 6 times the human result, so the gap is not caused by a few outlier episodes.
-
CEO: in the CEO-Bench paper, each model starts with USD1M in cash, is scored on its cash at the end of 500 days, and is reported by its best of 3 runs; Claude Opus 4.7 reaches USD70,620, GPT-5.5 about USD33K, and Qwen 3.7 Max USD365,346; "GLM 5.1, DeepSeek V4 Pro, Gemini 3 Flash, and Grok 4.20 bankrupt on all runs"; the rule-based policy reaches USD15.76M, and the abstract states that "all evaluated models remain below the rule-based baseline". In the paper, 27B Base (not a pretrained-only base model; I think the authors mean the original Qwen model) goes from USD106,694 to USD202,036 after SFT, and 35B from USD94,738 to USD115,229. If the USD1M starting point is kept, both Base and SFT lose 80–90% of the capital, and "+89.4%" means losing somewhat less, which the paper does not mention. Moreover, the mean over 256 episodes of 27B Base is higher than the best of 3 runs of Opus 4.7 and GPT-5.5. Also, the lowest single-episode CEO score over 2,048 episodes is USD86.1 thousand (35B Grid), with no bankruptcy at all, whereas bankruptcy is common in the original benchmark. The paper reports no rule-based score on the external benchmarks.
That said, what still holds is that Base and SFT are compared under the same setting; thus, the finding that "SFT scores higher under the authors' setting" is not itself affected. What is affected is the claim of following the original benchmarks and the meaning of the external scores.
This is my major concern about this paper. As long as these abnormal/inconsistent issues can be addressed, I believe this paper can be a solid work.
3. The admission thresholds are very low
The threshold values can only be found in the code. In the repository, comparisons() in domains/inventory/synthesis.py requires the reference controller to beat the version with one capability removed by at least 0.01 in normalized score, and to beat the simple baseline by at least 0 (that is, not to be worse). In the acceptance certificates, the network and grid templates also use 0.01 for the capability gap and 0 for simple baselines.
This threshold seems too low to me, unless the authors can justify it or the code is actually outdated (i.e., not the final version used for the paper).
4. The writing is hard to follow
The writing of the paper is not very good: the sentences and paragraphs lack transitions, and many self-coined terms are used without introduction, which makes the paper difficult to read. For example, the abstract and the introduction use terms such as "recipe", "task family", "admission" and "reference controller" before they are defined, and without explanation.
These four weaknesses, especially the second one, raise my major concerns, so my current inclination is to reject. I actually prefer to rate this paper 1.5, however, there's no such an option. I look forward to hearing back from the authors; as long as these issues can be resolved, I would be very glad to raise my score.