From Literature to Reasoning: Generalizable Reward Models for AI Optimization

AI4AI systems rely on reward models (RMs) to efficiently evaluate candidate system modifications. However, scenario-specific RM training often improves performance on seen tasks while degrading out-of-distribution (OOD) generalization. To address this issue, we propose LitRM, a literature-grounded reward modeling framework that transforms scientific knowledge into context-specific reasoning supervision. LitRM incorporates structured optimization knowledge through continual pretraining and retrieves mechanism-level evidence to guide a teacher model in reasoning over the current system state (BASE) and candidate modification (DIFF). The resulting reasoning guides evidence extraction, while joint training with teacher- and self-generated reasoning enables the student model to evaluate candidates without external retrieval at inference time. Experiments on multiple benchmarks show that LitRM consistently improves OOD generalization over current methods, yielding a $15.40$ percentage-point gain in average OOD AUC while maintaining strong performance on seen tasks.