From General Reward to Targeted Reward: Improving Open-ended Long-context Generation Models
Abstract
Current research on long-form context in Large Language Models (LLMs) primarily focuses on the understanding of long-contexts, the Openended Long Text Generation (Open-LTG) remains insufficiently explored.Training a longcontext generation model requires curation of gold-standard reference data, which is typically nonexistent for informative Open-LTG tasks.However, previous methods only utilize general assessments as reward signals, which limits accuracy.To bridge this gap, we introduce ProxyReward, an innovative reinforcement learning (RL) based framework, which includes a dataset and a reward signal computation method.Firstly, ProxyReward Dataset generation is accomplished through simple prompts that enables the model to create automatically, obviating extensive labeled data or significant manual effort.Secondly, ProxyReward Signal offers a targeted evaluation of information comprehensiveness and accuracy for specific questions.The experimental results indicate that our method Prox-yReward surpasses even GPT-4-Turbo.It can significantly enhance performance by 20% on the Open-LTG task when training widely used open-source models, while also surpassing the LLM-as-a-Judge approach.Our work presents effective methods to enhance the ability of LLMs to address complex open-ended questions posed by humans.
Authors: Zhihan Guo, Jiele Wu, Wenqian Cui, Yifei Zhang, Mindie Hu, Yufei Wang, Irwin King
Published in: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2025)