Reflexion: language agents with verbal reinforcement learning

This paper proposes Reflexion, a framework for reinforcing large language model agents that interact with external environments as goal-driven agents. Rather than updating model weights, as traditional reinforcement learning would require through extensive samples and expensive fine-tuning, Reflexion agents learn through linguistic feedback. They verbally reflect on task feedback signals and maintain their own reflective text in an episodic memory buffer, which induces better decision-making on subsequent trials. The framework is flexible enough to incorporate different feedback types, whether scalar values or free-form language, and different sources, whether external or internally simulated.

Across diverse tasks including sequential decision-making, coding, and language reasoning, Reflexion obtains significant improvements over a baseline agent. A headline result is 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4, which achieved 80%. The authors also present ablation and analysis studies varying feedback signals, feedback incorporation methods, and agent types, offering insight into how each factor affects performance and how verbal reinforcement can substitute for weight updates.