Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments

This paper develops deep reinforcement learning methods for multi-agent domains. It first analyzes why traditional algorithms falter: Q-learning is challenged by the inherent non-stationarity of an environment with multiple learners, while policy gradient methods suffer from variance that increases as the number of agents grows. The authors then propose an adaptation of actor-critic methods in which each agent's critic considers the action policies of the other agents, allowing agents to learn policies that require complex multi-agent coordination. They further introduce a training regimen that uses an ensemble of policies for each agent.

The ensemble approach leads to more robust multi-agent policies that are less brittle to the changing behavior of other agents. Evaluated in both cooperative and competitive scenarios, the method outperforms existing approaches, with agent populations discovering a range of physical and informational coordination strategies. This provided a practical way to extend actor-critic learning to settings with multiple interacting agents, both cooperating and competing.