ReMAC: Large Language Model-Driven Reward Design for Multi-Agent
Manipulation Collaboration

Abstract

Multi-agent collaboration, such as in multi-robot systems, often relies on carefully crafted reward functions. These functions are crucial for learning collaborative policies. However, designing efficient reward functions for multi-agent systems remains an open challenge. To bridge this gap, we propose ReMAC, a novel large language model-driven Reward generation framework for Multi-Agent Collaboration. ReMAC employs a hierarchical approach to generate and optimize multi-agent reward functions: The upper level maintains and iteratively optimizes a population of reward functions from both team-level and individual-agent perspectives. The lower level applies multi-agent reinforcement learning algorithms (MARL) to learn collaborative policies. This hierarchical design ensures efficient learning and optimization of multi-agent policies. Motivated by recent advances in robotics, especially in embodied AI, we observe that existing multi-agent benchmarks fall short in supporting collaborative manipulation tasks. To bridge this gap, we design the Multi-Agent Manipulation Collaboration benchmark, ManiCraft, aiming to advance research on robotic manipulation in the MARL community. Experimental results demonstrate that ReMAC successfully constructs high-quality reward functions that outperform even those manually designed by human experts. All videos will be updated by May 25.

Benchmark Image

ReMAC Framework

ReMAC leverages the extensive domain knowledge and coding capabilities of LLMs to generate structured reward functions for MARL. Specifically, the LLM first analyzes the individual skills required by each agent and the coordination demands at the team level. Based on this analysis, ReMAC constructs two types of rewards—agent-level and team-level—which are then combined to produce the final reward for each agent.

To ensure high-quality reward design, we maintain a reward population \( \mathcal{P}_\text{R} \), where each individual comprises both agent- and team-level reward functions. For every reward function, we instantiate a corresponding MARL agent, forming a MARL population \( \mathcal{P}_\text{MARL} \). The team policies in \( \mathcal{P}_\text{MARL} \) interact with the environment to generate experiences. Each experience is then labeled with rewards by the reward population and stored in a shared replay buffer for learning.

At regular intervals, the best-performing team is summarized and fed back to the LLM, which reflects on the design from skill, individual, and team perspectives. Based on this reflection, the LLM generates improved reward functions to replace the suboptimal ones in \( \mathcal{P}_\text{R} \), which are then used for subsequent policy training.

Benchmark Image

Multi-Agent Manipulation Benchmark ManiCraft

Recent advancements in the robotics field, particularly in embodied intelligence, have been remarkable. However, we observe that the MARL community lacks a benchmark for multi-agent manipulation tasks aimed at low-level collaborative policy learning.

To bridge the gap, we propose ManiCraft, a benchmark that has the following key features:

  • Diverse MA-Manipulation Tasks: A diverse set of collaborative manipulation tasks designed to facilitate low-level coordination policy learning.
  • Easy to Use & Extend: Implemented following the Gym standard, with each task implemented in a single file, making it easy to use and extend.
  • Fine-grained design for MARL: Carefully designed action space, state spaces, and reward functions to ensure each task is learnable by MARL algorithms.

Specifically, ManiCraft is developed based on MuJoCo and utilizes MoCap for end-effector pose control. We design 11 manipulation collaborative tasks, which typically require the coordination of a UR robotic arm and a Franka Panda robotic arm, or the coordination of two UR robotic arms. Each robot is mounted on opposite sides of a table, with the target objects to be placed on the table.

More Collaborative Policy Visualization

Experiments

We thoroughly evaluate ReMAC and the baselines on ManiCraft to validate the effectiveness of ReMAC.




Evaluation Results


We first compare performance across 9 different tasks in the ManiCraft benchmark. ReMAC demonstrates performance on par with, and in some tasks superior to, MARL algorithms using human-designed reward functions. The MARL algorithms trained with sparse rewards consistently fail to learn effective collaborative policies across all tasks. Besides, ReMAC outperforms the multi-agent extension of Eureka in both efficiency and performance. This advantage stems from two key factors: (i) the construction of reward functions from both agent-level and team-level perspectives, and (ii) the ability of ReMAC to more effectively leverage experience collected from different teams.



ReMAC achieves sample efficiency comparable to or even better than that of manually designed reward functions. MA-Eureka is only able to learn effective collaborative policies on relatively simple tasks, such as Co-Close. When the task difficulty increases even slightly, MA-Eureka tends to fail.