M³RL: Mind-aware Multi-agent Management Reinforcement Learning

As the main assumption and building block in economics, self-interested agents play a central roles in our daily life. Selfish agents, with their private beliefs, preferences, intentions, and skills, could collaborate (ad-hoc teaming) effectively to make great achievement with proper incentives and contracts, an amazing phenomenon that happens every day in every corner of the world.

However, most existing multi-agent reinforcement learning (MARL) methods focus on collaboration when agents selflessly share a common goal, expose its complete states and are willing to be trained towards the goal. While this is plausible in certain games, few papers address the more practical situations, in which agents are self-interested and inclined to show off, and only get motivated to work with proper incentives.

In this paper, we try to model such behaviors. We have multiple workers and a manager, together to work on a set of tasks. The manager gets an external reward upon the completion of some tasks, or one specific task. Each worker has a skill set and preference over the tasks. Note that their skills and preferences may not align with each other (Fig. 1(a)), and are not known to the manager (Fig. 1(b)). Furthermore, manager may not get any external reward until a specific task is complete, which depends on other tasks.

By default, the self-interested workers simply choose the most preferred tasks, which is often unproductive from the perspective of the entire project. Therefore, the manager gives additional incentives in the form of contracts. Each contract assigns a goal and a bonus for achieving the goal to a worker. With the external incentives, workers may choose different goals than their preferences. Upon completion of assigned goals, the manager receives the rewards associated with those goals and makes the promised payments to the workers. To generate optimal contracts, the manager must infer the workers’ minds and learn a good policy of goal and reward assignment.

Conventional approaches of mechanism design tackle similar problems by imposing strong assumptions (e.g., skill/preference distributions, task dependencies, etc) to find an analytic solution. In contrast, we aim to train a manager using reinforcement learning to i) assess minds of workers (skills, preferences, intentions, etc.) on the fly, ii) to optimally assign contracts to maximize a collaborative reward, and iii) is adapted to diverse and even evolving workers and environments.

For this, we propose a novel framework – Mind-aware Multi-agent Management Reinforcement Learning (M³RL), which entails both agent modeling for estimating workers’ minds and policy learning for contract generation. For agent modeling, we infer workers’ identities by their performance history, and track their internal states with a mind tracker trained by imitation learning (IL). For contract generation, we apply deep reinforcement learning (RL) to learn goal and bonus assignment policies. To improve the learning efficiency and adaptation, we also propose high-level successor representation (SR) learning (Kulkarni et al. 2016) and agent-wise $\epsilon$-greedy exploration.

As a proof of concept, we evaluate our approach in two environments: Resource Collection and Crafting in 2D Minecraft, to simulate multi-agent management problems. The setup and underlying assumptions are designed to mimic real world problems, where workers are not compelled to reveal their true preferences and skills, and there may be dependency between tasks resulting in delayed and sparse reward signals. Workers may also be deceitful (e.g., accepting a contract even when the assigned goal is unreachable). Our experiments demonstrate that the manager trained by our approach can i) estimate the mind of each worker from the recent behaviors, ii) motivate the workers to finish less preferable or intermediate tasks by assigning the right bonuses, iii) is adaptive to changing teams, e.g., change of members and/or change of workers’ skills and preferences, iv) and has good generalization in different team sizes and novel environments.

We have conducted substantial ablation studies by removing the key components, including IL, SR, agent-wise $\epsilon$-greedy exploration, and performance history. Our approach shows a consistent performance in standard settings as well as in more challenging ones where workers’ policies are stochastic and sub-optimal, or there are multiple levels of bonuses required to motivate workers.

Multi-agent reinforcement learning. For collaboration problems, common multi-agent reinforcement learning (Littman 1994; Busoniu et al. 2008) usually trains agents (Oliehoek et al. 2008; Foerster et al. 2016; Peng et al. 2017; Omidshafiei et al. 2017; Lowe et al. 2017) so that they will jointly maximize a shared reward. There also have been work on contributing different credits to agents by factorized value functions (Koller & Parr 1999; Guestrin et al. 2001; Sunehag et al. 2018; Rashid et al. 2018), but the spontaneous collaboration assumption is still required. In contrast, we instead train a manager to manage multiple self-interested workers for an optimal collaboration.

Principal-agent problems. Our problem setup is closely related to principal-agent problems (Laffont & Martimort 2002) (or moral hazard problems (Hölmstrom 1979)) in economics. Our manager and workers can be considered as the principal and agents respectively, where agents and principal have different objectives, and the principal needs to provide the right incentives to ensure that the agents make the best choices for what the principal delegates. These problems face similar technical challenges as our problem setup, e.g., information asymmetry between principals and agents, how to setup incentive cost, how to infer agents’ types, how to monitor their behaviors, etc. Traditional approaches in economics (Myerson 1982; Hölmstrom & Milgrom 1991; Sannikov 2008) build mathematical models to address these issues separately in stateless games, often with the assumption that the utility functions and the behavior patterns of the agents are known, leading to complicated models with many tunable parameters. In comparison, our paper provides a practical end-to-end computational framework to address this problem in a data-driven way without any assumption about the agents’ utilities and their decision making processes. Moreover, this framework is adaptive to changes of agents’ preferences and capabilities, which very few papers in economics have addressed. We also evaluate our approach in more complex game settings than the ones in the current economics literature.

Mechanism design. Similar to our problem setting, mechanism design also tackles problems where agents have different and private preferences (Myerson 1981; Conitzer & Sandholm 2002). Its core idea is to set up rules so that the agents will truthfully reveal their preferences for their own interests, and ultimately an optimal collective outcome can be achieved. Our work differs from mechanism design in several ways. First, in addition to preferences, we also acknowledge the fact that agents may have different skills. Second, mechanism design does not consider sequential decision problems, whereas we have to dynamically change the contracts over time.

Optimal reward design. The contract generation in our work can be seen as reward design. Some prior work has proposed optimal reward design approaches (Zhang et al. 2009; Zhang & Parkes 2008; Sorg et al. 2010; Ratner et al. 2018), where a teacher designs the best reward so that the student will learn faster or alter its policy towards the target policy. In contrast, we try to use deep RL to train optimal reward design policies to manage multi-agents in more complex tasks.

Meta-learning. Our work also resembles meta-learning (Wang et al. 2016; Finn et al. 2017), which typically aims at learning a meta strategy for multiple tasks (Maclaurin et al. 2015; Duan et al. 2017; Hariharan & Girshick 2017; Wichrowska et al. 2017; Yu et al. 2018; Baker et al. 2017) with good sample efficiency, or for a fast adaptation (Al-Shedivat et al. 2018). The meta-learning in this paper is for addressing the problem of ad-hoc teaming (Bowling & McCracken 2005; Stone et al. 2010) by training from a limited set of worker population.

Theory of Mind. Our agent modeling is inspired by the prior work on computational theory of mind, where both Bayesian inference (Baker et al. 2009) and end-to-end training (Rabinowitz et al. 2018) have been applied to understand a single agent’s decision making by inferring their minds. In this work, we extend this to optimal multi-agent management by understanding agents’ minds.

In an environment, there is a set of goals $\mathcal{G}$ corresponding to several tasks, $N$ self-interested workers with different minds, and a manager which can observe workers’ behaviors but is agnostic of their true minds. Different from the common Markov game setting for MARL in prior work (Littman 1994), we use an independent Markov Decision Process (MDP), i.e., $\langle\mathcal{S}_{i},\mathcal{A}_{i},R_{i},\mathcal{T}_{i}\rangle$, $\forall i\in N$, to model each worker, where $\mathcal{S}_{i}$ and $\mathcal{A}_{i}$ are the state space and action space, $R_{i}:\mathcal{S}_{i}\times\mathcal{G}_{i}\rightarrow{\mathbb{R}}$ is the reward function, and $\mathcal{T}_{i}:\mathcal{S}_{i}\times\mathcal{A}_{i}\rightarrow\mathcal{S}_{i}$ is the state transition probabilities. For achieving goals, a worker has its own policy $\pi_{i}:\mathcal{S}_{i}\times\mathcal{G}_{i}\rightarrow\mathcal{A}_{i}$. We define the key concepts in this work as follows.

Contract. A contract is a combination of goal and bonus assignment initiated by the manager to a specific worker. For simplicity, we consider discrete bonuses sampled from a finite set $\mathcal{B}$. Thus, for worker $i$ at time $t$, it will receive a contract defined as $(g_{i}^{t},b_{i}^{t})$, where $g_{i}^{t}\in\mathcal{G}$ is the goal and $b_{i}^{t}\in\mathcal{B}$ is the corresponding bonus for achieving the goal. Note that the contract will change over time.

Worker’s mind. We model a worker’s mind by its preferences, intentions, and skills. We do not study worker agents’ beliefs in this paper, which we leave as future work.

Preference. A worker’s preference is formally defined as its bounded internal utilities of achieving different goals, ${\bm{u}}_{i}=(u_{ig}:g\in\mathcal{G})$, where $0\leq u_{ig}\leq u_{\text{max}}$. Combined with received contract, the worker agent’s reward function can be defined as

where $s_{g}$ is the goal state.

Intention. The intention of a worker is the goal it is pursuing at any time, i.e., $\mathcal{I}_{i}^{t}\in\mathcal{G}$, which is not fully revealed to the manager. Based on the reward defined in Eq. (1), there are multiple ways to choose the goal. For a rational worker who is clear about its skills, it will choose the goal by maximizing expected return. I.e., $\mathcal{I}_{i}^{t}=\argmax_{g}\mathbb{E}[\sum_{t=0}^{\infty}\gamma_{i}^{t}r_{ig}^{t}]$, where $0<\gamma_{i}\leq 1$ is its discount factor. However, this requires a worker to have a good estimate of its skills and to be honest, which is not always true. E.g., a worker may want to pursue some valuable goal that it can not reach. So an alternative way is to maximize the utility instead: $\mathcal{I}_{i}^{t}=\argmax_{g}u_{ig}+\mathds{1}(g=g_{i}^{t})b_{i}^{t}$. This will make a worker’s behavior more deceptive as it may agree to pursue a goal but will rarely produce a fruitful result. In this work, we focus on the second way to achieve a more realistic simulation. After determine which goal to pursue, a worker will decide whether to sign the assigned contact. We denote this by $d_{i}^{t}\in\{0,1\}$, where $d_{i}^{t}=1$ means that worker $i$ signs the contract given at time $t$.

Skill. The skill of a worker is jointly determined by its state transition probabilities $\mathcal{T}_{i}$ and its policy conditioned on its intention, i.e., $\pi_{i}(\cdot|s_{i}^{t},\mathcal{I}_{i}^{t})$.

Manager’s objective. The manager in our setting has its own utility ${\bm{v}}=(v_{g}:g\in\mathcal{G})$, where $v_{g}\geq 0$ is the utility of achieving goal $g$. To maximize its gain, the manager needs to assign contracts to workers optimally. For the sake of realism, we do not assume that the manager knows for sure if a worker agent is really committed to the assignment. The only way to confirm this is to check whether the goal achieved by the worker is consistent with its last assigned goal. If so, then the manager will gain certain reward based on its utility of that goal and pay the promised bonus to the worker. Thus, we may define the manager’s reward function as:

where $S^{t+1}=\{s_{i}^{t+1}:i=1,\cdots,N\}$ is the collective states of all present worker agents at time $t+1$. The objective of the manager is to find optimal contract generation to maximize its expected return $\mathbb{E}[\sum_{t=0}^{\infty}\gamma^{t}r^{t}]$, where $0<\gamma\leq 1$ is the discount factor for the manager. Note that the manager may get the reward of a goal for multiple times if workers reach the goal respectively.

Population of worker agents. The trained manager should be able to manage an arbitrary composition of workers rather than only specific teams of workers. For this, we maintain a population of worker agents during training, and sample several ones from that population in each episode as the present workers in each episode. The identities of these workers are tracked across episodes. In testing, we will sample workers from a new population that has not been seen in training.

Our approach has three main components as shown in Figure 2: i) performance history module for identification, ii) mind tracker module for agent modeling, and iii) manager module for learning goal and bonus assignment policies. We introduce the details of these three components as follows.

In this paper, we propose Mind-aware Multi-agent Management Reinforcement Learning (M³RL) for solving the collaboration problems among self-interested workers with different skills and preferences. We train a manager to simultaneously infer workers’ minds and optimally assign contracts to workers for maximizing the overall productivity, for which we combine imitation learning and reinforcement learning for a joint training of agent modeling and management policy optimization. We also improve the model performance by a few techniques including learning high-level successor representation, agent-wise $\epsilon$-greedy exploration, and agent identification based on performance history. Results from extensive experiments demonstrate that our approach learns effectively, generalizes well, and has a fast and continuous adaptation.