We present results for our advisor evaluation algorithm (Algorithm <ref>) on the large state-action Pommerman environment [87]. The objective is to conduct the `pre-learning' phase to evaluate a set of advisors and pick a suitable $\epsilon'_0$ for learning using ADMIRAL-DM, which we  study in the upcoming sub-sections. We  use a two-agent version of Pommerman, which we denote as Domain OneVsOne of Pommerman (we will consider another domain of Pommerman shortly). Pommerman is a complex multi-agent domains, with each state containing more than 200 elements describing the position of the board, special features like bombs, and the position of other agents. Each agent can perform 6 actions, which include moving in the grid and laying bombs to kill the opponent. The reward function in Pommerman is quite sparse with the agents getting a +1 for winning the game, -1 for losing or a draw, with nothing in between. This game is general-sum since both agents get -1 for a draw. There is a maximum of 800 steps and the games where there are no winners after 800 steps are declared to be a draw. It is very hard for RL agents to learn good performance in Pommerman due to difficulties in balancing the twin goals of the killing of opponent and protecting themselves [26].

[Advisor 1]

[Advisor 2]

[Advisor 3]
[Advisor 4]
Analysis of ADMIRAL-AE algorithm on Pommerman (Domain OneVsOne) against (single-agent) DQN. The standard deviation in (a) and (d) are very small (negligible). The best advisor (Advisor 1) makes the agent reach the best overall performance. The performance steadily decreases from Advisor 1 to Advisor 4. All results are averages of 10 experimental runs ((a) and (d) have negligible standard deviations).

For these experiments, we use the neural network implementation of the ADMIRAL-AE algorithm (Algorithm <ref>). We  consider four different advisors.
Advisors are arranged in decreasing quality of advice, with Advisor 1 providing the best advice and Advisor 4 proving the worst (e.g. random advice).
The Advisors 1 and 2 have a positive influence on learning, as they can teach many useful techniques to win the game, while Advisor 4 has a negative influence on learning. The Advisor 3 is also capable of teaching some useful strategies and in general, is better than a random advisor (Advisor 4). However, it is much worse compared to Advisor 1 or Advisor 2.  Appendix <ref> contains  complete descriptions of the advisors and the implementation details of the algorithms used.

We conduct all experimental runs on 100,000 Pommerman games (episodes). Each episode is a full Pommerman game containing a maximum of 800 steps. Each experiment has a DQN (single-agent version as introduced in 65) agent and an ADMIRAL-AE agent training and competing against each other. The experiments analyze the performance of ADMIRAL-AE with each of the four advisors against the common opponent (DQN). The performance is plotted in Figure <ref>. We repeat the experiments 10 times and plot the averages and standard deviations. We observe that using the best advisor (Advisor 1) clearly results in the best performance of the ADMIRAL-AE algorithm, with an   overall cumulative reward reaching around 60,000 (Figure <ref>(a)). The second-best advisor (Advisor 2) results in cumulative reward around 35,000 (Figure <ref>(b)). When the ADMIRAL-AE uses Advisor 3 and Advisor 4, DQN results in  better performance cumulatively than the ADMIRAL-AE algorithm (Figures <ref>(c) and (d)).

The results (Figure <ref>) show that the ADMIRAL-AE algorithm can distinguish between different quality advisors. From Figure <ref>, it is clear that the advisor of choice for learning in this domain is Advisor 1. This result is obtained by running a separate instance of the ADMIRAL-AE algorithm with each of the advisors. This is consistent with our description of possible ways of evaluating the advisors using the ADMIRAL-AE algorithm in Section <ref>.

In Table <ref> we tabulate the results and normalize the average performances to obtain a suitable value for $\epsilon'_0$ (using Eq. <ref>). The procedure is the same as that adopted in Section <ref>. We adjust the column for maximum possible performance value for 10% random exploration as done in Table <ref>. The Advisor 1 along with its initial value of $\epsilon'$ is used in the next sub-section for learning using the ADMIRAL-DM method.

Advisor     Average cumulative reward (rounded to nearest 1000)     Maximum possible cumulative reward (adjusted for random exploration)     Average performance of the  ADMIRAL-AE using a random advisor (rounded to nearest 1000)     Normalized value (rounded up to nearest first decimal)
Advisor 1     58000     90000     -63000     0.8

Advisor 2     35000     90000     -63000     0.7

Advisor 3     -16000     90000     -63000     0.4

Advisor 4     -63000      90000     -63000     0

Finding $\epsilon'_0$ using ADMIRAL-AE for the Pommeran Domain OneVsOne against DQN.

§.§ Experimental Results - Function Approximation - ADMIRAL-DM

We now show that it is possible to extend our tabular ADMIRAL-DM method to function approximation based implementations that make our algorithms more generally applicable to environments with large state-action spaces. We will use the neural network-based version of ADMIRAL-DM as discussed in Section <ref>. Additionally, we show that ADMIRAL-DM is capable of outperforming several strong baselines from literature.

We perform comparative experiments in three domains. All our experiments in this section are repeated 30 times, and we plot the mean and standard deviation. The important elements of our experimental domains are mentioned here, while the complete details of the domains and implementation details of all algorithms are in Appendix <ref>. Neural network implementations of decision-making algorithms (ADMIRAL-DM, ADMIRAL-DM(AC)) are used in this sub-section. The first domain we consider is Domain OneVsOne of Pommerman introduced in Section <ref>. Our baselines are DQfD, CHAT, and DQN. We perform 50,000 episodes of training, where the algorithms train against specific opponents. Each episode is a full Pommerman game (lasting a maximum of 800 steps).
All the algorithms relying on demonstrations (DQfD, CHAT, ADMIRAL-DM, and ADMIRAL-DM(AC)) use the Advisor 1 considered in Section <ref>. The probability of using the advisor action ($\epsilon_t'$ in Algorithm <ref>) starts from 0.8 (obtained from Table <ref>) and linearly decays to be close to zero at the end of training for both ADMIRAL-DM and ADMIRAL-DM(AC). To provide data for offline pretraining in the case of DQfD, two instances of Advisor 1 is used to play many Pommerman games that generate the required data. The DQfD is pretrained with all of this data, before entering the training phase of our experiments.

[ADMIRAL-DM vs DQN]

[ADMIRAL-DM vs DQfD]

[ADMIRAL-DM vs CHAT]
[ADMIRAL-DM vs ADMIRAL-DM(AC)]

[Faceoff against ADMIRAL-DM]
Pommerman competition against ADMIRAL-DM. The ADMIRAL-DM beats all baselines in the execution phase. In the training phase ADMIRAL-DM(AC) performs better than ADMIRAL-DM in a head-to-head challenge.

After the training phase, the trained algorithms enter a face-off competition of 10,000 games where there is no more training, no further exploration and additionally ADMIRAL-DM and ADMIRAL-DM(AC) play without any advisor influence. ADMIRAL-DM(AC) is a CTDE technique, which
only performs decentralized execution in face-off using the trained actor-network.
We plot the cumulative rewards in the training phase (Figure <ref> (a), (b), (c), (d)), from which it can be seen that ADMIRAL-DM's performance is better than the baselines (DQN, DQfD, and CHAT). The face-off plots in Figure <ref>(e) show that ADMIRAL-DM wins more games on average against all the other baselines, showing its dominance. DQfD relies on pretraining, which is harder in MARL, as the nature of opponents that an agent will face during competition is impossible to determine upfront. The algorithms that use online advisors to give real-time feedback (that captures the changing nature of the opponent) tend to do better. DQfD has also been previously reported to have over-fitting issues [27], which is likely to hurt its performance more in multi-agent environments as compared to single-agent environments. In multi-agent environments, it is more important to be able to generalize to unseen dynamic opponent behaviour, which is different from that seen in pre-collected demonstration data. As discussed previously, CHAT maintains a confidence measure on the advisor, which depends on the advisor's consistency in action recommendations at different states. In MARL, this measure is not completely reliable, since even good advisors may need to formulate stochastic action recommendations as responses to the opponent. DQN, on the other hand, learns directly from interaction experiences and cannot learn from advisor inputs. This is a disadvantage in environments where external sources of knowledge, such as advisors, are available to be leveraged.
Furthermore, since our baselines are independent algorithms (that consider opponents to be part of the state), they lose out to ADMIRAL-DM, which explicitly tracks opponent action. ADMIRAL-DM loses to ADMIRAL-DM(AC) during training (Figure <ref> (d)). Though ADMIRAL-DM(AC) shows slower learning overall  (as it is training both actor and critic), it ultimately learns a higher performing policy. One important reason is that the actor-critic method trains a stochastic policy that can explore naturally, whereas the $Q$-learning method needs a hyperparameter to conduct a forced exploration ($\epsilon$-greedy). Another reason could be that ADMIRAL-DM(AC) learns from each recent experience, while the ADMIRAL-DM has delayed learning using the replay buffer. However, in the face-off, ADMIRAL-DM has an edge over ADMIRAL-DM(AC) (Figure <ref>(e)), probably due to being centralized. Since the performance of ADMIRAL-DM and ADMIRAL-DM (AC) in the face-off results given in Figure <ref>(e) are close, we perform a Fischer's exact test for the average performances to check statistical significance. We get p < 0.03 which shows that this result is statistically significant (we treat p < 0.05 as statistically significant as in common practice).

[ADMIRAL-DM(AC) vs DQN]

[ADMIRAL-DM(AC) vs DQfD]

[ADMIRAL-DM(AC) vs CHAT]
[Faceoff against ADMIRAL-DM(AC)]
Pommerman competition against ADMIRAL-DM(AC). ADMIRAL-DM(AC) defeats all other baselines in both the training and execution phases.

Next, we conduct similar experiments with ADMIRAL-DM(AC) that show it outperforms the baselines. The ADMIRAL-DM(AC) is explicitly compared to all the baselines in a training and face-off scheme similar to that done with ADMIRAL-DM. To recall, we perform training experiments of 50,000 full Pommerman games and face-off contests of 10,000 games, where the trained agents compete against each other without any further training or advisor influence. The results are plotted in Figure <ref>, where the ADMIRAL-DM(AC) shows better performance than the baselines (in both training and face-off phases). In training,  ADMIRAL-DM(AC) dominates all the other baselines by winning around 20,000 – 30,000 games out of the 50,000 games conducted. As observed in the previous experiments, the ADMIRAL-DM(AC) algorithm's learning is slower than that of the $Q$-learning variant, making the overall number of games won (captured by cumulative rewards), against the baselines to be lower than that of the corresponding training of ADMIRAL-DM against the baselines in Figure <ref>. In the face-off contests, the ADMIRAL-DM(AC) algorithm wins more than 50% of the games against all baselines except ADMIRAL-DM. As noted previously, the ADMIRAL-DM algorithm has a slight edge in performance over that of ADMIRAL-DM(AC) in the face-off stage.

Next, we use two cooperative domains from the Stanford Intelligent Systems Laboratory (SISL) [30]. These experiments  have two phases — training and execution. The algorithms train for 1000 games in the training phase and then enter an execution phase, where they execute the trained policy for 100 games. We choose to set the value of advisor influence $\epsilon_t'$ to 0.8 at the start of training and linearly decay it the same way as in the above experiments with Pommerman (since we are using good advisors). In the execution phase, there is no further exploratory actions for all algorithms, and no more influence of the advisor for ADMIRAL-DM and ADMIRAL-DM(AC).

[Waterworld - Training]
[Waterworld - Execution]

SISL Environments - Training and Execution. The ADMIRAL algorithms give a better performance than all the baselines in both the phases. Training graphs have been smoothed with a running average of 100.

The first SISL environment is a Pursuit environment, which contains 8 pursuers controlled by learning algorithms, trying to capture 30 evaders moving randomly in the grid-based environment. Rewards have a local structure, where the pursuers participating in the capture of an evader or the pursuers encountering evaders are rewarded individually. The game is general-sum and does not have a global reward structure. The local reward structure helps to tackle the issue of credit assignment.
We use a pre-trained policy of DQN (trained for 1000 episodes) as the advisor. We plot the reward obtained (averaged per agent) for the training and execution phases in Figures <ref>(a) and (b). The execution performance bars in Figure <ref>(b) is the average performance across the 100 execution games. The results show that ADMIRAL-DM has  better performance than all other baselines, including DQN used as the advisor, in both phases. Thus, our algorithm can ultimately outperform the advisor. This environment is highly non-stationary (due to having more number of learning agents), so completely centralized ADMIRAL-DM has an edge over ADMIRAL-DM(AC) which uses decentralized actors.

Since some performances in Figure <ref>(b) are close, we perform an unpaired 2-sided $t$-test for statistical significance. Regarding the performances of CHAT and ADMIRAL-DM(AC) we get a value of $p < 0.02$ and similarly for the performances of ADMIRAL-DM and ADMIRAL-DM(AC) we get a $p < 0.02$, which shows that both these comparisons are statistically significant.[We consider $p < 0.05$ as statistically significant.]

Our second SISL environment is the continuous action space Waterworld environment, which has 5 pursuer agents trying to consume food and avoid poison. The actions are continuous-valued thrust, that the agents can apply to move in a particular direction, and with a desired speed. Here, multiple pursuer agents need to work together to consume food.

Agents get rewards based on foods captured and punishments based on poison consumed. Rewards have a local structure similar to the Pursuit environment.
The advisor here is a pre-trained proximal policy optimization (PPO) [92] agent (trained for 1000 episodes).
Similar to the Pursuit environment, we plot the performances for both training and execution phases (averaged per agent) in the Waterworld environment (see Figures <ref>(c) and (d)).
The ADMIRAL-DM(AC) alone is used for these experiments, as the $Q$-learning variant is not applicable for continuous action spaces.
We use two popular RL algorithms for continuous control, PPO and deep deterministic policy gradients (DDPG) [55] as baselines.

The results show that ADMIRAL-DM(AC) has better performance than others in both phases (Figure <ref>(c) and (d)). The ADMIRAL-DM(AC) algorithm has two important advantages over the other baselines here. The first advantage is that it is capable of leveraging an advisor.  The second advantage is that ADMIRAL-DM(AC) is trained in a centralized fashion by tracking the opponent behaviour while the other algorithms are independent methods. Still, ADMIRAL-DM(AC) is decentralized in execution. Notably, the ADMIRAL-DM(AC) algorithm also improves upon PPO, used as the advisor, similar to our observation in the Pursuit environment.

In both the above SISL experiments, we see a small improvement in performance in the execution phases for both algorithms, ADMIRAL-DM and ADMIRAL-DM(AC), as compared to the final training performances. This is due to the fact that, at the end of the training, there is still a small amount of exploration and advisor influence (1 % of actions) involved, whereas during execution both these influences are completely removed, which contributes to a net improvement in performance.

To summarize, our experimental results show that the ADMIRAL-DM and ADMIRAL-DM(AC) algorithms  make the best use of advisors in multi-agent settings compared to the other state-of-the-art algorithms. After the advisor influence completely stops, the performance of ADMIRAL-DM and ADMIRAL-DM(AC) is better than the others. We have also demonstrated that our methods can be extended to continuous action spaces and work in decentralized environments using the popular CTDE technique.

§.§ Performance of ADMIRAL-DM Under The Influence Of Different Advisors

Next we  study the impact of using the ADMIRAL-AE in a `pre-learning' phase to determine the value of $\epsilon'_0$. Towards the same, we would like to use an algorithm to serve as a common opponent. We choose to use a different algorithm compared to the baselines considered in the previous sub-section (where the objective was to show better performances of ADMIRAL-DM as compared to these baselines). The algorithm we choose to use as the opponent is Deep Sarsa, which is similar to DQN but uses a “Sarsa-like” [104] Bellman update for the $Q$-values. We clarify that our objective in this section is not to show better performances against any baseline (which we have already done in Section <ref>).

Further, in this section, we provide additional experiments that evaluate ADMIRAL-DM on different advisors with the common opponent (Deep Sarsa) and show that ADMIRAL-DM is capable of recovering from bad action-advice. All results reported in this section use averages and standard deviations of 30 runs.

[Advisor 1]

[Advisor 2]

[Advisor 3]
[Advisor 4]
Results in Domain OneVsOne of Pommerman using different advisors with ADMIRAL-AE and Deep Sarsa.  The best advisor (Advisor 1) gives the highest overall performance while the worst advisor (Advisor 4) gives the lowest performance.

We describe two sets of experiments with two different domains of Pommerman. In the first set of experiments, we  use the neural network implementation of ADMIRAL-DM and ADMIRAL-AE on the Domain OneVsOne of Pommerman using the four different advisors introduced in Section <ref>. To recall, Advisor 1 is the best advisor who can give the best action (relative to other advisors) at all states and Advisor 4 is the worst. The quality of advisors reduces from Advisor 1 to Advisor 4. As mentioned, we use a common opponent as an agent playing Deep Sarsa.

First, we wish to evaluate the given advisors against the performance of Deep Sarsa in the `pre-learning' phase. To do this, we run a series of training experiments, where we implement ADMIRAL-AE using each of the four advisors against Deep Sarsa. The results are plotted in Figure <ref>. As observed earlier, the best advisor leads to the best overall performance and the worst advisor leads to the worst performance. Using these performances the $\epsilon'_0$ values are determined in Table <ref> (using Eq. <ref>). It can be seen that the suggested value of $\epsilon'_0$ is highest (0.9) for the best advisor and the smallest (0) for the worst advisor. These values of $\epsilon'_0$ show that the agent will listen more to the good advisors and listen less (or not at all) to the bad ones.

Advisor     Average cumulative reward (rounded to nearest 1000)     Maximum possible cumulative reward (adjusted for random exploration)     Average performance of agent using a random advisor (rounded to nearest 1000)     Normalized value (rounded up to nearest first decimal)
Advisor 1     63000     90,000     -54000     0.9

Advisor 2     34000     90,000     -54000     0.7

Advisor 3     -16400     90,000     -54000     0.3

Advisor 4     -54000     90,000     -54000     0

Finding an initial value for $\epsilon'_0$ using ADMIRAL-AE for the Pommeran Domain OneVsOne against Deep Sarsa.

Next, we run the ADMIRAL-DM algorithm against Deep Sarsa, using each of the four advisors. We use four initial values of $\epsilon'_0$ for each of the advisors, where one of these values  corresponds to the choice of $\epsilon'_0$ as obtained from our previous experiment with ADMIRAL-AE reflected in Table <ref>. In addition to these four values, we also consider a value of 0 for $\epsilon'_0$, which considers the performance of ADMIRAL-DM with no advisor inputs to serve as a baseline. As done previously, the value of $\epsilon'$ is decayed linearly, during training, for ADMIRAL-DM in all the experiments. All training is conducted for 100,000 episodes with the advisor influence ($\epsilon'$) being linearly decayed to 0 at 50,000 episodes, i.e. there is no advisor influence after 50,000 episodes. Each episode is a complete Pommerman game involving a maximum of 800 steps as in the previous experiments. More details of the game conditions and the advisors can be found in Appendix <ref>. The results showing the performance of ADMIRAL-DM in each of these experiments are presented in Figure <ref>.

[Advisor 1 ]

[Advisor 2]

[Advisor 3]
[Advisor 4]
Results in Domain OneVsOne of Pommerman using different advisors with ADMIRAL-DM and Deep Sarsa. The result plots show the performance of ADMIRAL-DM in the training against Deep Sarsa. The results show that $\epsilon'_0$ value obtained from the performance of ADMIRAL-AE in Table <ref> show either the best performance or is very close to the best possible performances amongst all the $\epsilon'_0$ values.

[OneVsOne - $\epsilon'=0.3$ ]

[OneVsOne - $\epsilon'=0.5$]

[OneVsOne - $\epsilon'=0.7$]

Results of ADMIRAL-DM vs Deep Sarsa using the Advisor 4, which is the worst advisor among the ones considered. The plots correspond to the OneVsOne domain. All the figures show that, ADMIRAL-DM is capable of recovering from bad action advice.  Greater influence of the bad advisor (larger $\epsilon'_0$) leads to a larger time needed for recovering from the bad action influence.

We highlight several observations. Figure <ref>(a) and (b) show that ADMIRAL-DM, using the good advisors (Advisor 1 and Advisor 2), achieves the maximum overall performance since the positive influence from the good advisors helps. However, if the advisor influence is limited ($\epsilon'=0.3$), the good advisors  only have a limited impact. As the value of $\epsilon'_0$ increases, we see that the performance of ADMIRAL-DM using the first two advisors improves (Figures <ref>(a) and (b)), as expected. Since Advisor 1 is even better than Advisor 2, we see from Figures <ref>(a) and (b) that ADMIRAL-DM using Advisor 1 clearly shows superior performance to that of Advisor 2 for the highest value of $\epsilon'_0$, 0.9. When $\epsilon'_0$ values were lower (such as 0.5), performance using both Advisor 1 and Advisor 2 are comparable since these advisors did not have many opportunities to make an impact. Notably, ADMIRAL-DM using Advisor 1 shows the best performance for $\epsilon'=0.9$, the value obtained from ADMIRAL-AE as seen in Table <ref>. ADMIRAL-DM using Advisor 2 almost provides the same performances for the highest values of $\epsilon'_0$, 0.7 and 0.9. Additional inputs from this advisor are not as useful (compared to Advisor 1), since it is weaker. Hence, a value of $\epsilon'=0.7$ as obtained from ADMIRAL-AE is sufficient for this advisor.

Turning our attention to the performance of ADMIRAL-DM with the third advisor, we find that it is significantly inferior compared to the other two advisors, yet still has a limited positive influence (Figure <ref>(c)). Furthermore,  the performance using this advisor is considerably better than using no advisor at all ($\epsilon'=0$). However, while using Advisor 3, we notice that, as the values of $\epsilon'_0$ increases, there is no appreciable improvement in performance. This shows that more influence of a comparatively less effective advisor does not lead to much improvement in performance. Again, the value suggested by ADMIRAL-AE (0.3), comes very close to the best possible performance with other values of $\epsilon'_0$.

The performance of ADMIRAL-DM using the last advisor (Advisor 4) is interesting. This advisor has a negative influence on learning and makes ADMIRAL-DM lose  for the first few episodes (Figure <ref>(d)). However, ADMIRAL-DM recovers after the advisor influence wanes in all cases. As the value of $\epsilon'_0$ increases, we see that further influence from the bad advisor is  detrimental and a larger number of episodes is required before ADMIRAL-DM shows signs of recovery from the poor advice. Hence, the best value of $\epsilon'_0$, in this case, is 0, since listening to this advisor only harms learning. Again, this was the value obtained for Advisor 4, in Table <ref>. We present a more elaborate set of results on the experiments with Advisor 4 in Figure <ref>. Here we show that for all cases of $\epsilon'_0$, ADMIRAL-DM is capable of recovering from bad action-advising and after a suitable number of episodes, can overtake the performance of Deep Sarsa. However, higher values of $\epsilon'_0$ makes the learning from the bad advisor  problematic, since while ADMIRAL-DM shows signs of recovery it still cannot overtake the cumulative performance of Deep Sarsa even after 300,000 episodes (Fig: <ref>(c)).

[Advisor 1]

[Advisor 2]

[Advisor 3]
[Advisor 4]
Results in Domain TwoVsTwo of Pommerman using different advisors with ADMIRAL-AE and Deep Sarsa.  The best advisor (Advisor 1) gives the highest overall performance while the worst advisor (Advisor 4) gives the lowest performance. The standard deviation of (a) and (d) are negligible.

We  now provide a brief description of another domain of Pommerman. Domain TwoVsTwo is a larger version of Pommerman, where there are a total of four agents, with two of the four belonging to the same team. The state space is much larger than Domain OneVsOne, with each state containing 372 elements. The reward function remains sparse, with the two agents belonging to the winning team getting +1 and the two agents belonging to the losing team getting -1 at the end of the game. In case of a draw, all the agents get -1. In Domain TwoVsTwo, we consider one team of Deep Sarsa and one team of the ADMIRAL-DM. This makes this domain harder than the Domain OneVsOne, as the agents must learn to cooperate amongst the members of the same team and compete against the members of the opponent team to win the game.  The Domain TwoVsTwo is a mixed competitive-cooperative domain, which is different from Domain OneVsOne that had only pure competition. We  use the same four advisors as considered before for the Domain TwoVsTwo.

Similar to the experiments with Domain OneVsOne, we first evaluate the advisors against Deep Sarsa using the ADMIRAL-AE algorithm. The results are presented in Figure <ref>. Again, the best advisor gives the maximum overall performance and the worst advisor results in  the minimum performance.  The $\epsilon'_0$ values are determined based on these results in Table <ref> (using Eq. <ref>). These values are used in further experiments using ADMIRAL-DM and Deep Sarsa in the Domain TwoVsTwo of Pommerman.

Advisor     Average cumulative reward (rounded to nearest 1000)     Maximum possible cumulative reward (adjusted for random exploration)     Average performance of agent using a random advisor (rounded to nearest 1000)     Normalized value (rounded up to nearest first decimal)
Advisor 1     79000     90000     -81000     1

Advisor 2     39000     90000     -81000     0.7

Advisor 3     -21000     90000     -81000     0.4

Advisor 4     -81000     90000     -81000     0

Finding $\epsilon'_0$ using ADMIRAL-AE for the Pommeran Domain TwoVsTwo against Deep Sarsa.

[Advisor 1 ]

[Advisor 2]

[Advisor 3]
[Advisor 4]
Results in Domain TwoVsTwo of Pommerman using different advisors with ADMIRAL-DM and Deep Sarsa. The result plots show the performance of team playing ADMIRAL-DM in training competitions against Deep Sarsa. For this domain too, $\epsilon'_0$ value obtained from the performance of ADMIRAL-AE in Table <ref> show either the best performance or is very close to the best possible performances amongst all the $\epsilon'_0$ values.

[TwoVsTwo - $\epsilon'=0.3$]
[TwoVsTwo - $\epsilon'=0.5$]

[TwoVsTwo - $\epsilon'=0.7$]

Results of ADMIRAL-DM vs. Deep Sarsa using the Advisor 4 in the TwoVsTwo domain. Similar to the first domain, all the figures show that ADMIRAL-DM is capable of eventually recovering from bad action advice.

Our next set of experiments train in the Domain TwoVsTwo with each agent in one team of Pommerman agents playing ADMIRAL-DM and each agent in the other team playing Deep Sarsa. We  consider four different initial values of $\epsilon'$ (where one of these values will correspond to the value obtained in Table <ref>), as we did in the  OneVsOne domain. In addition to these $\epsilon'_0$ values, we  continue to use the value of $\epsilon'_0=0$ as the baseline. This corresponds to the situation of ADMIRAL-DM learning with no advisor influence. All training is run for 100,000 episodes, with the value of $\epsilon'$ decaying linearly to 0  at 50,000 episodes. Thus, there is no more influence from advisors for the last 50,000 episodes of training. The results showing the performance of the team playing ADMIRAL-DM (in the competition against a team playing Deep Sarsa) are in Figure <ref>. The ADMIRAL-DM performances show similar characteristics to that seen in Domain OneVsOne. ADMIRAL-DM, using the best advisor (Advisor 1), shows the best performance for all values of $\epsilon'_0$ and its performance keeps improving with the increase in value of $\epsilon'_0$ (refer Figure <ref>(a)). Notably, comparing Figure <ref>(a) and Figure <ref>(a), the performance of ADMIRAL-DM using the best advisor is better in the case of Domain TwoVsTwo as compared to the Domain OneVsOne. This suggests that a good advisor has comparatively higher impact when tasks get harder. This is because there are far more strategies that need to be learned to do well in this domain and the opponent learning from scratch needs more time for learning the hard task. A good advisor, on the other hand, can teach the different strategies needed much faster and provide an early lead for ADMIRAL-DM. Similar observations apply to   the performances of ADMIRAL-DM using Advisor 2 as well (refer Figure <ref>(b) and Figure <ref>(b)).

For Advisor 3, the performance does not change much with varying values of $\epsilon'_0$ (refer Figure <ref>(c)) as observed in the case of Domain OneVsOne. Comparing Figure <ref>(c) and Figure <ref>(c), we note that the best performance of ADMIRAL-DM using  Advisor 3 is better for the Domain TwoVsTwo as compared to Domain OneVsOne, reinforcing our inferences earlier about a higher potential for impact in useful advisors when the tasks get harder. Regarding Advisor 4, the greater negative influence from this advisor necessitates a longer time for recovery (refer Figure <ref>(d)).

Again, from Figure <ref>, for all the four advisors, we  observe that the value for $\epsilon'_0$ as obtained from Table <ref> gives the best possible performance or comes quite close to the best possible performance, compared to other possible values of $\epsilon'_0$. This shows the advantage of evaluation using ADMIRAL-AE. In Figures <ref>(a), (b) and (c), ADMIRAL-DM shows signs of recovery for all values of $\epsilon'_0$ while using Advisor 4 for learning. However, for Advisor 4, the smaller the value of $\epsilon'_0$ the better. When making a comparison between  Figure <ref> and Figure<ref>, we note that, as the negative influence from the advisor increases (through a larger $\epsilon'_0$) the time needed for recovery also rises in the case of Domain TwoVsTwo and is greater than that needed in the Domain OneVsOne. As the complexity of the tasks increase, the agents need a lot more time to learn good policies that recovers the loss of performance from bad action-advice.  This shows that negative influence from an advisor is more costly in the case of harder MARL tasks/environments as against comparatively simpler environments.

From our experiments, we conclude that ADMIRAL-DM is capable of recovering from bad advisor recommendations and is able to suitably leverage  advisors who have some positive influence on learning. However, if possible, it is best to evaluate an advisor using the ADMIRAL-AE method and obtain a suitable initial value for the hyperparameter that determines the advisor influence ($\epsilon'_0$). This helps in learning good policies faster.

§.§ Summary

To summarize, in Section <ref> we showed an experimental illustration of our algorithms in a tabular domain. We showed that, while using ADMIRAL-AE, the best advisor gives the best overall performance. Further, ADMIRAL-AE provides a suitable value for the hyperparameter $\epsilon'_0$, which when used by ADMIRAL-DM subsequently, provides good performances with different types of advisors. We also provided an experimental illustration of our theoretical convergence results in the case of both ADMIRAL-DM and ADMIRAL-AE. In Section <ref> we provided an illustration of ADMIRAL-AE in a large environment with neural networks as function approximators. Again, we illustrated that using ADMIRAL-AE with the best advisor provides the best performance amongst other (comparatively worse) advisors. Obtaining an appropriate value for $\epsilon'_0$ from Section <ref> we showed that ADMIRAL-DM and ADMIRAL-DM(AC) provide better performances than a set of baselines in Section <ref>. We tested our algorithms in both competitive and cooperative domains, as well as settings with discrete and continuous action spaces.

In Section <ref>, we used two Pommerman domains to illustrate that using ADMIRAL-AE to obtain a suitable value for $\epsilon'_0$, provides the best performance for ADMIRAL-DM. Hence, when possible, it would be best to use ADMIRAL-AE for determining $\epsilon'_0$ using a pre-learning phase. Also, we illustrated that ADMIRAL-DM is capable of recovering from bad action advice from advisors if appropriate values for $\epsilon'_0$ cannot be determined before training ADMIRAL-DM.

Additionally, in Appendix <ref> we show another important advantage of using the principled method of ADMIRAL-AE for advisor evaluation in environments having dynamically learning and adapting advisors. We show that a principled method like ADMIRAL-AE would find a suitable value for $\epsilon'_0$ when it is presented with a learning advisor, where other methods based on simple heuristics may have a high chance of failure.

§ CONCLUSION

In this paper, we  introduced the problem of learning under the influence of external advisors in MARL. We provided a  principled framework for MARL algorithms learning to use advisors.
Using $Q$-learning based methods, we proposed two MARL algorithms for this problem. We conducted theoretical analyses of these algorithms, establishing conditions under which fixed point guarantees can be provided regarding their learning in general-sum stochastic games. We  proved that previous theoretical results can extend to this setting under a comparatively weaker set of assumptions than previously considered. Empirically, we showed that our algorithms can be scaled to domains with large state-action spaces using traditional function approximators like neural networks. We also introduced an additional actor-critic variant of our ADMIRAL-DM algorithm that can operate under the CTDE paradigm and can learn in environments with continuous action spaces. Our empirical results further established the superiority of our algorithms compared to standard baselines. Furthermore,  we have shown that our methods would be useful in a wide variety of problems and that the algorithms can recover from the influence of weak/bad advisors during learning.

While there is a rich body of literature on the use of external knowledge sources in single-agent RL [5], MARL provides additional challenges which mean that not all the results and approaches can transfer over directly. We discussed the important problems of directly using the single-agent based methods that learn from external sources in MARL. Additionally, we performed direct comparison experiments to elucidate a few of these problems.  Our approach to learning from advisors in MARL may look more complex compared to other single-agent approaches, however, the non-stationarity of the environment makes learning under the influence of advisors in MARL considerably more challenging. In MARL, quick adaptation to the changing environment is the key to better performance [59]. Our approach of using an online advisor is a more appropriate formulation of advisors in MARL, as real-time feedback against non-stationary opponents are critical for learning effective multi-agent policies, as demonstrated in our experiments.

Importantly, we consider a general setting, where we had no restrictions on the type or quality of the advisor, and no restrictions on the relation between the reward functions of different agents (general-sum). Particularly in MARL, the assumption of optimal advisors could be overly strong, since performance depends on the nature of opponents. The advisor could be capable of providing good feedback in strategizing against a particular class of opponents yet be useless against another class of opponents. Furthermore, a sub-optimal advisor could be good only in a very narrow portion of the state space, which is still useful for an agent learning from scratch in this environment. By explicitly allowing nonrestrictive sub-optimal advisors, our work is more widely applicable than previous methods that make an assumption of optimal (or near-optimal) experts to help RL training [88, 28, 100].

From the empirical perspective, as future work, we would like to study the performance of ADMIRAL algorithms under the simultaneous influence of multiple advisors providing conflicting demonstrations. This problem has been studied in single-agent RL environments [54], but not yet in the MARL context. There is an emerging line of work that studies the possibility of multiple agents learning from peers in cooperative MARL settings [73]. Our paper has the potential to contribute to this line of work as well. Further, in this paper, we provided a simple technique of making use of the evaluation of advisors in a learning algorithm by setting the value of $\epsilon'_0$. More sophisticated ways of analyzing the performance of ADMIRAL-AE and using the results for learning faster and more effective decision-making policies is left to future work. An observation about ADMIRAL-AE is that, in MARL, the advisors can be used as a way to predict the behaviour of other agents as well, which is not relevant in single-agent settings. In MARL, each agent needs to have the ability to perform accurate opponent modelling, based on its observations, to obtain strong performances [34]. This is because the reward function and the transition dynamics depends on the joint action at each state. Previous methods have used several techniques, such as using a separate neural network for learning opponent behaviour [33], learning policy features from raw observations [37], and using the agent's own policy to predict opponent actions [81]. However, many of these methods are computationally expensive and scale poorly with the number of states, actions, and agents. Another possibility for opponent modelling is leveraging an external advisor that can possibly predict opponent behaviour, as done in ADMIRAL-AE, which could be relatively computationally friendly given the availability of an appropriate advisor. This could open up a very interesting research direction in learning from advisors in MARL. Both our ADMIRAL-DM and ADMIRAL-AE algorithms are exponential in the number of agents as described in our complexity analysis (Section <ref>). Hence, our algorithms are not easily scalable to environments with a large number of agents which is one limitation of our framework. As future work, our framework could be combined with works on mean field games [49] which can make the approach more scalable (since mean field methods guarantee a constant dependence on the number of agents).

From the theoretical perspective, as future work, we would like to fully characterize the convergence rates of our algorithms. Additionally, some of our theoretical assumptions for the environmental settings may seem restrictive, however, we assert that this work is the first to provide a theoretical foundation for MARL with advisors and that these assumptions are useful in understanding the strengths and limitations of such an approach. Furthermore, we note that the assumptions we make are also made by other works exploring the foundations of MARL, such as 38. In future work, we wish to explore the ramifications of relaxing some of these assumptions. The theoretical understanding of MARL, in general, is still in its infancy and much more research into MARL theory is required to enhance our understanding of this area [128]. In this paper, we restrict the theoretical analysis to tabular settings, which is in line with the state-of-the-art in theoretical analysis of learning in general-sum stochastic games [128]. The objective is to provide a theoretical guarantee in the most basic (baseline) setting possible. Using a similar approach to single-agent RL methods that extend the tabular results to the function approximation setting [9], it would be possible to extend our theoretical results in this paper to the function approximation setting as well. An elaborate theoretical study of this is left to future work.

Two specific motivational applications for our work were discussed in Section <ref>. Similar to these examples, other domains such as managing natural disasters, controlling disease outbreaks, and learning to play multi-player/multi-team sports would also find benefit from our approach. These domains also have state-of-the-art advisor models. These models, while not optimal, are still used in practice. Due to the inherent poor sample efficiency of MARL methods, it becomes critical to use all available sources of knowledge judiciously. Hence, learning from fallible advisors is important for many real-world domains that typically have the structure of multiple agent interaction.

The framework we have introduced in this paper on using external information through advisors is expected to improve the sample complexity of MARL algorithms and is an important step towards making MARL methods usable in real-world environments. We expect that our paper will spark more work in the  area of accelerating RL training using other available sources of knowledge (or advisors) and that it will interest a broad community of researchers in the area of RL, game theory, machine learning, and multi-agent systems.

§ ACKNOWLEDGEMENTS

Kate Larson is an affiliate of the Vector Institute, Toronto. The authors would like to thank Seyed Majid Zahedi at  University of Waterloo for his comments on the paper draft. Resources used in preparing this research at the University of Waterloo were provided by the province of Ontario and the government of Canada through CIFAR, NRC, NSERC and companies sponsoring the Vector Institute. Part of this work has taken place in the Intelligent Robot Learning (IRL) Lab at the University of Alberta, which is supported in part by research grants from the Alberta Machine Intelligence Institute (Amii); a Canada CIFAR AI Chair, Amii; Compute Canada; Mitacs; and NSERC.

§ PROOF FOR THE LEMMAS IN THE MAIN PAPER

In this section, we will restate all the lemmas in the main paper with detailed proofs. No new lemmas are given in this section.

Let us fix an arbitrary positive constant $C$, an arbitrary $w_0$, and a sequence $\epsilon$. Then provided that

(i) $ y_t(w_0, \epsilon)$ converges to some point (independent of $t$) $\mathcal{D}$

(ii) The sequence $\epsilon$ converges to 0 in the limit ($t \xrightarrow{} \infty$)

The homogeneous process $x_t(w_0, \epsilon)$ converges to a point $\frac{1}{\hat{\beta}} \mathcal{D}$ w. p. 1, where $\hat{\beta}$, satisfying $0 < \hat{\beta} \leq 1$, is the scaling factor applied.

We state that

\begin{equation}\label{eq:rescaling}
\begin{array}{l}
y_t(w, \epsilon) = x_t(d_t w, c_t \epsilon_t)
\end{array}
\end{equation}

for some sequences $\{c_t\}$ and $\{d_t\}$, where $c_t = (c_{t0}, c_{t1}, \ldots, c_{ti}, \ldots)$, $\{c_t\}$ and $\{d_t\}$ satisfy $0,d_t, c_{ti} \leq 1$, and $c_{ti} = 1$ if $i\geq t$. Here, the product of the sequences $c_t$ and $\epsilon_t$ is component wise: $(c_t \epsilon_t)_i = c_{ti} \epsilon_i$. Note that $y_t(w, \epsilon_t)$ and $x_t(w,\epsilon_t)$ depend only on $\epsilon_0, \cdots, \epsilon_{t-1}$. Thus, it is possible to prove Eq. <ref> by constructing the appropriate sequence $c_t$ and $d_t$.

Set $c_{0i} = d_i = 1$ for all $i = 0,1,2, \ldots$. Then Eq. <ref> holds for $t=0$. Let us assume that $(c_i, d_i)$ is defined in a way that Eq. <ref> holds for $t$. Let $B_t$ be the “scaling coefficient” of $y_t$ at step $t+1$ ($B_t = 1$ if there is no scaling, otherwise $0<B_t<1$ with $B_t = C/||G_t(y_t, \epsilon_t||$). Now we have:

\begin{equation}
\begin{array}{l}
y_t(w, \epsilon_t) = B_t G_t (y_t(w,\epsilon_t), \epsilon_t) \\

= G_t(B_t y_t(w, \epsilon_t), B_t \epsilon_t)\\

= G_t(B_t x_t(d_tw, c_t\epsilon_t), B_t \epsilon_t).
\end{array}
\end{equation}

We claim that

\begin{equation}\label{eq:claimforS}
\begin{array}{l}
B{x_t} (w, \epsilon_t) = x_t(B w, B \epsilon_t)
\end{array}
\end{equation}

holds for all $w$, $\epsilon_t$ and $B>0$. For $t = 0$, this obviously holds. Assume that it holds for some time $t$. Then, from Eq. <ref>,

\begin{equation}
\begin{array}{l}
B{x_{t+1}} (w, \epsilon_t) = B G_t (x_t(w, \epsilon_t), \epsilon_t)\\
= G_t(B{x_t}(w, \epsilon_t), B\epsilon_t)
= x_{t+1}(B w, B \epsilon_t)
\end{array}
\end{equation}

\begin{equation}
\begin{array}{l}
y_{t+1}(w,\epsilon_t) = G_t (x_t (B_t d_t w, B_t c_t \epsilon_t), B_t \epsilon_t),

\end{array}
\end{equation}

and we see that Eq. <ref> holds if we define $c_{t+1,i}$ as $c_{t+1,i} = B_t c_{ti}$ if $0 \leq i \leq t$, $c_{t+1, i} = 1$ if $i>t$ and $d_{t+1} = B_t d_t$.

Thus, we get that with the sequences

\begin{equation}
\begin{array}{l}
c_{t,i} = \Pi_{j=i}^{t-1}B_j, \textrm{ if } i<t;
\\
c_{t,i} = 1, otherwise;

\end{array}
\end{equation}

$d_0 = 1$ and

\begin{equation}
\begin{array}{l}
d_{t+1} = \Pi^t_{i=0} B_i,
\end{array}
\end{equation}

Eq. <ref> is satisfied for all $t\geq0$.

We know that $y_t(w, \epsilon_t) \xrightarrow{} \mathcal{D}$ w. p. 1. Since the process $y_t$ has been constructed by bounding the process with $|| y_t ||\leq C$, it follows that $\mathcal{D} \leq C$. Then, there exists a finite index $M$ such that if $t>M$ then

\begin{equation}\label{eq:probability}
\begin{array}{l}
Pr(||y_t(w, \epsilon_t)|| < C) > 1 - \delta
\end{array}
\end{equation}

without applying any more rescaling. Now, let us restrict our attention to those events $\omega$ for which $||y_t(w, \epsilon_t(\omega))|| < C$ for all $t > M$ without rescaling. Thus, we have no rescaling for all $t$, such that $t>M$. Thus, $c_{t,i} = c_{M+1,i}$ for all $t \geq M+1$ and $i$, and specifically $c_{ti} = 1$ if $i,t \geq M + 1$. Similarly, if $t>M$ then $d_{t+1}(\omega) = \Pi_{i=0}^{M} B_i (\omega) = d_{M +1} (\omega)$. Let $A_\omega = \{\omega: ||y_t(w, \epsilon)(\omega)|| < C\}$ without rescaling. By Eq. <ref>, we have that if $t>M$ then,

\begin{equation}
\begin{array}{l}
y_t(w, \epsilon_t(\omega)) =
x_t(d_{M+1}(\omega)w, c_{M+1}(\omega) \epsilon_t(\omega)).
\end{array}
\end{equation}

Thus, it follows from our assumption concerning $y_t$ that $x_t(d_{M+1}(\omega) w, c_{M+1} \epsilon_t(\omega))  $ converges to $\mathcal{D}$ almost everywhere (a. e.) on $A_\omega$ and, consequently, by Eq. <ref>, $x_t(w, c_{M+1}\epsilon_t(\omega)/d_{M+1}(\omega))$ converges to $\mathcal{D}' = \frac{1}{d_{M+1}}\mathcal{D}$ a. e. on $A_{\omega}$. Since $c_{M+1} = 1$ in the limit and we are analyzing in the space of $A_\omega$, $x_t(w, \epsilon_t(\omega)/d_{M+1}(\omega))$ converges to $\mathcal{D}'$ too. Now, since $\epsilon$ converges to 0 in the limit, and we are analyzing values in the space of $A_\omega$, we can write $x_t(w, \epsilon_t(\omega)) \approx x_t(w, \epsilon_t(\omega)/d_{M+1}(\omega)))$ which also converges to $\mathcal{D}'$. All these hold with probability at least $1-\delta$, since, by Eq. <ref>  $Pr(A_{\omega} > 1- \delta)$. Since $\delta$ was arbitrary, the lemma follows.

Let $X$ and $Y$ be normed vector spaces, $U_t: X \times Y \xrightarrow{} X (t=0,1,2, \ldots)$ be a sequence of mappings, and $\theta_t \in Y$ be an arbitrary sequence. Let $\theta_\infty \in Y$ and $x_\infty \in X$. Consider the sequences $x_{t+1} = U_t(x_t, \theta_\infty)$, and $y_{t+1} = U_t(y-t, \theta_t)$ and suppose that $x_t$ and $\theta_t$ converge to $x_\infty$ and $\theta_\infty$ respectively, in the norm of the appropriate spaces.

Let $L^\theta_k$ be the uniform Lipschitz index of $U_k(x,\theta)$ with respect to $\theta$ at $\theta_\infty$ and, similarly, let $L^\mathscr{X}_t$ and $L^\theta_t$ satisfy the relations $L^\theta_t \leq C(1 - L^\mathscr{X}_t)$, and $\Pi_{m=t}^\infty L^\mathscr{X}_m = 0$ where $C>0$ is some constant and $t = 0,1,2, \ldots,$ then $\lim_{t \xrightarrow{} \infty} || y_t - x_\infty|| = 0$.

See Theorem 15 in 107 for detailed proof.

Let $\mathcal{Z}$ be an arbitrary set and consider the process

\begin{equation*}
\begin{array}{l}
x_{t+1}(z) = G_t(z)x_t(z) + F_t(z) (C + k_t(z) C)
\end{array}{}
\end{equation*}

where $x_1, F_t, G_t \geq 0 $ are random processes,  $||x_1|| < C < \infty$ w. p. 1 for some $C>0$, and $z$ is an element in $\mathcal{Z}$. Assume that for all $k$, $\lim_{n \xrightarrow{} \infty} \Pi_{t=k}^n G_t(z) = 0$ uniformly in $z$ w. p. 1 and $F_t(z) = \gamma(1 - G_t(z))$, for some $0 \leq \gamma < 1$, and $\forall z \in \mathcal{Z}$,  w. p. 1. Also, $k_t(z)$ converges to $K(z)$ in the limit. Then, $x_t(z)$ converges to a point $D(z) = \gamma(C + K(z)C)$ w. p. 1.

Consider the process that is obtained from substituting the value of $F_t(z)$ in terms of $G_t(z)$ in Eq. <ref>,

\begin{equation}\label{eq:limitlemma2}
\begin{array}{l}
x_{t+1}(z) = G_t(z)x_t(z) + \gamma(1 - G_t(z)) (C + k_t(z) C).
\end{array}{}
\end{equation}

Now, subtracting $\gamma (C + k_t(z)C)$ on both sides of the equation we get

\begin{equation}\label{eq:limitlemma3}
\begin{array}{l}
x_{t+1}(z) - \gamma (C + k_t(z)C) =  G_t(z)(x_t(z) - \gamma (C + k_t(z) C)).
\end{array}{}
\end{equation}

The above equation converges to 0 in the limit as $\lim_{n \xrightarrow{} \infty} \Pi_{t=k}^n G_t(z) = 0$. Thus, we have that $x_{t+1}(z) - \gamma (C + k_t(z)C)$ converges to 0 in the limit. Hence, $x_t$ converges to $\gamma (C + K(z)C)$, since $k_t(z)$ converges to $K(z)$ in the limit.

Consider an equation of the form

\begin{equation*}
\begin{array}{l}
x_{t+1}(z) = G_t(z) x_t(z) + F_t(z) (||x_t|| + \epsilon_t + k_t(z) ||x_t||)
\end{array}
\end{equation*}

where the sequence $\epsilon_t $ converges to zero w. p. 1. Assume that for all $k$, $\lim_{n \xrightarrow{} \infty} \Pi_{t=k}^n G_t(z) = 0$ uniformly in $z$ w. p. 1 and $F_t(z) = \gamma(1 - G_t(z))$, for some $0 \leq \gamma < 1$, and $\forall z \in \mathcal{Z}$,  w. p. 1. Assume further that $k_t(z)$ is finite, and it converges to $K(z)$ in the limit ($t \xrightarrow{} \infty$). Then $x_t(z)$ converges to a point represented by $S'(z) = \frac{1}{\hat{\beta}} (\gamma C_1 + K(z) C_1) $, where $C_1$ is a small positive constant, w. p. 1 uniformly over $\mathcal{Z}$. Here $\hat{\beta}$ is a scaling factor satisfying $0 < \hat{\beta} \leq 1$.

Let us consider a process $y_t$ that is obtained from keeping the original process $||x_t||$ bounded by a constant $C_1$. This is an arbitrary bound, with $C_1$ specified to be a small positive constant. Since, $||x_t||$ is guaranteed to be positive, we can find such a positive $C_1$. Now we get,

\begin{equation}\label{eq:originalprocess}
\begin{array}{l}
y_{t+1}(z) = G_t(z) y_t(z)  + \gamma (1 - G_t(z)) (C_1 + \epsilon_t + k_t(z) C_1 ).
\end{array}
\end{equation}

By Lemma <ref>, $y_t$ converges to $\gamma C_1 + K(z) C_1 $ as the following bindings show $X, Y := \textbf{R}, \theta_t := \epsilon_t, U_t(x,\theta) := G_t(z)x + \gamma (1 - G_t(z)) (C_1 + k_t(z) C_1 + \theta)$, where $z \in Z$ is arbitrary. Then, $L^X_t = G_t(z)$ and $L^\theta_t = \gamma(1 - G_t(z))$ satisfying the conditions of Theorem <ref>. We know that $x$ in the expression $U_t(x,\theta)$ converges to $\gamma C_1 + K(z) C_1 $ as proved in Lemma <ref>.

In Lemma <ref>, we proved that the original process will converge to a point $\frac{1}{\hat{\beta}} \mathcal{D}$, if the bounded process converges to $\mathcal{D}$. Here $\hat{\beta}$ is the scaling factor applied, satisfying the relation $0 < \hat{\beta} \leq 1$. Using this result, now we get that the process represented by the Eq. <ref> converges to a point $S'(z) = \frac{1}{\hat{\beta}} (\gamma C_1 + K(z) C_1) $ which is a constant for a given $z$. This gives an expression for the point $S$ in Theorem <ref>.

Let $\mathcal{F}_t$ be an increasing sequence of $\sigma$-fields, let $0\leq \alpha_t$ and $w_t$ be random variables such that $\alpha_t$ and $w_{t-1}$ are $\mathcal{F}_t$ measurable. Assume that the following hold w. p. 1: $E[w_t|\mathcal{F}_t, \alpha_t \neq 0] = A$, $E[w^2_t|\mathcal{F}_t] < B < \infty$, $\sum_{t=1}^{\infty} \alpha_t = \infty$ and $\sum_{t=1}^{\infty} \alpha^2_t < C < \infty$ for some $B,C > 0$. Then the process

\begin{equation*}
\begin{array}{l}
Q_{t+1} = (1 - \alpha_t)Q_t + \alpha_t w_t
\end{array}
\end{equation*}

converges to A w. p. 1.

Refer to Lemma 4 in [107] for detailed proof.

Assume that $\alpha_t$ satisfies Assumption 2 and the mapping $P_t: \mathcal{Q} \xrightarrow{} \mathcal{Q}$ satisfies the condition that, there exists a scalar $\gamma$ satisfying $0 \leq \gamma < 1$, a sequence $\lambda_t \geq 0$ converging to zero w. p. 1, and a finite sequence $k_t(s)$ such that $|| P_tQ  - P_t Q_{*} || = \beta ||Q - Q_{*} || + \lambda_t + \beta k_t(s)||Q - Q_{*} ||$ for all $Q$, and all $s \in \mathcal{S}$. Assume further that, $k_t(s)$ converges to a finite point $K(s)$ in the limit. Additionally, $Q_{*}(s, \boldsymbol{a}) = E[P_tQ_{*}(s, \boldsymbol{a}) ]$, then the iteration defined by

\begin{equation*}
\begin{array}{l}
Q_{t+1}(s, \boldsymbol{a}) = (1 - \alpha_t)Q_t(s, \boldsymbol{a})  +\alpha_t[P_t Q_t(s, \boldsymbol{a})]
\end{array}
\end{equation*}

converges to $(Q_{*} - S)$ w. p. 1, where $S$ is as given in Theorem <ref>.

This lemma directly follows from Corollary <ref> and Lemma <ref>.

For a $n$-player stochastic game, $E[P_t Q_{*}] = Q_{*}$ where $Q_* = (Q^1_{*}, \ldots, Q^n_{*}).$

Refer to Lemma 11 in  [38] for proof.

A random iterative process

\begin{equation*}
\begin{array}{l}
\Delta_{t+1}(x) = (1 - \alpha_t(x))\Delta_t(x) + \alpha_t(x) F_t(x)
\end{array}{}
\end{equation*}

where $x \in X$, $t = 0,1, \ldots, \infty$, converges to zero with probability one (w. p. 1) if the following properties hold:

1. The set of possible states $X$ is finite.

2. $0 \leq \alpha_t(x) \leq 1$, $\sum_t \alpha_t(x) = \infty$, $\sum_t \alpha^2_t(x) < \infty$ w. p. 1, where the probability is over the learning rates $\alpha_t$.

3. $|| \E \{{F_t(x)|\mathscr{P}_t}\} ||_W \leq \mathscr{K} ||\Delta_t||_W + c_t$, where $\mathscr{K} \in [0,1)$ and $c_t$ converges to zero w. p. 1.

4. $\textrm{\textbf{var}}\{F_t(x) | \mathscr{P}_t\} \leq K(1 + ||\Delta_t||_W)^2$, where $K$ is some constant.

Here $\mathscr{P}_t$ is an increasing sequence of $\sigma$-fields that includes the past of the process. In particular, we assume that $\alpha_t, \Delta_t, F_{t-1} \in \mathscr{P}_t$. The notation $||\cdot||_W$ refers to some (fixed) weighted maximum norm.

Refer to Theorem 1 in 41 for proof.

Under Assumption <ref>, the Nash operator as defined in Eq. <ref> forms a contraction mapping with the fixed point being the Nash $Q$-value of the game.

The detailed proof is given in Theorem 17 of 38.

§ ADDITIONAL DEFINITIONS FOR THEOREM <REF>

We restate some definitions from [107], needed for us in Theorem <ref>, to stay self-contained.

Let us consider an arbitrary operator, $T: \mathcal{B} \xrightarrow{} \mathcal{B}$, where $\mathcal{B}$ is a normed vector space with norm $||.||$. Let $\mathcal{T} = (T_0, T_1, \cdots, T_t, \cdots)$ be a sequence of random operators, $T_t$ mapping $\mathcal{B} \times \mathcal{B}$ to $\mathcal{B}$.

Let $F \subseteq \mathcal{B} $ be a subset of $\mathcal{B}$ and let $\mathcal{F}_0: F \xrightarrow{} 2^{\mathcal{B}}$ be a mapping that associates subsets of $\mathcal{B}$ with the elements of $F$. If, for all $f \in F$ and all $m_0 \in \mathcal{F}_0(f)$, the sequence generated by the recursion $m_{t+1} = T_t(m_t, f)$ converges to $Tf$ in the norm of $\mathcal{B}$ with probability 1, then we say that $\mathcal{T}$ approximates T for initial values from $\mathcal{F}_0(f)$ and on the set $F \subseteq \mathcal{B}$. Further, we say that $\mathcal{T}$ approximates $T$ on the singleton set ${f}$ and the initial value mapping $\mathcal{F}_0: F \xrightarrow{} B$ defined by $\mathcal{F}_0(f) = F_0$.

The subset $F \subseteq \mathcal{B}$ is invariant under $T: \mathcal{B} \times \mathcal{B} \xrightarrow{} \mathcal{B}$ if, for all $f,g \in F, T(f,g) \in F$. If $\mathcal{T}$ is an operator sequence as above, then $F$ is said to be invariant under $\mathcal{T}$ if for all $i \geq 0$, $F$ is invariant under $T_i$.

§ ADMIRAL-AE USING AN ADAPTIVE ADVISOR

In this sub-section, we aim to provide an illustration of the behaviour of ADMIRAL-AE in the presence of an adaptive advisor. This advisor would actively change and adapt its strategies according to the changing opponent. The objective is to show that the ADMIRAL-AE algorithm will be able to capture the strength of the adaptive advisor, and hence there is merit in using a principled approach to evaluate an advisor. Also, we would like to discuss the advantages of keeping this evaluation method separate from another approach that aims to learn from the advisor. In this section, we clarify that we are only considering the `pre-learning' phase introduced in our paper, since our objective is to analyze the performance of ADMIRAL-AE with two different advisors. Results in this section use the average and standard deviation of 30 runs.

Two instances of ADMIRAL-AE along with an adaptive advisor and a non-adaptive advisor on Pommerman Domain OneVsOne. The plot shows the performance of ADMIRAL-AE using the Advisor Non-Adaptive against the common opponent of DQN (orange line) and the performance of ADMIRAL-AE using the Advisor Adaptive against the common opponent of DQN (blue line). This plot shows that the ADMIRAL-AE using an adaptive advisor starts off week, but eventually surpasses the performance of the non-adaptive advisor. Thus, a principled method like ADMIRAL-AE can evaluate an adaptive advisor appropriately, while other non-principled approaches may have a high percentage of failure.

We will continue to use the two-agent version (Domain OneVsOne) of Pommerman for this experiment. We consider two different advisors, namely, Advisor Non-Adaptive and Advisor Adaptive. The Advisor Non-Adaptive does not actively track opponent strategies or adapt to them. This advisor plays a balanced strategy — choosing to be risk-seeking (actively laying bombs to kill the opponent) while the opponent is in proximity and choosing to be risk-averse (escaping from the enemy) otherwise. This advisor is reactive and only responds to the present position of the opponent, and does not attempt to model the opponent's nature actively. It has been observed that this is a relatively good strategy in Pommerman, particularly in the early stages of training [62].  At this stage, an agent playing a relatively conservative strategy could wait for the opponent to make a mistake and kill itself. However, this strategy is not very strong and could lose out once the opponent is well-trained. A well-trained opponent is less likely to make the mistake of killing itself, and there is the added possibility of the non-adaptive strategy becoming predictable, which could be figured out by the opponent. Hence, the Advisor Non-Adaptive will find it hard to win games in the middle and later stages of training.  On the other hand, the Advisor Adaptive uses an adaptive strategy that plays a risk-averse strategy when the enemy is risk-seeking, and a risk-seeking strategy  when the enemy is risk-averse. This is a very strong strategy for winning in Pommerman [130] but requires active modelling of the opponent. The Advisor Adaptive tracks the percentage of bombs played by the opponent to determine the nature of the opponent. In the initial few episodes (about 5000) the Advisor Adaptive's behaviour is close to random since it still does not have enough information to learn the nature of the opponent.

We implement a separate instance of the ADMIRAL-AE algorithm with both the advisors and plot the performance against a common baseline agent using DQN for learning. We run all the training for 100,000 episodes and plot the performances of both algorithms. The results are captured in Figure <ref>. The results show that ADMIRAL-AE using the Advisor Adaptive loses out in the beginning while it is still figuring out the nature of the opponent. However, it soon shows a much stronger performance that surpasses the performance of the ADMIRAL-AE using the Advisor Non-Adaptive. ADMIRAL-AE using the Advisor Non-Adaptive, while showing good overall performance, does not quite reach the levels of the performance of ADMIRAL-AE using the Advisor Adaptive, due to the non-adaptive strategy possibly becoming predictable and prone to exploitation by the opponent, after the opponent has trained for a sufficient number of episodes. This performance shows that an agent during learning should listen more to the Advisor Adaptive as compared to the Advisor Non-Adaptive for the best outcome. If the advisors were directly used for learning without being evaluated separately, then a learning agent would be prone to discarding Advisor Adaptive quickly due to its initial weak performance, while the agent would listen more to the Advisor Non-Adaptive. Yet we have seen that the opposite behaviour would actually be better using an implementation of ADMIRAL-AE with each of these advisors. This demonstrates that the evaluation would be prone to inaccuracy and inconsistency if it is combined with learning a policy. The analysis in this sub-section shows the advantage of using a principled evaluation algorithm (ADMIRAL-AE) for evaluating an advisor, especially when they have adaptive characteristics.

§ EXPERIMENTAL DETAILS

In this section, we provide the experimental details for all the experiments. We have given detailed information about the advisors and reward functions for all the experiments. Hyperparameters for all algorithms are also provided.

§.§ Grid Maze Domain

The reward function is defined in such a way that the agents get a +1 if any one of two agents reaches the goal, and they get a -1 if any one of the two gets to the pitfall. Both the agents get a +2 if both reach the goal at the same instant, and both the agents get a -2 if they reach a pitfall at the same instant. If one of the two gets to the goal and the other gets to the pitfall, they both still get a +1. In this environment, each agent obtains a local state (observation) which corresponds to the coordinates of the grid cell the agent is currently located. We use the joint observations of the two agents as the state in the environment for determining the $Q$-values in the experiments showing convergence in Section <ref>. This state is available to both agents. The other experiments simply use the observation of the concerned agent in the $Q$-updates, in case of both ADMIRAL-DM and ADMIRAL-AE, to make the challenge harder.

The actions that the agents can take in this game are one of moving up, down, left or right. If the wall obstructs an action, then the agent will remain at the same spot.

All four advisors used are rule-based agents, where Advisor 1 follows high-quality rules at each grid cell, which enables the agent to move to the goal state and avoid the pitfalls. Advisor 2 on the other hand can only suggest the correct actions for the agents to reach the goal and escape the pitfall if the agent is right next to the goal or pitfall (within one step). It is not capable of giving the right action in the other parts of the grid. Thus, Advisor 2 cannot teach perfect coordination to obtain the large positive reward (+2). Advisor 3 can only suggest actions that make the agents move closer to the goal state, but cannot give accurate actions that avoid pitfalls. Advisor 4 is a random advisor which only suggests random actions.

The ADMIRAL-AE implementation for this experiment chooses to do the advisor actions with a probability of 50% ($\eta' = 0.5$). The action that maximizes the $Q$-value (greedy action) is chosen with probability 45%  and a random action with probability 5% ($\eta = 0.05$) to satisfy Assumption <ref>. We use the previous actions of the other agents while determining actions at the current time step $t$ as done in the other algorithms. The ADMIRAL-DM algorithm simply follows the scheme in Algorithm <ref> where the advisor suggestions are performed with decreasing probabilities and the algorithm becomes completely greedy (dependent only on the agent's own policy) after a finite number of episodes.

§.§ Pommerman Domain

All the advisors are based on the rule-based agent (called the simple agent) already provided in the Pommerman framework. It is well documented that the simple agent is hard to beat even for complex RL algorithms in the Pommerman environment [87]. We use four advisors for the ADMIRAL-AE experiments in Section <ref> and <ref>, of which the first advisor alone is used in the experiments that study the performance of ADMIRAL-DM and ADMIRAL-DM(AC) in Section <ref>. The first advisor (Advisor 1) is the best of all the four advisors and has rules for all the different aspects of the game (escaping from enemies, collecting bombs, laying bombs, etc.). The second advisor (Advisor 2) has rules for moving away from danger and collecting powerups like life and bombs. This is a relatively conservative advisor which relies on staying safe and hoping for the opponent to make a mistake. Actually, this is a very effective strategy in the Pommerman game and therefore Advisor 2 is also a useful one. Advisor 3 only has rules for collecting powerups, but cannot teach any of the other strategies needed to win the Pommerman game. When there is no possibility of teaching any useful strategy based on the situation of the game (no nearly enemies, powerups, etc.), the first three advisors provide pseudo-random strategies that encourage exploration of states which are relatively less visited. The fourth advisor (Advisor 4) is the weakest advisor of the four. It suggests random actions to the agent and does not contribute to the learning of the agent (rather, it harms learning).

Each new Pommerman game gives a randomized game map and there is a maximum of 800 steps for each game.

§.§ Pursuit Domain

The Pursuit domain was initially defined in SISL [30]. We use the canonical domain implemented in the petting zoo environment [110]. All the environmental parameters including the rewards are left as the same as the defaults in [30]. For the training phase, the game has 30 evader agents and 8 pursuer agents, where the evader agents move randomly, and the pursuer agents are controlled by learning algorithms. The game is cooperative, with the pursuers receiving a reward of 5 for fully surrounding an evader (the evader is removed from the environment). The pursuers receive a reward of 0.01 for each time they touch an evader. The actions space is discrete with 5 actions, each action corresponding to moving to a neighbouring grid (including stay). Each episode is a full game with a maximum of 500 steps. Each agent in this environment is able to observe a grid of 7 $\times$ 7 centered around itself (the whole grid is 16 $\times$ 16). Since this environment is fully cooperative with homogeneous learning agents, all training (for each algorithm) is completely centralized (all agents train the same network).

§.§ Waterworld Domain

Waterworld is also a SISL environment with a set of 5 pursuers tasked with collecting food and avoiding poison. We use the corresponding petting zoo environment [110]. The environmental parameters are the same as that in [30]. This is also a cooperative environment where multiple pursuers need to work together to capture food. The environment is a two-dimensional continuous space, where the action space is a continuous two-dimensional value representing the horizontal and vertical thrust that makes the agents move in particular directions with the desired speed. The local state (observation) of each agent consists of multiple sensor features and two other elements that indicate the collision of the agent with a food or poison respectively. In this environment, at least two agents need to attack a food together to capture it. There are a total of 5 food particles (not destroyed upon capture) and 10 poison particles in the environment. The agents get a reward of 10.0 for capturing food and a 0.01 for encountering food. Further, the agents have a thrust penalty of -0.5, and a penalty of -1.0 for encountering poison. The poison particles and food particles move in the environment with a speed of 0.01. Since this environment is also fully cooperative with homogenous learning agents, all training is completely centralized for this domain too.

§.§ Hyperparameters And Implementation Details

The hyperparameters for the baselines were chosen to be the same as those recommended by the respective papers. Some minor modifications were made due to performance and computational efficiency considerations.

Regarding the hyperparameters of DQfD, we set $1 \times 10^6$ as the demo buffer size and perform 50,000 mini-batch updates for pretraining. The replay buffer size is twice the size of the demo buffer. The N-step return weight is 1.0, the supervised loss weight is 1.0 and the L2 regularization weight is $10^{-5}$. The epsilon greedy exploration is 0.9. The discount factor is 0.99 and the learning rate is 0.002.
The pretraining for DQfD comes from a data buffer related to a series of games where two rule-based agents (advisors) compete against each other.  All other values are similar to that used in [35].

The CHAT [118] implementation uses a neural network for confidence measurement (termed NNHAT in [118]). The learning rate is 0.01, we use a discount factor of 0.9 and a fixed exploration constant ($\epsilon$-greedy) of 0.9. We use the extra action variant of HAT [109] in the CHAT implementation, as this gave the best performance in most of our comparative experiments. A neural network is used as the function approximator, as described in [65]. The target net is replaced every 10 learning iterations. The confidence threshold is set as 0.6 and the default action as “action-0”. The mini-batch size is 32 and learning rate is $\alpha = 0.01$. The CHAT algorithm can directly use advisors in an online fashion similar to ADMIRAL-DM. We omit the rule summarization step of CHAT, and directly use the advisor, to make the performance as good as possible.  The replay buffer size is $2 \times 10^6$.

The DQN and ADMIRAL-DM hyperparameters are the same as those mentioned for CHAT (as relevant). These algorithms also perform $\epsilon$-greedy exploration with a constant value of 0.9 for $\epsilon$. The advisor influence parameter ($\epsilon'_t$ in Algorithm <ref>) for ADMIRAL-DM and ADMIRAL-DM(AC) starts at a high value of 0.8 at the beginning of the training and linearly decays to 0.01 during training. The face-off and execution phases have this parameter to be 0 (no advisor influence). All other hyperparameters for DQN, ADMIRAL-DM and CHAT are similar to that used in  65. The ADMIRAL-DM(AC) has the critic learning rate set at $10^{-3}$ and the actor learning rate to be $10^{-5}$. The discount factor for ADMIRAL-DM(AC) is 0.9.

The ADMIRAL-AE algorithm in Section <ref> chooses to do the advisor action with a probability of 50%. The action that maximizes the $Q$-value (the greedy action) is chosen with probability 45% and a random action with probability 5% to satisfy Assumption <ref>. This is the same as that in the tabular domain (Section <ref>). The hyperparameters of ADMIRAL-AE are the same as ADMIRAL-DM.

For the function approximation experiments, both ADMIRAL-DM and ADMIRAL-DM(AC) simply use the observed current actions of other agents for action selection instead of maintaining copies of policies of the other agents as specified in Algorithm <ref> and Algorithm <ref>, since the opponents could possibly be using different algorithms (all agents are not always using the same algorithmic steps). This is also computationally more efficient. The current actions of all other agents are either directly observed or provided by the game engine to each agent in all our experiments, to perform a joint action update.

The DDPG uses the learning rate of the actor as 0.001 and the critic as 0.002. The discount factor is 0.9. We use the soft replacement strategy with a learning rate of 0.01. The batch size is 32. The PPO implementation also uses the same batch size and actor and critic learning rates. All other values are similar to those used in [92] (PPO) and [55] (DDPG).
For PPO, we used a single-thread implementation, which we found to be as good as the multi-threaded implementation for our experiments, and more computationally efficient. This could be because the data correlations are already being broken by the multi-agent (non-stationary) nature of the domains.

For the Deep Sarsa implementation used in this paper, we follow almost the same steps as done in Algorithm <ref>, except that we do not have any advisors and hence the algorithm does not have the term $\epsilon'_t$ in the implementation. As a consequence, the greedy action is selected with probability $(1 - \epsilon_t)$. Also, we are using an independent implementation, where the actions of the other agents are not considered during action selection by the algorithms playing Deep Sarsa. The action is chosen only based on the current state. Deep Sarsa uses the same hyperparameters as ADMIRAL-DM. The values are either the same or closely match those considered in previous research [65].

We use a set of 30 random seeds (1–30) for all training experiments and a new set of 30 random seeds (31–60) for all execution experiments.

§.§ Wall Clock Times

The experiments on the Grid Maze domain can just be run on the CPU and takes less than 20 minutes to complete.

The training for all the experiments on the Pommerman domain in Section <ref> and Section <ref> was run on a 2 GPU virtual machine with 16 GB GPU memory per GPU. The experiments take an average of 18 hours wall clock time to complete. We use Nvidia Volta-100 (V100) GPUs for all these experiments. The CPUs use Skylake as the processor microarchitecture.

The experiments on Pursuit and Waterworld domains in Section <ref> were run on a virtual machine having the same configuration containing 2 GPUs. These experiments take an average of 12 hours wall clock time to complete. The experiments on Pommeran in Section <ref> took an average of 15 hours wall clock time to complete.

[1]
Abbeel, P. and Ng, A. Y. (2004).
Apprenticeship Learning via Inverse Reinforcement Learning.
In Proceedings of the Twenty-first International Conference of
Machine Learning (ICML 2004), Banff, Alberta, Canada, July 4-8, 2004,
volume 69, pages 1–9. ACM.

[2]
Amir, O., Kamar, E., Kolobov, A., and Grosz, B. J. (2016).
Interactive Teaching Strategies for Agent Training.
In Proceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence, IJCAI, New York, NY, USA, 9-15 July
2016, pages 804–811. IJCAI/AAAI Press.

[3]
Atkeson, C. G. and Schaal, S. (1997).
Robot Learning From Demonstration.
In Proceedings of the Fourteenth International Conference on
Machine Learning (ICML 1997), Nashville, Tennessee, USA, July 8-12, 1997,
pages 12–20. Morgan Kaufmann.

[4]
Barrett, S., Rosenfeld, A., Kraus, S., and Stone, P. (2017).
Making friends on the fly: Cooperating with new teammates.
Artificial Intelligence, 242:132–171.

[5]
Bignold, A., Cruz, F., Taylor, M. E., Brys, T., Dazeley, R., Vamplew, P., and
Foale, C. (2021).
A Conceptual Framework for Externally-influenced Agents: An
Assisted Reinforcement Learning Review.
Journal of Ambient Intelligence and Humanized Computing.

[6]
Bogert, K. D. and Doshi, P. (2014).
Multi-robot inverse reinforcement learning under occlusion with
In International conference on Autonomous Agents and Multi-Agent
Systems, (AAMAS 2014), Paris, France, May 5-9, 2014, pages 173–180.

[7]
Boularias, A., Kober, J., and Peters, J. (2011).
Relative Entropy Inverse Reinforcement Learning.
In Proceedings of the Fourteenth International Conference on
Artificial Intelligence and Statistics, (AISTATS 2011), Fort Lauderdale,
USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pages
182–189. JMLR.org.

[8]
Bowling, M. H. (2000).
Convergence Problems of General-Sum Multiagent
Reinforcement Learning.
In Proceedings of the Seventeenth International Conference on
Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June
29 - July 2, 2000, pages 89–94. Morgan Kaufmann.

[9]
Carvalho, D., Melo, F. S., and Santos, P. (2020).
A new convergent variant of Q-learning with linear function
In Advances in Neural Information Processing Systems 33: Annual
Conference on Neural Information Processing Systems (NeurIPS 2020), December
6-12, 2020, virtual.

[10]
Cesa-Bianchi, N., Conconi, A., and Gentile, C. (2001).
On the Generalization Ability of On-Line Learning
In Advances in Neural Information Processing Systems, (NeurIPS
2001), Vancouver, British Columbia, Canada, December 3-8, 2001, pages
359–366. MIT Press.

[11]
Chemali, J. and Lazaric, A. (2015).
Direct Policy Iteration with Demonstrations.
In Proceedings of the Twenty-Fourth International Joint
Conference on Artificial Intelligence, (IJCAI 2015), Buenos Aires,
Argentina, July 25-31, 2015, pages 3380–3386. AAAI Press.

[12]
Codevilla, F., Müller, M., López, A. M., Koltun, V., and
Dosovitskiy, A. (2018).
End-to-End Driving via Conditional Imitation Learning.
In International Conference on Robotics and Automation, (ICRA
2018), Brisbane, Australia, May 21-25, 2018, pages 1–9. IEEE.

[13]
Codevilla, F., Santana, E., López, A. M., and Gaidon, A. (2019).
Exploring the Limitations of Behavior Cloning for Autonomous
In International Conference on Computer Vision, (ICCV 2019)
Seoul, Korea (South), October 27 - November 2, 2019, pages 9328–9337.

[14]
Conitzer, V. and Sandholm, T. (2003).
AWESOME: A General Multiagent Learning Algorithm that
Converges in Self-Play and Learns a Best Response Against
Stationary Opponents.
In Proceedings of the Twentieth International Conference of
Machine Learning, (ICML 2003), August 21-24, 2003, Washington, DC, USA,
pages 83–90. AAAI Press.

[15]
Cruz, M. and Alexander, M. (2010).
Assessing Crown Fire Potential in Coniferous Forests of
Western North America: A Critique of Current Approaches and
Recent Simulation Studies.
The Bark Beetles, Fuels, and Fire Bibliography, 19.

[16]
Devlin, S., Kudenko, D., and Grześ, M. (2011).
An empirical study of potential-based reward shaping and advice in
complex, multi-agent systems.
Advances in Complex Systems, 14(02):251–278.

[17]
Dietterich, T. G. (2000).
Ensemble Methods in Machine Learning.
In Multiple Classifier Systems, First International Workshop,
(MCS 2000), Cagliari, Italy, June 21-23, 2000, Proceedings, volume 1857 of
Lecture Notes in Computer Science, pages 1–15. Springer.

[18]
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal,
S., and Hester, T. (2021).
Challenges of real-world reinforcement learning: Definitions,
Benchmarks and Analysis.
Machine Learning, 1:1–50.

[19]
Eryigit, C. (2017).
Marketing Models: A Review of the Literature.
International Journal of Market Research, 59(3):355–381.

[20]
Fink, A. M. et al. (1964).
Equilibrium in a stochastic $ n $-person game.
Journal of Science of the Hiroshima University, Series AI
(Mathematics), 28(1):89–93.

[21]
Finn, C., Christiano, P., Abbeel, P., and Levine, S. (2016).
A Connection between Generative Adversarial Networks,
Inverse Reinforcement Learning, and Energy-Based Models.
arXiv preprint arXiv:1611.03852.

[22]
Finney, M. A. (1998).
FARSITE, Fire Area Simulator–model development and
4. US Department of Agriculture, Forest Service, Rocky Mountain
Research Station.

[23]
Fu, J., Luo, K., and Levine, S. (2018).
Learning Robust Rewards with Adverserial Inverse
Reinforcement Learning.
In 6th International Conference on Learning Representations,
ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018. OpenReview.net.

[24]
Ganesh, S., Vadori, N., Xu, M., Zheng, H., Reddy, P., and Veloso, M. (2019).
Reinforcement learning for market making in a multi-agent dealer
arXiv preprint arXiv:1911.05892.

[25]
Gangwani, T., Zhou, Y., and Peng, J. (2020).
Learning Guidance Rewards with Trajectory-space Smoothing.
In Advances in Neural Information Processing Systems (NeurIPS
2020), December 6-12, 2020, virtual.

[26]
Gao, C., Kartal, B., Hernandez-Leal, P., and Taylor, M. E. (2019).
On Hard Exploration for Reinforcement Learning: A Case
Study in Pommerman.
In Proceedings of the Fifteenth AAAI Conference on Artificial
Intelligence and Interactive Digital Entertainment, (AIIDE 2019), October
8-12, 2019, Atlanta, Georgia, USA, pages 24–30. AAAI Press.

[27]
Gao, Y., Xu, H., Lin, J., Yu, F., Levine, S., and Darrell, T. (2018).
Reinforcement Learning from Imperfect Demonstrations.
In 6th International Conference on Learning Representations,
(ICLR 2018), Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track
Proceedings. OpenReview.net.

[28]
Giusti, A., Guzzi, J., Cireşan, D. C., He, F.-L., Rodríguez, J. P.,
Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Di Caro, G., et al.
A machine learning approach to visual perception of forest trails for
mobile robots.
IEEE Robotics and Automation Letters, 1(2):661–667.

[29]
Goecks, V. G., Gremillion, G. M., Lawhern, V. J., Valasek, J., and Waytowich,
N. R. (2020).
Integrating Behavior Cloning and Reinforcement Learning for
Improved Performance in Dense and Sparse Reward Environments.
In Proceedings of the 19th International Conference on
Autonomous Agents and Multiagent Systems, (AAMAS 2020), Auckland, New
Zealand, May 9-13, 2020, pages 465–473. IFAAMAS.

[30]
Gupta, J. K., Egorov, M., and Kochenderfer, M. J. (2017).
Cooperative Multi-agent Control Using Deep Reinforcement
In Autonomous Agents and Multiagent Systems - (AAMAS 2017)
Workshops, São Paulo, Brazil, May 8-12, 2017, volume 10642 of Lecture Notes in Computer Science, pages 66–83. Springer.

[31]
Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. D. (2016).
Cooperative Inverse Reinforcement Learning.
In Advances in Neural Information Processing Systems (NeurIPS
2016), Barcelona, Spain, December 5-10, 2016, pages 3909–3917.

[32]
Hazan, E., Agarwal, A., and Kale, S. (2007).
Logarithmic regret algorithms for online convex optimization.
Machine Learning, 69(2-3):169–192.

[33]
He, H. and Boyd-Graber, J. L. (2016).
Opponent Modeling in Deep Reinforcement Learning.
In Proceedings of the 33nd International Conference on Machine
Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48
of JMLR Workshop and Conference Proceedings, pages 1804–1813.

[34]
Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2019).
A survey and critique of multiagent deep reinforcement learning.
Journal of Autonomous Agents and Multi-Agent Systems (JAAMAS),

[35]
Hester, T., Vecerík, M., Pietquin, O., Lanctot, M., Schaul, T., Piot,
B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G.,
Agapiou, J. P., Leibo, J. Z., and Gruslys, A. (2018).
Deep Q-learning From Demonstrations.
In Thirty-Second Conference of Association for the Advancement
of Artificial Intelligence (AAAI-18), New Orleans, Louisiana, USA, February
2-7, 2018, pages 3223–3230. AAAI Press.

[36]
Ho, J. and Ermon, S. (2016).
Generative Adversarial Imitation Learning.
In Advances in Neural Information Processing Systems (NeurIPS
2016), Barcelona, Spain, December 5-10, 2016, pages 4565–4573.

[37]
Hong, Z., Su, S., Shann, T., Chang, Y., and Lee, C. (2018).
A Deep Policy Inference Q-Network for Multi-Agent
In Proceedings of the 17th International Conference on
Autonomous Agents and MultiAgent Systems, AAMAS 2018 Stockholm, Sweden,
July 10-15, 2018, pages 1388–1396. IFAAMAS.

[38]
Hu, J. and Wellman, M. P. (2003).
Nash Q-learning for general-sum stochastic games.
Journal of Machine Learning Research (JMLR), 4(Nov):1039–1069.

[39]
Hu, Y., Li, J., Li, X., Pan, G., and Xu, M. (2018).
Knowledge-Guided Agent-Tactic-Aware Learning for
Starcraft Micromanagement.
In Proceedings of the Twenty-Seventh International Joint
Conference on Artificial Intelligence, (IJCAI 2018), July 13-19, 2018,
Stockholm, Sweden, pages 1471–1477. ijcai.org.

[40]
Hussein, M., Crowe, B., Petrik, M., and Begum, M. (2021).
Robust Maximum Entropy Behavior Cloning.
arXiv preprint arXiv:2101.01251.

[41]
Jaakkola, T., Jordan, M. I., and Singh, S. P. (1994).
On the Convergence of Stochastic Iterative Dynamic
Programming Algorithms.
Neural Computation, 6(6):1185–1201.

[42]
Jahdi, R., Salis, M., Darvishsefat, A. A., Alcasena, F., Mostafavi, M. A.,
Etemad, V., Lozano, O. M., and Spano, D. (2015).
Evaluating fire modelling systems in recent wildfires of the
Golestan National Park, Iran.
Forestry, 89(2):136–149.

[43]
Jain, P., Coogan, S. C., Subramanian, S. G., Crowley, M., Taylor, S., and
Flannigan, M. D. (2020).
A review of machine learning applications in wildfire science and
Environmental Reviews, 28(4):478–505.

[44]
Jing, M., Ma, X., Huang, W., Sun, F., Yang, C., Fang, B., and Liu, H. (2020).
Reinforcement Learning from Imperfect Demonstrations under
Soft Expert Guidance.
In The Thirty-Fourth Conference of Association for the
Advancement of Artificial Intelligence (AAAI 2020), New York, NY, USA,
February 7-12, 2020, pages 5109–5116. AAAI Press.

[45]
Kakade, S. M. (2003).
On the sample complexity of reinforcement learning.
PhD thesis, UCL (University College London).

[46]
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A. A., Yogamani,
S., and Pérez, P. (2021).
Deep reinforcement learning for autonomous driving: A survey.
IEEE Transactions on Intelligent Transportation Systems.

[47]
Kuefler, A., Morton, J., Wheeler, T., and Kochenderfer, M. (2017).
Imitating Driver Behavior with Generative Adversarial
In IEEE Intelligent Vehicles Symposium (IV 2017), pages
204–211. IEEE.

[48]
Laskey, M., Lee, J., Fox, R., Dragan, A. D., and Goldberg, K. (2017).
DART: Noise Injection for Robust Imitation Learning.
In 1st Annual Conference on Robot Learning, (CoRL 2017),
Mountain View, California, USA, 2017, volume 78 of Proceedings of
Machine Learning Research, pages 143–156. PMLR.

[49]
Lasry, J.-M. and Lions, P.-L. (2007).
Mean field games.
Japanese journal of mathematics, 2(1):229–260.

[50]
Laud, A. D. (2004).
Theory and application of reward shaping in reinforcement
University of Illinois at Urbana-Champaign.

[51]
Le, H. M., Yue, Y., Carr, P., and Lucey, P. (2017).
Coordinated Multi-Agent Imitation Learning.
In Proceedings of the 34th International Conference on Machine
Learning, (ICML 2017), Sydney, NSW, Australia, 6-11 August 2017, volume 70
of Proceedings of Machine Learning Research, pages 1995–2003. PMLR.

[52]
Leblon, B., Bourgeau-Chavez, L., and San-Miguel-Ayanz, J. (2012).
Use of remote sensing in wildfire management.
In Sustainable Development-Authoritative and Leading Edge
Content for Environmental Management. IntechOpen.

[53]
Levine, S., Popovic, Z., and Koltun, V. (2011).
Nonlinear Inverse Reinforcement Learning with Gaussian
In Advances in Neural Information Processing Systems (NeurIPS
2011) Granada, Spain, December 2011, pages 19–27.

[54]
Li, M., Wei, Y., and Kudenko, D. (2019).
Two-level Q-learning: learning from conflict demonstrations.
The Knowledge Engineering Review, 34.

[55]
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y.,
Silver, D., and Wierstra, D. (2016).
Continuous control with deep reinforcement learning.
In 4th International Conference on Learning Representations,
(ICLR 2016), San Juan, Puerto Rico, May 2-4, 2016, Conference Track

[56]
Lin, X., Adams, S. C., and Beling, P. A. (2019).
Multi-agent inverse reinforcement learning for certain general-sum
stochastic games.
Journal of Artificial Intelligence Research (JAIR),

[57]
Lin, X., Beling, P. A., and Cogill, R. (2017).
Multiagent inverse reinforcement learning for two-person zero-sum
IEEE Transactions on Games, 10(1):56–68.

[58]
Littman, M. and Moore, A. (1996).
Reinforcement Learning: A Survey.
Journal of Artificial Intelligence Research (JAIR), 4:237–285.

[59]
Littman, M. L. (2001).
Friend-or-Foe Q-learning in General-Sum Games.
In Proceedings of the Eighteenth International Conference on
Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June
28 - July 1, 2001, pages 322–328. Morgan Kaufmann.

[60]
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017).
Multi-Agent Actor-Critic for Mixed
Cooperative-Competitive Environments.
In Advances in Neural Information Processing Systems (NeurIPS
2017), Long Beach, CA, USA, December 4-9, 2017, pages 6379–6390.

[61]
Marom, O. and Rosman, B. (2018).
Belief Reward Shaping in Reinforcement Learning.
In Proceedings of the Thirty-Second AAAI Conference on
Artificial Intelligence, (AAAI-18), New Orleans, Louisiana, USA, February
2-7, 2018, pages 3762–3769. AAAI Press.

[62]
Meisheri, H., Shelke, O., Verma, R., and Khadilkar, H. (2019).
Accelerating training in Pommerman with imitation and reinforcement
arXiv preprint arXiv:1911.04947.

[63]
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018).
Spectral Normalization for Generative Adversarial Networks.
In 6th International Conference on Learning Representations,
(ICLR 2018), Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track
Proceedings. OpenReview.net.

[64]
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T.,
Silver, D., and Kavukcuoglu, K. (2016).
Asynchronous Methods for Deep Reinforcement Learning.
In Proceedings of the Thirty-Third International Conference on
Machine Learning, (ICML 2016), New York City, NY, USA, June 19-24, 2016,
volume 48 of JMLR Workshop and Conference Proceedings, pages
1928–1937. JMLR.org.

[65]
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare,
M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.
Human-level control through deep reinforcement learning.
Nature, 518(7540):529.

[66]
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018).
Overcoming Exploration in Reinforcement Learning with
In 2018 IEEE International Conference on Robotics and
Automation, (ICRA 2018) Brisbane, Australia, May 21-25, 2018, pages
6292–6299. IEEE.

[67]
Nash, J. F. (1951).
Non-Cooperative Games.
Princeton University Press.

[68]
Natarajan, S., Kunapuli, G., Judah, K., Tadepalli, P., Kersting, K., and
Shavlik, J. W. (2010).
Multi-Agent Inverse Reinforcement Learning.
In The Ninth International Conference on Machine Learning and
Applications, (ICMLA 2010), Washington, DC, USA, 12-14 December 2010,
pages 395–400. IEEE Computer Society.

[69]
Neumann, J. v. (1928).
Zur theorie der gesellschaftsspiele.
Mathematische annalen, 100(1):295–320.

[70]
Ng, A. Y., Harada, D., and Russell, S. J. (1999).
Policy Invariance Under Reward Transformations: Theory and
Application to Reward Shaping.
In Proceedings of the Sixteenth International Conference on
Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999, pages
278–287. Morgan Kaufmann.

[71]
Ng, A. Y. and Russell, S. J. (2000).
Algorithms for Inverse Reinforcement Learning.
In Proceedings of the Seventeenth International Conference on
Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June
29 - July 2, 2000, pages 663–670. Morgan Kaufmann.

[72]
Nikitin, V., Golubin, S., Belov, R., Gusev, V., and Andrianov, N. (2019).
Development of a robotic vehicle complex for wildfire-fighting by
means of fire-protection roll screens.
In IOP Conference Series: Earth and Environmental Science,
volume 226, page 012003. IOP Publishing.

[73]
Omidshafiei, S., Kim, D., Liu, M., Tesauro, G., Riemer, M., Amato, C.,
Campbell, M., and How, J. P. (2019).
Learning to Teach in Cooperative Multiagent Reinforcement
In Thirty-Third Conference of Association for the Advancement of
Artificial Intelligence (AAAI-19), Honolulu, Hawaii, USA, January 27 -
February 1, 2019, pages 6128–6136. AAAI Press.

[74]
Peng, P., Xing, J., and Cao, L. (2020).
Hybrid Learning for Multi-agent Cooperation with Sub-optimal
In Proceedings of the Twenty-Ninth International Joint
Conference on Artificial Intelligence, (IJCAI-2020), pages 3037–3043.
International Joint Conferences on Artificial Intelligence Organization.
Main track.

[75]
Peters, J. and Schaal, S. (2008).
Reinforcement learning of motor skills with policy gradients.
Neural networks, 21(4):682–697.

[76]
Phan, C. and Liu, H. H. (2008).
A cooperative UAV/UGV platform for wildfire detection and fighting.
In Asia Simulation Conference-7th International Conference on
System Simulation and Scientific Computing, pages 494–498. IEEE.

[77]
Piot, B., Geist, M., and Pietquin, O. (2014).
Boosted Bellman Residual Minimization Handling Expert
In Machine Learning and Knowledge Discovery in Databases -
European Conference, (ECML-PKDD 2014), Nancy, France, September 15-19,
2014. Proceedings, Part II, volume 8725 of Lecture Notes in Computer
Science, pages 549–564. Springer.

[78]
Pomerleau, D. (1988).
ALVINN: An Autonomous Land Vehicle in a Neural Network.
In Advances in Neural Information Processing Systems (NeurIPS
1988) Denver, Colorado, USA, 1988, pages 305–313. Morgan Kaufmann.

[79]
Pomerleau, D. A. (1991).
Efficient training of artificial neural networks for autonomous
Neural Computation, 3(1):88–97.

[80]
Price, B. and Boutilier, C. (2003).
Accelerating Reinforcement Learning through Implicit
Journal of Artificial Intelligence Research, 19:569–629.

[81]
Raileanu, R., Denton, E., Szlam, A., and Fergus, R. (2018).
Modeling Others using Oneself in Multi-Agent Reinforcement
In Proceedings of the 35th International Conference on Machine
Learning ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15,
2018, volume 80 of Proceedings of Machine Learning Research, pages
4254–4263. PMLR.

[82]
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E.,
and Levine, S. (2018).
Learning Complex Dexterous Manipulation with Deep
Reinforcement Learning and Demonstrations.
In Robotics: Science and Systems XIV, Carnegie Mellon
University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018.

[83]
Rasmussen, C. E. (2003).
Gaussian Processes in Machine Learning.
In Advanced Lectures on Machine Learning, ML Summer Schools
2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany,
August 4-16, 2003, Revised Lectures, volume 3176 of Lecture Notes in
Computer Science, pages 63–71. Springer.

[84]
Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. (2006a).
Maximum Margin Planning.
In Proceedings of the Twenty-Third International Conference on
Machine Learning (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29,
2006, volume 148 of ACM International Conference Proceeding Series,
pages 729–736. ACM.

[85]
Ratliff, N. D., Bradley, D. M., Bagnell, J. A., and Chestnutt, J. E. (2006b).
Boosting Structured Prediction for Imitation Learning.
In Advances in Neural Information Processing Systems (NeurIPS
2006) Vancouver, British Columbia, Canada, December 4-7, 2006, pages
1153–1160. MIT Press.

[86]
Reddy, T. S., Gopikrishna, V., Záruba, G. V., and Huber, M. (2012).
Inverse reinforcement learning for decentralized non-cooperative
multiagent systems.
In Proceedings of the IEEE International Conference on
Systems, Man, and Cybernetics (SMC 2012), Seoul, Korea (South), October
14-17, 2012, pages 1930–1935. IEEE.

[87]
Resnick, C., Eldridge, W., Ha, D., Britz, D., Foerster, J., Togelius, J., Cho,
K., and Bruna, J. (2018).
Pommerman: A Multi-Agent Playground.
In arXiv preprint arXiv:1809.07124.

[88]
Ross, S., Gordon, G. J., and Bagnell, D. (2011).
A Reduction of Imitation Learning and Structured Prediction
to No-Regret Online Learning.
In Proceedings of the Fourteenth International Conference on
Artificial Intelligence and Statistics (AISTATS 2011), Fort Lauderdale,
USA, April 11-13, 2011, volume 15 of JMLR Proceedings, pages
627–635. JMLR.org.

[89]
Rothermel, R. C. (1972).
A mathematical model for predicting fire spread in wildland
fuels, volume 115.
Intermountain Forest & Range Experiment Station, Forest Service, US.

[90]
Santana, E. and Hotz, G. (2016).
Learning a driving simulator.
arXiv preprint arXiv:1608.01230.

[91]
Schaal, S. (1996).
Learning from Demonstration.
In Advances in Neural Information Processing Systems (NeurIPS
1996), Denver, CO, USA, December 2-5, 1996, pages 1040–1046. MIT Press.

[92]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).
Proximal policy optimization algorithms.
In arXiv preprint arXiv:1707.06347.

[93]
Shalev-Shwartz, S. and Kakade, S. M. (2008).
Mind the Duality Gap: Logarithmic regret algorithms for online
In Advances in Neural Information Processing Systems (NeurIPS
2008), Vancouver, British Columbia, Canada, December 8-11, 2008, pages
1457–1464. Curran Associates, Inc.

[94]
Shapley, L. S. (1953).
Stochastic games.
Proceedings of the national academy of sciences,

[95]
Shoham, Y. and Leyton-Brown, K. (2008).
Multiagent systems: Algorithmic, game-theoretic, and logical
Cambridge University Press.

[96]
Silva, F. L. D. and Costa, A. H. R. (2019).
A Survey on Transfer Learning for Multiagent Reinforcement
Learning Systems.
Journal of Artificial Intelligence Research (JAIR),

[97]
Silva, F. L. D., Glatt, R., and Costa, A. H. R. (2017).
Simultaneously Learning and Advising in Multiagent
Reinforcement Learning.
In Proceedings of the 16th Conference on Autonomous Agents and
MultiAgent Systems, (AAMAS 2017), São Paulo, Brazil, May 8-12, 2017,
pages 1100–1108. ACM.

[98]
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche,
G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M.,
et al. (2016).
Mastering the game of Go with deep neural networks and tree search.
Nature, 529(7587):484–489.

[99]
Singh, S., Jaakkola, T., Littman, M. L., and Szepesvári, C. (2000).
Convergence results for single-step on-policy reinforcement-learning
Machine Learning, 38(3):287–308.

[100]
Sonabend, A., Lu, J., Celi, L. A., Cai, T., and Szolovits, P. (2020).
Expert-Supervised Reinforcement Learning for Offline Policy
Learning and Evaluation.
In Advances in Neural Information Processing Systems (NeurIPS
2020), virtual, December 6-12, 2020.

[101]
Song, J., Ren, H., Sadigh, D., and Ermon, S. (2018).
Multi-Agent Generative Adversarial Imitation Learning.
In Advances in Neural Information Processing Systems (NeurIPS
2018), Montreal, Canada, December 3-8, 2018,, pages 7472–7483.

[102]
Storbacka, K. and Moser, T. (2020).
The changing role of marketing: transformed propositions, processes
and partnerships.
AMS Review, 10(3):299–310.

[103]
Subramanian, S. G. (2022).
Multi-Agent Advisor Q-Learning.

[104]
Sutton, R. S. and Barto, A. G. (1998).
Introduction to Reinforcement Learning, volume 135.
MIT Press Cambridge.

[105]
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y. (1999).
Policy Gradient Methods for Reinforcement Learning with
Function Approximation.
In Advances in Neural Information Processing Systems 12,
[NuerIPS Conference, Denver, Colorado, USA, November 29 - December 4,
1999], pages 1057–1063. The MIT Press.

[106]
Syed, U. and Schapire, R. E. (2010).
A Reduction from Apprenticeship Learning to Classification.
In Advances in Neural Information Processing Systems (NeurIPS
2010) Vancouver, British Columbia, Canada, December 6-9, 2010, pages
2253–2261. Curran Associates, Inc.

[107]
Szepesvári, C. and Littman, M. L. (1999).
A unified analysis of value-function-based reinforcement-learning
Neural Computation, 11(8):2017–2060.

[108]
Tan, M. (1997).
Multi-agent reinforcement learning: Independent vs Cooperative
Readings in Agents, pages 487–494.

[109]
Taylor, M. E., Suay, H. B., and Chernova, S. (2011).
Integrating reinforcement learning with human demonstrations of
varying ability.
In International Conference on Autonomous Agents and Multiagent
Systems (AAMAS 2011), Taipei, Taiwan, May 2-6, 2011, pages 617–624.

[110]
Terry, J. K., Black, B., Jayakumar, M., Hari, A., Santos, L., Dieffendahl, C.,
Williams, N. L., Lokesh, Y., Sullivan, R., Horsch, C., and Ravi, P. (2020).
PettingZoo: Gym for Multi-Agent Reinforcement Learning.
In arXiv preprint arXiv:2009.14471.

[111]
Theodorou, E. A., Buchli, J., and Schaal, S. (2010).
Reinforcement learning of motor skills in high dimensions: A path
integral approach.
In International Conference on Robotics and Automation, (ICRA
2010), Anchorage, Alaska, USA, 3-7 May 2010, pages 2397–2403. IEEE.

[112]
Thompson, M. P., Lauer, C. J., Calkin, D. E., Rieck, J. D., Stonesifer, C. S.,
and Hand, M. S. (2018).
Wildfire Response Performance Measurement: Current and
Future Directions.
Fire, 1(2).

[113]
Thompson, M. P., Wei, Y., Calkin, D. E., O’Connor, C. D., Dunn, C. J.,
Anderson, N. M., and Hogland, J. S. (2019).
Risk management and analytics in wildfire response.
Current Forestry Reports, 5(4):226–239.

[114]
Torrey, L. and Taylor, M. E. (2013).
Teaching on a budget: agents advising agents in reinforcement
In International conference on Autonomous Agents and
Multi-Agent Systems, AAMAS '13, Saint Paul, MN, USA, May 6-10, 2013,
pages 1053–1060. IFAAMAS.

[115]
Tymstra, C., Stocks, B. J., Cai, X., and Flannigan, M. D. (2020).
Wildfire management in Canada: Review, challenges and
Progress in Disaster Science, 5:100045.

[116]
Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess,
N., Rothörl, T., Lampe, T., and Riedmiller, M. (2017).
Leveraging Demonstrations for Deep Reinforcement Learning on
Robotics Problems with Sparse Rewards.
arXiv preprint arXiv:1707.08817.

[117]
Wang, X. and Klabjan, D. (2018).
Competitive Multi-agent Inverse Reinforcement Learning with
Sub-optimal Demonstrations.
In Proceedings of the Thirty-fifth International Conference on
Machine Learning (ICML 2018), Stockholm, Sweden, July 10-15, 2018, volume 80
of Proceedings of Machine Learning Research, pages 5130–5138. PMLR.

[118]
Wang, Z. and Taylor, M. E. (2017).
Improving Reinforcement Learning with Confidence-Based
In Proceedings of the Twenty-Sixth International Joint
Conference on Artificial Intelligence (IJCAI 2017), Melbourne, Australia,
August 19-25, 2017, pages 3027–3033. ijcai.org.

[119]
Watkins, C. J. and Dayan, P. (1992).
Machine Learning, 8(3-4):279–292.

[120]
Waugh, K., Ziebart, B. D., and Bagnell, D. (2011).
Computational Rationalization: The Inverse Equilibrium
In Proceedings of the Twenty-Eighth International Conference on
Machine Learning, (ICML 2011), Bellevue, Washington, USA, June 28 - July 2,
2011, pages 1169–1176. Omnipress.

[121]
Wiewiora, E., Cottrell, G. W., and Elkan, C. (2003).
Principled Methods for Advising Reinforcement Learning
In Proceedings of the Twentieth International Conference of
Machine Learning (ICML 2003), August 21-24, 2003, Washington, DC, USA,
pages 792–799. AAAI Press.

[122]
Wulfmeier, M., Ondruska, P., and Posner, I. (2015).
Maximum Entropy Deep Inverse Reinforcement Learning.
arXiv preprint arXiv:1507.04888.

[123]
Xiao, B., Ramasubramanian, B., and Poovendran, R. (2021).
Shaping Advice in Deep Multi-Agent Reinforcement
CoRR, abs/2103.15941.

[124]
Ye, D., Zhu, T., Cheng, Z., Zhou, W., and Yu, P. S. (2020).
Differential Advising in Multi-Agent Reinforcement
CoRR, abs/2011.03640.

[125]
Yogeswaran, M. and Ponnambalam, S. (2012).
Reinforcement learning: Exploration–exploitation dilemma in
multi-agent foraging task.
Opsearch, 49(3):223–236.

[126]
Yu, L., Song, J., and Ermon, S. (2019).
Multi-Agent Adversarial Inverse Reinforcement Learning.
In Proceedings of the Thirty-Sixth International Conference on
Machine Learning, (ICML 2019), Long Beach, California, USA, 9-15 June
2019, volume 97 of Proceedings of Machine Learning Research, pages
7194–7201. PMLR.

[127]
Zhan, Y., Bou-Ammar, H., and Taylor, M. E. (2016).
Theoretically-Grounded Policy Advice from Multiple Teachers
in Reinforcement Learning Settings with Applications to Negative
In Proceedings of the Twenty-Fifth International Joint
Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15
July 2016, pages 2315–2321. IJCAI/AAAI Press.

[128]
Zhang, K., Yang, Z., and Bashar, T. (2019).
Multi-agent reinforcement learning: A selective overview of theories
and algorithms.
arXiv preprint arXiv:1911.10635, 1.

[129]
Zheng, J., Liu, S., and Ni, L. M. (2014).
Robust Bayesian Inverse Reinforcement Learning with Sparse
Behavior Noise.
In Proceedings of the Twenty-Eighth Conference of Association of
Artificial Intelligence (AAAI-2014), Quebec City, Quebec, Canada, July 27
-31, 2014, pages 2198–2205. AAAI Press.

[130]
Zhou, H., Gong, Y., Mugrai, L., Khalifa, A., Nealen, A., and Togelius, J.
A hybrid search agent in Pommerman.
In Proceedings of the 13th International Conference on the
Foundations of Digital Games (FDG 2018), Malmo, Sweden, August 07-10,
2018, pages 46:1–46:4. ACM.

[131]
Zhou, M., Luo, J., Villela, J., Yang, Y., Rusu, D., Miao, J., Zhang, W., Alban,
M., Fadakar, I., Chen, Z., Huang, A. C., Wen, Y., Hassanzadeh, K., Graves,
D., Chen, D., Zhu, Z., Nguyen, N. M., Elsayed, M., Shao, K., Ahilan, S.,
Zhang, B., Wu, J., Fu, Z., Rezaee, K., Yadmellat, P., Rohani, M., Nieves,
N. P., Ni, Y., Banijamali, S., Cowen-Rivers, A. I., Tian, Z., Palenicek,
D., Bou-Ammar, H., Zhang, H., Liu, W., Hao, J., and Wang, J. (2020).
SMARTS: Scalable Multi-agent Reinforcement Learning
Training School for Autonomous Driving.
CoRR, abs/2010.09776.

[132]
Zhu, Y., Wang, Z., Merel, J., Rusu, A. A., Erez, T., Cabi, S., Tunyasuvunakool,
S., Kramár, J., Hadsell, R., de Freitas, N., and Heess, N. (2018).
Reinforcement and Imitation Learning for Diverse Visuomotor
In Robotics: Science and Systems XIV, Carnegie Mellon
University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018.

[133]
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. (2008).
Maximum Entropy Inverse Reinforcement Learning.
In Proceedings of the Twenty-Third Conference of Association for
the Advancement of Artificial Intelligence (AAAI-2008), Chicago, Illinois,
USA, July 13-17, 2008, pages 1433–1438. AAAI Press.

[134]
Zigner, K., Carvalho, L., Peterson, S., Fujioka, F., Duine, G.-J., Jones, C.,
Roberts, D., and Moritz, M. (2020).
Evaluating the ability of FARSITE to simulate wildfires influenced
by extreme, downslope winds in Santa Barbara, California.
Fire, 3(3):29.