Sarsa
Sarsa
**Sarsa** is an on-policy TD control algorithm:

$$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} + \gamma{Q}\left(S\_{t+1}, A\_{t+1}\right) - Q\left(S\_{t}, A\_{t}\right)\right] $$

This update is done after every transition from a nonterminal state $S\_{t}$. if $S\_{t+1}$ is terminal, then $Q\left(S\_{t+1}, A\_{t+1}\right)$ is defined as zero.

To design an on-policy control algorithm using Sarsa, we estimate $q\_{\pi}$ for a behaviour policy $\pi$ and then change $\pi$ towards greediness with respect to $q\_{\pi}$.

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition