Add policy search
]
},
{
- "cell_type": "code",
- "execution_count": null,
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
- "source": []
+ "source": [
+ "## Policy Search"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The final approach we'll consider here is the policy search. Policy-search methods operate directly on the representation of the policy, attemptimg to improve it based on the observed performance. As we know that, policy $\\pi$ is a function that maps states to actions, we are interested primarily in parameterized representation of $\\pi$ that have far fewer parameters than there are states in the state space. For example we could represent $\\pi$ by a collection of parameterized Q-functions, one for each action, and take the action with the highest predicted value.\n",
+ "\n",
+ "$\\pi(s) = max_a\\hat{Q}_\\theta(s,a)$\n",
+ "\n",
+ "Policy search adjusts the parameter $\\theta$ to improve the policy. But one problem with policy representation of this kind is that the policy is a discontinuous function of parameters when the actions are discrete i.e. there will be values of $\\theta$ such that an infinitesimal change in $\\theta$ causes the policy to switch from one action to another. This means that the value of the policy may also change discontinuously, which makes gradient-based search difficult.\n",
+ "\n",
+ "For this reason, policy search methods often use a **stochastic policy** representation $\\pi_{\\theta}(s,a)$, which specifies the probability of selecting action $a$ in state $s$. One popular representation is the **softmax function**:\n",
+ "\n",
+ "$\\pi_{\\theta}(s,a) = e^{\\hat{Q}_\\theta(s,a)}/\\sum _{a'}e^{\\hat{Q}_\\theta(s,a')}$\n",
+ "\n",
+ "Softmax becomes nearly deterministic if one action is much better than the others, but it always gives a differentiable function of $\\theta$, hence, the value of policy is always a differentiable function of $\\theta$. "
+ ]
}
],
"metadata": {
