Introduction to Reinforcement Learning.

4. Summary of the experimental part

To analyze the experimental part, we need to start by defining the following concepts:

• Policy: term used to refer to the actions that the agent will decide. The politics $\varepsilon$ -greedy means that the agent will almost always take the best possible action given the information it has.
• Exploration vs exploitation: From time to time, with a probability of $\varepsilon$ , the agent will take a completely random action. In this way, if after the first action the agent has If you get a positive reward, you won't be stuck choosing that same action all the time. with probability $\varepsilon$ the agent will explore other options. This value is parameterizable and will be responsible for balancing the exploration and exploitation

In a complete reinforcement learning problem, the state changes every time we execute an action. The agent receives the state (state) in which the environment is located (environment), which we will represent with the letter s (state). The agent then executes the action it chooses, represented by the letter a (action). By executing that action, The environment responds by providing a reward, represented by the letter r (reward), and the environment moves to a new state, represented with s' (next state). This cycle can be seen in the image: [1] .

The Q-Learning algorithm, used in the article, tries to learn how many rewards it will get in the long term for each pair of states and actions (s,a). We call that function the action-value function. and this algorithm represents it as the function Q(s,a), which returns the reward that the agent will receive when execute action a from state s, and assuming that it will follow the same policy dictated by the function Q until end of the episode. For example, if Q(s,a1)=1 and Q(s,a2)=4, the agent knows that action a2 is better and will bring more reward, so it will be the action that will be executed.

The formal definition of this algorithm is done as follows: $Q(s, a; \theta) = r + \gamma \max_{a'}{Q(s', a'; \theta')}$ The Q-value of the state s and the action a (Q(s, a)) is defined as the reward r obtained by executing that action, plus the Q-value of executing the best possible action a' from the next state s', multiplied by a discount factor $\gamma$ (discount factor), which is a value with a range $\gamma$$\in$ (0, 1].

Now, when there are billions of different states and hundreds of different actions, Q-Learning is not able to be used optimally. Therefore, in the work [1] define a new technique called Deep Q-Network. This algorithm combines Q-learning with deep neural networks to approximate the function Q, thus avoiding using a table to represent it. It actually uses two networks neurons to stabilize the learning process. The first, the main neural network, represented by the parameters $\theta$ , is used to estimate the Q-values of the current state s and action a: Q(s, a; $\theta$ ). The second, the target neural network, parameterized by $\theta$ ', will have the same architecture as the main network, but will be used to approximate the Q-values of the next state s' and the next action a'. Learning happens on the main network and not on the target. Target network freezes (its parameters are not changed) for several iterations (usually around 10000), and then the parameters from the main network are copied to the target network, thus transmitting the learning from one to the other, making the estimates calculated by the target network are more accurate.

In order to train a neural network, we need a loss function, which we define as the square of the difference between both sides of the equation [1], that is, the following calculation: $L(\theta) = E [(r + \gamma \max_{a'}{Q(s', a'; \theta')}-Q(s, a; \theta) )^{2}]$