Gfootball

Recent advances in the field of reinforcement learning have been supported in virtual learning environments like video games, where new algorithms and ideas can be tested quickly in a safe and reproducible manner. In this sense, Google has presented a new research environment based on football, a new reinforcement learning system where agents can train and simulate a soccer match in a 3D system. The resulting environment is challenging, easy to use, customize and, in addition, it is available under a free software license.

Apart from all this, Google has provided us with a format capable of creating multiplayer and multi-agent experiments, that is, capable of conducting competitions such as https://www.kaggle.com/c/google-football/overview, one AI league where you can earn up to 6000$ with your algorithm!!!!!

That said, in this article we are going to show you how to install this environment, be able to run it, and experiment with it. and play a match against your own AI.

1.Installation

The installation of this game can only be done on Linux or Mac systems by following the following commands:

Linux

   sudo apt-get install git cmake build-essential libgl1-mesa-dev libsdl2-dev \
   libsdl2-image-dev libsdl2-ttf-dev libsdl2-gfx-dev libboost-all-dev \
   libdirectfb-dev libst-dev mesa-utils xvfb x11vnc libsdl-sge-dev python3-pip

macOS

To perform this installation, it is necessary to have installed brew, a The installation program explains the process step by step, what it is going to do and takes a pause to confirm before starting each of the steps.

Once installed, the following commands must be executed:

     brew install git python3 cmake sdl2 sdl2_image sdl2_ttf sdl2_gfx boost boost-python3

To install pygame, you will also need the following packages:

     brew install sdl sdl_image sdl_mixer sdl_ttf portmidi

Subsequently, you can use the library with our project with the following command:

     pip3 install gfootball

Finally, and as a test, we can play a game against the base AI of the game:

 python3 -m gfootball.play_game --action_set=full

That's it. An image like the following will appear:

Ejemplo de visualización del juego
de fútbol de google para entrenar ia de google — 1 Game display example google soccer game to train google AI.

Command guide:

UP ARROW: run up.
DOWN ARROW: run down.
ARROW TO THE LEFT; run to the left.
RIGHT ARROW:runs to the right.
S: short pass in attack mode, pressure in defense mode.
A: high pass in attack mode, sliding in defense mode.
D: shooting in attack mode, team pressure in defense mode.
W: long pass in attack mode, goalkeeper pressure in defense mode.
Q: changes the active player in defense mode.
C: dribble in attack mode.
E: sprint.

The result will be similar to what you can find in the following video: https://youtu.be/YKv3WTZcV28

2.Implement your AI for the contest

In this section we will focus on the code necessary to implement AI. Inside the project https://github.com/al118345/OpenAi_Examples there is a folder called code we will find a package called gfootball all projects dedicated to this project ( https://github.com/al118345/OpenAi_Examples/tree/master/code/gfootball

In a first approximation, we wanted to experiment a little with the representation of the players, the ball, their position in the field and what values each of them represent.

To carry out this objective, we have used the "GFootball-11_vs_11_kaggle-SMM-v0" environment where the SMMWrapper container to return frames showing team positions, the position of the ball and the current position of the active player.

These frames can be used by a convolutional neural network, but they first require processing. which could consist of:

Reorganization 2D convolutional layers typically expect an input of the form (None, x, y, c), where c is the channel of color. We need to divide the observation matrix (x, y, 4) into 4 matrices.
Climbing The data range is 0 -> 255. It is necessary to scale to 0 -> 1.
Adding time Static frames do not convey direction or speed, only position. We can add temporary information to each frame in two ways:
- Remember the previous n frames and deliver them to the network in the form (None, x, y, n_buffer).
- Remember the previous frame and calculate the difference between this and the last frame.

Without going much further into this section, we can see an example of this representation in the following image. In the upper part both teams are represented and in the lower part the ball and the position of the active player in each played.

2 Example of viewing a match gfootball soccer game distributed by the different components of the game.

The code used can be found at the following url https://github.com/al118345/OpenAi_Examples/blob/master/code/gfootball/__init__.py

3-View an agent.

Now, what happens if you want to view an agent? For this I have implemented the subfolder example_test_agent_invideo ( link ) which is made up of the following files:

template.py: Example of a completely random agent. I use this algorithm as a test, I understand that a random agent is perfect for comparing behaviors.
submission.py: Example of an intelligent agent. It is an example of how to implement an agent proposing a strategy of hitting the goal when you are close, passing when you are in the middle of the field or in your area

__init__.py: File composed of the following code:

     from kaggle_environments import make
     env = make("football", configuration=&#123;"save_video": True, "scenario_name": "11_vs_11_kaggle","logdir": "." ,
       "render": True, "running_in_notebook": False&#125;)
     output = env.run(["submission.py","template.py"])[-1]
     print('Left player: reward = %s, status = %s, info = %s' % (output[0]['reward'], output[0]['status'], output[0]['info']))
     print('Right player: reward = %s, status = %s, info = %s' % (output[1]['reward'], output[1]['status'], output[1]['info']))
     env.render(mode="human", width=800, height=600)

This example has as an important element the configuration of the environment. Through the configuration "save_video": True and "logdir": "." We can generate a folder with a .webm file where the behavior of the agent is stored visually.

Finally, the line output = env.run(["submission.py","template.py"])[-1] serves to inform the game what agent will be in the game. On the left side the intelligent agent and on the right the random one in my case.

You can consult the result of this implementation at the url https://youtu.be/YKv3WTZcV28 and more information is available Kaglle documentation about the bookstore kaggle_environments

Example code

In the repository https://github.com/al118345/OpenAi_Examples I have made available the code used in this project and the following video https://www.youtube.com/watch?v=ME2FxGIzG7E explanatory: