You Can Use GTA 5 To Train Your AI Cars

Don’t forget to give your virtual car some treats

Every now and then, some kind of whacky art project or scientific research initiative pops up that desperately tries to prove that GTA 5 is actually more than just another violent murder simulator. Whenever this happens, masses of talent deprived YouTubers upload “brutal kill montage” videos en masse.

Regardless, the intellectuals of our time are undeterred in their quest to actually take the extremely impressive and detailed game world of GTA 5 and do science with it. We’ve often praised the work Rockstar did when crafting world of their latest open world title, and we’re not alone in this.

After all, not every open world out there is used for art projects exploring the identity crises of modern youth, or to help develop environment detection and indexing algorithms used for self-driving cars. GTA 5 has also been used to argue that the real world may very well be a simulation too.

With accolades like that, it should come as no surprise that GTA 5 is so highly regarded and immensely popular as it is. Players and reviewers alike have cited the detailed and immersive map as the reason for their high opinions of the game, and other AAA developers looked to GTA 5 for pointers.

In any case, it seems that GTA 5 has a future in the realm of self-driving car training programs. Yet another project has launched that uses the game’s world to ‘teach’ AI programs how to drive cars safely, but this time it’s something you can do yourself.

Yes, you can actually nurture a little budding car-driving AI right on your desktop, so long as you own a legitimate copy of GTA 5. How futurisitic is that? Craig Quiter’s project DeepDrive is behind this system, called Universe. It’s all open-source, so anyone can download the mod and the pre-trained Agent (the program that does the driving) and plug it into GTA 5, but they won’t benefit much from it.

This release includes a baseline agent, trained via imitation learning on 21 hours (about 600,000 images) on the game’s AI driving. (The built-in game AI is a good initial target: it performs better than a typical human since it can access internal game state, though it still makes mistakes such as making U-turns on the freeway.) The baseline agent can drive in a variety of different weather conditions, react to traffic, and keep to its lane. This agent is a start, which we invite the community to improve upon!

To the average Joe, this is nothing more than an interesting little tidbit or pastime. “Hey, this is cool” is about as much as you’ll get out of this before going back to playing the actual game. However, other researchers and AI developers can use the system to hook up their own Agents and train them in GTA 5.

The reason GTA 5 is used so often for AI research and development is the complexity of the open world. It’s pretty much a full blown simulation with a wide array of biomes, terrains, traffic scenarios and weather conditions that allow the Agent to “experience” and learn from a multitude of situations.

GTA V gives researchers access to a rich, diverse world for testing and developing AI. Its island setting is almost one-fifth the size of Los Angeles, giving access to a broad range of scenarios to test systems. Add to that the 257 different vehicles, 7 types of bicycles, and 14 weather types, and it’s possible to explore a huge number of permutations using a single simulator.

The other benefit highlighted by OpenAI’s Universe is the same one we covered some time ago: the indexing and labeling of objects “seen” by the Agent. Another self-driving AI training project has made use of GTA 5 before, allowing the program to learn within the bounds of the game without risking things like bodily harm or material damage.


In both cases, a mod is used to reduce violence in GTA 5, preventing the AIs from encountering random gunfights or drawing the ire of law enforcement should they ever make a mistake. In case you’re worried this little experiment will lead to the rise of Skynet of something, be assured that all of these driving Agents are learning in a completely violence-free environment and are really quite cuddly.

Another benefit of using GTA 5 as a platform to train driving AIs is one many players have noted in comments. The built-in driving AI of NPCs in the game is horrendous, and they are frequently startled into erratic behavior. If an AI learns how to drive safely in GTA 5, it’s going to be absolutely fool-proof in the real world. You might think the average real-life drive is a bum, but no-one is as bad as the NPCs in this game.

Quiter also cites the flexibility of the game thanks to modding support, allowing users to add any kind of scenario that isn’t present to the simulation. Whether you want new environments, vehicles, weather conditions or entirely new cities to train your fledgling AI in, you can always mod the game.

The environment also enables collecting massive amounts of labelled data: you can use the underlying GTA V engine to collect 2D or 3D bounding boxes and segmentation labels for cars, pedestrians, bicycles, animals, road surface, traffic signs, or any one of GTA V’s other 7000+ objects. The environment can also be extended via mods for real-world vehicles, road construction, and even entire cities.

While this is a curious project and certainly useful for other developers experimenting with this technology, it really is just a second tentative step in this vast new land of possibilities. One great way to expand this would be using the accessibility of the open-source Universe project to crowd-source AI learning.

While I’m not an expert by any means, it could be potentially beneficial to set up a system that networks these individual Agents all learning out there, pooling their collective experiences. This way thousands of hours of training can be accumulated as a much quicker pace and the various mods used by the different users will result in a wider range of scenarios trained for.

If the collective experiences of these Agents are then pooled into a single AI program, the development safe(r) self-driving vehicles could be a quicker process. Of course, there are likely a million technical reasons why this is either difficult or impossible to achieve, but it would be pretty cool nonetheless.

Will you be trying out this little GTA 5 tool, or do you just prefer reading about this stuff?

🙊What do you think? Sound off below!🙊