A Researcher Has Created A New Force For Game Making With Computer Vision

Computer vision is changing the digital landscape as we know it. The AI technology facilitates faster and simpler processes by using computers to see and understand the visual world with digital images and videos. But while many researchers and companies have been using computer vision to focus on areas like face detection and tracking, independent researcher and Syracuse University graduate, Tempest Storm, also known as Curtis Robinson, has taken a unique path to create a breakthrough in the gaming industry.

Utilizing his computer science background, the software engineer created AI software for video games using computer vision. He found ways to make AI bots fight Pokemon matches independently by only using computer vision tools without hacking, thus giving gamers the option to use bots to fight their opponent or compete with other users who can make better bots. This innovation could change the way games are played.

I have been studying how to make AI bots for Pokemon games for a little over a year, Tempest told The Plug. As a result, I have become quite good at it and built tools that can lay the foundations of making AI esports a reality. My tools lower the barrier of entry for building AI bots for Pokemon matches.

Tempest’s self-funded software works on a Nintendo Switch, a gaming system without a large body of computer vision software development. Large companies that partner with researchers and game developers who work with AI bots give them access to proprietary resources unavailable to the public, but Tempest worked with open source tools and libraries available to everyone. He created his software by collecting and studying hundreds of images from Pokemon Sword and Shield to track patterns and find ways to teach those patterns to the software.

The tool collects statistics on how players and their opponent perform on-screen overtime without hacking the game or asking game developers for the gamer’s stats and performance. 

It also keeps track of which decisions are legal or illegal and maintains the state of the game. Tempest stores all of the information in the software through a template for each Pokemon character with placeholders for expected values like health, attack stats and attack names. The software then teaches the AI bots to compete in the video game based on the data collected from human players.”Not only can the software collect the statistics of how you did, but it can also collect your replay data of the game,” Tempest added. “The replay data could be used to train bots to clone your behavior. From there, you can use any behavior cloning techniques you know to make AI bots that can play similar to a human. That’s how that can evolve into AI eSports.”

He admits that working on the Pokemon is pretty complicated because of the situational aspects involved in the game.

“I haven’t heard of any other researchers using AI on Pokemon,” Renee Gittins, the International Game Developers Association executive director, told The Plug. 

Pokemon is often used as an example of a very difficult challenge because the game overall has many, many different mechanics.”

Gittens has been in the gaming industry for a decade as a technical project management. She has also worked as a programmer and software engineer within game development. The Pokemon battles themselves are challenging because there are many different types of Pokemon and abilities,” she said. It is a much, much more complex environment than something like chess. Evaluating all of these factors and properly storing them is a challenge that requires a large database of Pokemon and abilities and an extremely thorough understanding of the move and status data being presented on screen. 

Gittens said game developers and researchers creating AI bots for competitive games use an API to connect to the game’s backend, so they’re not running through the actual rendered graphics, but few use visuals data and interact with controller-style inputs outside of a few programs like Twitch Plays Pokemon or the Fish Plays Pokemon.” With computer vision, you just need just to capture what’s currently on the screen and then the bot can analyze what’s going on and react accordingly and provide the correct inputs,” she said.

“That is much more difficult to do because trying to determine what’s on the screen using computer vision is a much more difficult task than simply getting raw data back for a computer, Gittens said. 

Now Google and Facebook and other large tech companies put lots of technology and power just in using computer vision to determine what’s going on, on the screen.”

Tempest said that companies are continually acquiring computer vision technologies, startups, and patents. His patent related to automating tasks with computer vision is currently pending. North American revenues from eSports are expected to reach $300 million this year and surpass $1.5 billion by 2023. 

Although his software works on Pokemon Sword and Shield, Tempest said it is general-purpose and can be applied to many games, but he must first study the game for it to work.”I can reuse a lot of the codes, but I would still have to sit down and study those images and patterns because even though the information I want is the same between each game, the location and how to get there would not be the same, ” Tempest said.

“I’ll have to navigate different menus and probably different patterns to teach them. So that grunt work of studying the images, that part will not go away regardless of which game,” he said. Tempest’s tools will be valuable to researchers because they will have access to data that is usually kept by game developers, according to Tempest, who added that companies’ privatized data is increasingly available, and if players start making better bots than developers, he expects they will improve their results. 

Game developers are not used to fighting bots made in this way. They will have to learn to live in a world where people can make intelligent bots without maliciously attacking their software, he said. Tempest hopes chipmakers, graphics processor companies that use computer vision like Intel and Nvidia, and researchers and universities who are studying AI will eventually adopt his software. According to Robotics and Automation News, graphics processing units (GPUs) were originally used to quicken the images for computer games, but their main component is processing large amounts of data quickly, including visual data. Although a GPU isn’t required for Tempest’s software, it can speed up the software using optical character recognition (OCR) to help translate an image into text. 

The software can also handle various languages and translate it into English with high accuracy.

If companies can get my software to work on their single board computers, this will show that they can get powerful computer vision applications in the hands of the masses due to their low cost. The reason why some of these work better than standard laptops for desktops is because they are cheaper and geared towards computer vision  Tempest said.

I see a great opportunity to bring more of these devices into the educational space for classes that focus on computer vision gaming and machine learning, he said. 

Overall, he believes that his breakthrough will influence people to focus more on computer vision in games, and it will allow them to compete in ways “that were previously impossible. His software is as novel as it is to be a Black game developer. A 2019 Developer Satisfaction Survey (DSS) conducted by IDGA and Western University revealed that out of 1116 respondents, only 2% identified as Black, African American, African or Afro-Caribbean. I intend to profit off my work before corporations attempt to set themselves up as gatekeepers over it. This technology will lay the foundation of automating a good portion of the world in the future, Tempest said.

Brianna Rhodes

Brianna Rhodes is a DMV-based journalist and entrepreneur who has written for organizations such as Blavity, Inc., The National Trust for Historic Preservation, theGrio and The AFRO-American Newspaper. She is an alumna of the University of Maryland Philip Merrill College of Journalism and the University of North Carolina at Chapel Hill.