Artificial intelligence has proven itself adept at the most complex games and puzzles, beating humans at chess, poker, and Go Now a YouTuber is trying to teach a generative AI and AI vision tool to play Super Mario 64

Josh Bickett created a new AI framework that allows GPT-4-Vision, the model behind ChatGPT, to play a variety of games, starting with the Nintendo classic

While it "could" play the games, it did not do a particularly good job because it took a long time to process the information The results demonstrated the capabilities of AI Vision

AI vision is a rapidly growing field that can look at the real world, analyze what is happening, and make decisions based on what it sees This approach has proven especially useful with robots, and could even be seen in smart cat flaps at CES

Currently, you can't tell ChatGPT to play Super Mario 64 and win It's not that smart, but the model built on top of it has the potential to do much more than what can be done within a chatbot

Bicket has used AI vision models to create a framework for multimodal gaming It works by looking at a screen, examining what it sees, and directing the controls

The multimodal gamer framework takes a screenshot of the game, determines what to do next, and uses the controller to direct the action

In tests using a web emulation of Super Mario 64, it was able to detect Mario's position and path on the screen and instruct Mario to follow the path It could also determine how long to hold down a key to make Mario move, jump, or dodge an object

The biggest problem is lag; there is often a long gap between when GPT-4-Vision observes the screen and when it makes a decision about what to have Mario do, resulting in the bad guy getting beat up

"We found that there is some latency in these models, which is a major problem in the way they navigate and make decisions It would be interesting to see what would happen to these models if there were no latency," Bickett wondered

At the end of the video, after several iterations of the code during testing, the AI was able to move around, jump, and interact, but it was not perfect It sometimes made the wrong choices and looked more like a toddler mashing buttons than a true gamer playing

This is only version 1 of the tool, which uses an AI model running in the cloud This caused latency issues and delays in decision making Also, Super Mario 64 did not allow for fine-tuning

In the future, as technology improves, we may see true AI-based walkthroughs, players, and guides for every game in the library

This could be driven by custom local AI vision models running on local NPUs such as those in the new Intel Core Ultra chips, ultimately presenting a use case for a new generation of AI PCs