The new Bing GPT-4 chat mode can write Xbox gaming guides, but it’s not perfect

The new Bing’s GPT-4 Chat mode has shown its potential as an Xbox gaming guides writer, but it looks like there’s a long way to go before it rivals actual human writers. 

While it’s currently unknown whether the new Bing AI chat mode will be coming to Xbox Series X, the software is now available in preview to those who sign up for a waiting list. According to Xbox Wire Editor Mike Nelson, it: “knows a hell of a lot about video games”. This has been tested in a way that depicts both the potential and some problems that could arise when an ever-learning algorithm tackles specific gaming knowledge. 

Detailed in the full press release, Bing sites where it’s getting the information from to try and answer what queries you’ve got about some of the best Xbox Series X games. One particular example involved asking the new GPT-4 AI chat mode to give them a detailed recap of what happens in the first 20 hours of The Witcher 3: Wild Hunt, which was pulled from ten different publications, including our sister site PC Gamer and various YouTube sources. 

While the information presented was accurate to the first half or so of CD Projekt Red’s opus, the sheer breadth of sources, including those from YouTube itself, asks exactly how thorough will the Bing AI be in vetting what’s given.

Not all guides online are created equal, and information on ideal strategies in-game can vary so widely. So, pooling from an array of sources varying in credibility isn’t always going to guarantee an accurate answer. 

What’s the right answer again?

(Image credit: Blizzard )

My concerns extend to the accuracy presented in the Bing AI GPT-4 chat mode’s Overwatch 2 question. The test message asks: “What is the best Overwatch 2 character for me?” To this question, the artificial intelligence responds that there’s a total of 33 characters to choose from in the game with Damage, Tank, and Support classes. 

Unfortunately, at the time of writing, there are, in fact, 36 total characters playable in the game, meaning the information that’s being pulled through is outdated. Rammatra is the newest character and was added back in December in Season 2, meaning the AI is around three months behind. 

Of the 10 sources cited, it seems that the GPT-4 has struggled to discern what’s the most up-to-date answer as it provides a vague answer of: “you might want to take a quiz that matches your personality and preferences with the characters” instead of offering up viable options based on the current meta. Bing was able to produce something which sounded fine on the surface but didn’t really answer the question in a meaningful way. It liked Echo, the AI character, but explained nothing about the playstyle or quirks of the character outside of that. 

This appears to be the biggest issue with asking Bing gaming questions and hoping for accuracy. Seeing that YouTube has been cited as a credible source multiple times, what’s stopping people from deliberately spreading misinformation on a subject and then having that adapted into copy from the AI? 

It also raises questions regarding how GPT-4 directly cites existing websites and the extent to which what’s being said in the original material is adapted. Writers aren’t getting the credit they deserve, and the information attached to their names and publications may not fully reflect the original intention. 

I asked ChatGPT to program a game with me, and we failed for hours