Sony’s AI Hawk-Eye tech could finally solve soccer’s VAR controversies

Sony’s Hawk-Eye division has already brought goal-line technology to soccer, but now a next-gen version called Skeletrack could finally end the sport’s VAR offside controversies.

In a demo of Hawk-Eye’s new Skeletrack tech at CES 2022, Sony showed how the ‘virtual recreation’ of on-pitch action could soon be used to automatically judge offside decisions with similar accuracy to its current goal-line technology.

VAR, or Video Assistant Referees, have become a mainstay in soccer leagues and tournaments around the world, with the English Premier League adopting video replays for incidents including potential penalties and red cards from 2020. 

But the technology’s application to the offside rule, which stipulates that an attacking player cannot be beyond the last defender when receiving the ball, has been controversial. This is because it’s still reliant on human assistant referees viewing video footage, with the aid of one-pixel-thick lines to help them make the offside decision.

Skeletrack is potentially a big improvement on the current VAR process because it combines optical tracking technology with AI tech that can analyze video capture in real time.

During the demo, Sony said that “Skeletrack analyzes the ball position as well as the players’ positions and postures in real-time with AI, then visualizes the game through virtual recreation”. It then showed how this might be applied to a VAR offside decision, with 3D models of the players (and their relevant body parts) being automatically analyzed in real-time to see whether an offside call should be made.

Sony didn’t reveal any more details about when this next-gen Hawk-Eye tech could come to VAR, simply saying that “Skeletrack can advance the fairness and transparency of decision making across sport”. But if it works as well the demos suggest, we could see it appear in soccer leagues and tournaments in the next few years.

While offside decisions are one big use case for Skeletrack, Sony was also keen to highlight at CES 2022 that the tech is suitable for “unlimited use cases”, including gamifying the watching experience for fans. By effectively giving armchair viewers a virtual camera, it could let fans see the action from the goalkeeper’s perspective, or give them “a seat in the stands in one of the best stadiums in the world”.

The first stadium to be recreated virtually will be Manchester City’s Etihad stadium, with Sony using CES 2022 to highlight a partnership with the club that it first announced in November 2021. This will apparently deliver an online ‘metaverse’ for fans to interact with in VR, including a digital version of Manchester City’s home ground. 

Analysis: another step towards AI referees

(Image credit: Sony)

On-pitch referees will always bring an essential human element to managing soccer matches, but their decisions are becoming increasingly automated thanks to the accuracy of real-time video analysis – and Hawk-Eye’s new Skeletrack tech is another step down that path.

Hawk-Eye, which Sony acquired in 2011, has become the world’s most popular soccer goal-line technology system, but offside decisions have proved a tougher nut for technology to crack due to the number of moving parts involved: the ball, plus the fast movement of attackers and defenders.

While there are still obstacles to a 100% accurate offside system, including determining the precise moment when a ball is struck, Skeletrack does look like a promising advance over the current VAR process.

In England’s Premier League, which is the most-watched sports league in the world, offside decisions are still determined by the manual positioning of virtual lines over video replays, which are taken from five video cameras around the stadium.  

But this process was so controversial that two big changes were made for the current 2021/2022 season. Firstly, for marginal offsides, thicker lines are now applied over the system’s default one-pixel lines, with incidents where those lines overlap now being deemed onside. Also, to avoid showing the clunky decision-making process to TV viewers, only a final image confirming the decision is now broadcast.

The big advance with Skeletrack is the inclusion of real-time AI analysis, which is apparently advanced enough to automatically make a call on sporting decisions as complex as soccer’s offside rule. The tech is already being used by analysts in the USA’s NBA league and Italy’s Serie A soccer league, but football fans and long-suffering assistant referees will hope to see it also taking the uncertainty out of offside decisions soon.

CES 2022: all the news straight from the world’s largest electronics expo