Categories crunchfx

Improve live football broadcasts by eliminating camera distractions with AI

KTU researchers improve live football broadcasts by removing cameramen's distractions with AI

Videographers video painting basic usage example. Credit: Serhii Postupaiev

While the sports industry is constantly improving the viewing experience for viewers at home, some issues remain unresolved. One such issue for football fans is cameramen accidentally appearing in each other’s footage during live broadcasts. These incidents not only negatively impact critical game moments, but can also lead to revenue losses for broadcasters due to viewer dissatisfaction.

To solve this problem, researchers at Kaunas University of Technology (KTU) have developed an end-to-end system that improves the viewing experience by eliminating visual distractions caused by overlapping camera angles.

“Our new invention is an algorithm for detecting video operators,” says KTU professor Rytis Maskeliūnas, one of the creators of the innovation.

Another member of the research team, Serhii Postupaiev, emphasizes that due to the complex nature of live sports broadcasts and the large number of cameras around the stadium, the presence of cameramen in the frame is a common problem in football broadcasts.

“The number of camera points at prestigious tournaments can start at nine, and there can be many overlapping viewpoints, which contributes to visual distraction issues. These issues severely constrain camera crews, as they have to constantly film the game while avoiding catching each other, which can lead to a loss of context during some game moments or a less dynamic and immersive broadcast,” explains Postupaiev.

Elimination of visual distractions

To solve this problem and eliminate unwanted objects during live broadcast, KTÜ scientists designed and implemented an end-to-end system.

For its work, it used the YOLOv8 model, a cutting-edge object detection system known for its speed and accuracy. Standing for “You Only Look Once,” YOLOv8 can detect and classify objects in images in a single pass, making it ideal for real-time events like live football broadcasts.

“It works by dividing the image into a grid and estimating the bounding boxes, class probabilities, and segmentation polygons for each grid cell. This allows it to identify and segment the cameramen,” says Serhii Postupaiev, who recently graduated from KTU with a Master of Science in Artificial Intelligence in Computer Science.

To enable the YOLOv8 model to accurately detect and segment cameramen during football matches, a dataset needed to be created.

“I created this dataset to include a variety of cameramen of different sizes, shapes, and equipment types, captured under different conditions and at different stages of the game. Now YOLOv8 uses this dataset to determine where the cameramen are in the video frames,” adds Postupaiev.

As the inventor explained, this process was necessary to create the basis for the actual removal of operators. For this purpose, video painting technology was used.

Researchers use AI to improve live football broadcasts by removing cameramen's distractions

Strategic camera positions and overlapping viewpoints (dotted lines) in a football stadium. Source: Kaunas University of Technology

“The term inpainting in deep learning refers to the process of reconstructing lost or corrupted parts of images and videos. Specifically, in this case, it is used to remove cameramen from football video feeds,” says Postupaiev.

Technology based on artificial intelligence (AI) and computer vision analyzes video frames to detect unwanted objects, such as cameramen, and fills in the removed areas with relevant background details. The modified frames are then streamed back to viewers, providing a more immersive and professional broadcast.

Maskeliūnas adds that in television presenters this algorithm can process the recorded image before it goes on air, with a delay of a few seconds from the actual captured moment, which is considered a live broadcast. As the equipment improves, he believes that artificial intelligence will perfectly fill this time gap.

Shifting focus from just capturing the action

With this new technology, watching football matches at home will be significantly improved, including a smoother viewing experience.

“The broadcast will feel more polished and professional without interruptions caused by cameramen appearing in places they shouldn’t. This improvement will reduce the number of cases where important moments of the game are missed due to distracting shots,” emphasizes Postupaiev, who earned his Master’s degree for this project.

According to Postupaiev, further research in this area could usher in a new era in sports broadcasting, shifting the focus from merely capturing the action to creating a fully immersive and seamless viewing experience.

“By using the cameraman’s painting method, broadcast companies can explore innovative camera angles, perspectives and effects, bringing games to life in new and exciting ways,” he says.

Additionally, camera operator inpainting can extend beyond live broadcasts to enhance pre- and post-match analysis, offline highlights processing, and restoration of archive footage.

“This could even breathe new life into old recordings of classic matches,” adds a KTU graduate.

The invention is not limited to football; it can also be applied to other sports with similar broadcast challenges. Dynamic sports such as futsal and basketball, which require immersive broadcasts, could also benefit from this technology.

“This is another example of what modern AI applications can do. We often hear about medical applications, but here we have a consumer-oriented approach to editing images that we do not like. In the future, such technology will be able to remove ads, for example, or replace them with others, thus constantly updating content with a level of precision that the human eye will not notice,” says Maskeliūnas, professor at the Faculty of Informatics at KTU.

The article, titled “Real-time camera operator segmentation in football video broadcasts with YOLOv8”, was published in the journal Artificial intelligence.

More information:
Serhii Postupaiev et al., Real-time Camera Operator Segmentation with YOLOv8 in Football Video Streams, Artificial intelligence (2024).DOI: 10.3390/ai5020042

Provided by Kaunas University of Technology

Quotation: Eliminating cameramen’s distractions with AI to improve live soccer broadcasts (2024, July 10) Retrieved July 10, 2024 from

This document is subject to copyright. No part may be reproduced without written permission, except for any fair use for private study or research. The content is provided for informational purposes only.