Watching Football for Science

Computers that learn have long been an interest for Alan Fern, associate professor in the School of Electrical Engineering and Computer Science (EECS). As early as high school he was writing programs like a Connect Four game in which the computer played both sides and adjusted strategy based on which side won.

“Human intelligence is really an amazing thing, and trying to understand how to reproduce that in a computer is one of the great unsolved mysteries. If that's ever solved it would be one of the biggest landmarks in human history,” Fern says, who believes it can be done.

But artificial intelligence is not his only interest. He also likes to watch football. And it occurred to him that people who have just a rudimentary understanding of football can still follow a game, but computers struggle to just keep track of the players. It was just the kind of problem he likes to solve.

Although he had no background in computer vision, he decided to pursue this self-proclaimed “crackpot idea” of trying to get a computer to understand a football game. Fortunately, soon after he embarked on this new direction, Sinisa Todorovic, an expert in computer vision, joined the EECS faculty. Together they collaborated on a high-level vision project using basketball and volleyball video.

Fern started the football project by visiting the video crew of the Oregon State football team and learned about how much human effort currently goes into annotation, organization, and analysis of football video.

“It was clear that the project could have commercial potential,” Fern says.

Indeed, he and Todorovic recently began working with a top company in the area of sports video storage and organization. The company services over 13,000 high school, college, and professional teams including the Oregon State Beavers.

Combining their areas of expertise in machine learning and computer vision has allowed Fern and Todorovic to make advances in research that could be useful in multiple domains. Applications include monitoring security cameras, identifying patients who need care in settings like hospitals or nursing homes and searching for content in large video collections.

Video: Advancing Computer Vision by Watching Football

Football offers a number of challenges for the computer. For example, the camera is often panning and zooming which the brain can easily adapt to, but confuses the computer. The football players move erratically, sometimes running into each other, which makes tracking difficult. They are also dressed alike and can sometimes blend into the background creating problems for identification of individuals.

Fern and Todorovic found that by using machine learning techniques they could train the computer to track individuals moving around on the field. The computer essentially “watches” a lot of Oregon State football games in which the players and their locations have been identified by a human, and adjusts its program to increase its tracking accuracy. By practicing on instances where the player locations are known, the computer can develop strategies that generalize to new videos.

Starting with the basics of identification and tracking, the computer can then process the trajectories of the players to try and understand more complicated concepts such as strategy.

“One of the most interesting things about this project is we’re working in this fun domain of football, but the results are really applicable quite widely,” Fern says.

What a great excuse to watch some football.