The goal of the Digital Scout Project is to develop a computer system that can learn to interpret and reason about American football by watching football video. Such a system will contain many components. Below is an overview of each of the components we have developed so far, many of which perform tasks similar to those that consume many man hours of manual labor from the crews of video scouts employed by all professional and most college football teams. You can click on the title of a component to find out more about it.


Input video frame The input to our system is a collection of several hundred video clips from Oregon State football games graciously supplied by the OSU football program. Each clip contains a sideline view of one entire football play.

Video Registration

A registered video frame Registering the input video to a model of the field is an essential first step in the interpretation process because it provides a means by which to evaluate video data in terms of a static coordinate system—namely, in 2D football field coordinates—rather than in terms of video coordinates, where spatial information is almost meaningless because of the erratic motion of the camera. This ability is important in later stages of our system, where the spatial relationships between players and their locations on the field are very meaningful.

Formation Recognition

A labeled football formation At the beginning of every football play, the players line up in a structured formation, the configuration of which is a part of the coach’s strategy. An important part of our system is hence the ability to recognize formations—that is, given an image of a formation, to be able to figure out which players are in that formation as well as to determine their locations. In addition, formation recognition provides a means by which to initialize the player tracking algorithms that will be used in later stages of our system.

Player Tracking

A frame of player tracking In order to analyze players’ actions, we must be able to follow them. Player tracking provides raw player trajectories that can be analyzed in later stages of our system in order to determine what players are doing and what plays are being run.


Football play diagram Given a video clip containing a single football play, we would like to be able to compute a high-level description of that play, similar to a play diagram from a coach’s play book. Our long-term research goal is to be able to analyze a collection of video of a certain team to determine that team’s strengths and weaknesses and to recommend football strategy to an opposing team based on our analysis.