Associate Professor KITAHARA Itaru, Faculty of Engineering, Information and Systems

While watching a broadcast of a sports event such as a soccer match, have you ever wished that you could see a play from a different angle? To be able to see a play from the perspective of a favorite player or the goalkeeper, for example, would certainly create a feeling of being right in the middle of the action. The day is not far off when we will be able to watch sports using free-viewpoint video (FVV) image processing technology that renders captured video in real time 3D for viewing from various angles.

There are more chances now than ever to see 3D content, with 3D movies and 3D TV. Yet that content is pre-made and does not allow a peak behind the scenes. What Prof. Kitahara is developing is a free-viewpoint video (FVV) technology that converts real-world video footage to 3D in real time and allows viewing from any angle or distance one wishes. The potential applications extend beyond sports events to any number of fields, such as the medical field—for viewing the interactions or facial expressions of a nurse or therapist, for example, or to confirm a difficult-to-see spot in surgery.

There is no direct way to film in 3D. What has to be done is to film a scene using cameras at multiple angles and piece together those flat images to create a 3D model. A video can be produced by continuously switching among the images. In a soccer match, for example, there may be one camera on the stadium ceiling and eight other cameras distributed in the stands around the field. One would assume that focus could not be maintained with this method, but the important point is to be able to capture the scene in its entirety. One can decide later which part to view and from what angle.

Video: FVV

The key to filming with multiple cameras is to synchronize all of the cameras to shoot at exactly the same time. When you have fast-moving action, such as in sports events, even the slightest lag can prevent proper 3D image processing. Prof. Kitahara and his colleague's idea to resolve this issue was to use the time information from a GPS signal to closely control the timing of the filming.

A second technical issue is determination of the correspondences between the 3D space and the filmed footage. To create a 3D model from 2D footage requires knowing what position in the 2D footage a point in the 3D space is being viewed from. The problem was that no one had ever tried to determine such correspondences in a large space like that of a soccer stadium. Fortunately, someone involved in the design and building of soccer stadiums was brought on board the research team. The construction expert came up with the unexpected approach of using 3D surveying instruments, used at construction sites?not something people in the field of imaging studies would have thought of. The research team quickly accepted this brilliant idea and proceeded to create a revolutionary world-leading large-scale filming system.

Bullet time shooting studio

Based on images shot in such a way, a method called the "human billboard" was used to create the 3D model. For a soccer match, the position of each player is estimated from the filmed images. A standing two-dimensional screen is set up onto which images shot from various angles are pasted. By switching between the pasted images according to the viewing angle of the screen, it is possible to reproduce a pseudo-three-dimensional view. Moreover, this method is simpler and faster for displaying 3D images than building a 3D model using computer graphics. Also, the system produces a 3D model with a small amount of data, practical for live broadcasts over the Internet.

The next area to focus on was the interface. A mouse and keyboard could not be used to perform FVV operations because it would interfere with watching a live match. A more intuitive interface was needed, so a system was developed that uses both hands, each holding a marker that when moved would respectively adjust the virtual camera position and the angle that the viewer wishes to see. If one zooms in to the position of one player, it is possible to experience the game as if one were on the field. Methods of operation using a tablet or other device are under development that will make it even easier to experience free-viewpoint video.

Video showing an attack on goal from the perspective of the center forward can be reproduced

High-resolution images are also essential. To capture these, the technique employed in the movie feature The Matrix was used, known as "bullet time". Multiple cameras surrounding the subject shoot the scene and the freeze-framed subject is viewed from various angles. Prof. Kitahara is trying to apply this shooting technique on a stadium scale. Bullet time is effective when shooting a person or other subject at close range, but requires dozens of cameras when shooting a large space. Image processing technologies also need to be improved. At present, a trial has been set up with the Kashima Antlers professional J. League soccer team that uses 10 high-resolution cameras set up behind the goal at intervals of several meters to allow FVV of the goal area.

Video: Soccer match using FVV

Prof. Kitahara originally took an interest in 3D images as a junior high school student in 1985 when he attended Expo '85 in Tsukuba. He tells the story of wandering into a futuristic city and putting on blue-and-red glasses to see 3D images for the first time. The visual experience stimulated his other senses and made a lasting impression. Guided at the subconscious level by that experience, he came to study at the University of Tsukuba and steadily worked on research on 3D free-viewpoint video from that point forward.

The current issue he faces is creating a sense of immersion. The key is in improving image quality and data transmission and processing speeds, working in conjunction with broadcasting systems. The pace of research in other areas, such as 4K and 8K video technologies and the development of information-based infrastructure, is picking up and there is great excitement that an entirely new experience in watching sports is emerging as we head toward the Tokyo Olympics in 2020.

Article by Science Communicator at the Office of Public Relations