When we reported on the free-roaming Groot animatronic, we mentioned that Imagineers were working on eyes that will be able to recognize guests. A Disney patent application published April 29, 2021 explains one way Imagineers could add natural eye contact and facial recognition to animatronic figures.

“Eye Contact Sensing and Control for Robotic Characters” describes a system that uses light passing through the animatronic’s eye to “see” guests and make eye contact. Stationary cameras assist the animatronic as it locates the guest in the scene.

Fig. 1 provides an overview of the system. The audio-animatronic character (120) is represented by the box on the left. A light source (150) is embedded inside one robotic eye (130). The light (152) is projected into the scene (102), which is occupied by a person (104).

The trick is finding a light source that can be embedded inside the animatronic’s eye and be detected by the image processor (170).

Such light sources can be operated in a continuous manner or in an on/off sequence known to an image processor to produce an easy to detect beam or structured light emission that will point to exactly where the robotic character is presently looking as the robotic eye aims or targets the output of the light source held within it.

Patent Application US 2021/0122055 A1

How does the animatronic determine where it should look? The answer is through a system of cameras and image processing. The character is able to fix its gaze on the observer (104), which is located by a system of cameras (160). The image processor (170) analyzes the camera output (162), locates the observer’s face in the scene, then tells the animatronic character where to look. An eye movement module (184) drives the servos (140) which move the character’s eyes. As the eyes move, the light source (150) follows. When the eye target (154) — think of this as the character’s focal point — fixes on the observer, the eyes will lock on to the observer.

As an observer comes closer to the character, the eyes are capable of “toe-in.”

There are many techniques and/or modifications that can be made to the robot systems taught herein to handle toe-in or vergence movements of the eyes. These movements of the robot’s eyes are desirable when an object that is the point of interest (such as a person’s face or their hand or the like) is positioned in close proximity to the robot’s face.

Patent Application US 2021/0122055 A1

To make the eyes even more realistic, the character’s eyes are capable of “saccading.” Like human eyes, these audio-animatronic versions rarely stop moving.

The system is able to quickly make adjustments. In fact, eye contact can be maintained throughout character movements. For example, the character’s head can tilt while still maintaining eye contact with the observer.

Fig. 2 shows how the system uses a beam splitter (220) to direct light from the scene (210) to both the target-sensing camera (250) and a face-sensing camera (234). The output from both cameras is parsed by the image processor (260) then passed on to the eye direction controller (280). For a roaming character like Groot, these cameras would need to be hidden throughout the area in which the character roams.

This patent application likely details the system previewed in a Disney Research Hub video released in October 2020. The video below shows the system at work. The camera used to track guest movement is clearly visible.

This eye contact control system would bring a new level of interactivity to Disney audio-animatronics. Are you ready for characters that will lock eyes with you? Sound off in the comments.