Using Thermal Imaging Cameras Monitor Stress Levels of Helicopter Pilots

Flying a helicopter is a stressful activity. Pilots have a high workload and are constantly under pressure. Italian helicopter manufacturer Leonardo wanted a reliable and objective way to measure the stress that pilots experience, so they would be able to design more functional cockpits and train helicopter pilots more efficiently. The company brought in high-end thermal cameras to monitor pilots during flight simulations and to exactly see which procedures and operations cause stress.

Leonardo’s Helicopter Division manufactures an extensive range of modern helicopters for commercial, public service, and security and defence applications. The company’s A. Marchetti Training Academy in Sesto Calende, Italy uses the latest training technologies and offers a wide range of training services. The Academy has an impressive track record in successfully training commercial and military pilots from all over the world.
“Working the cockpit of a modern helicopter is quite challenging, especially because we are receiving information from inside and outside the cockpit,” explained Dario de Liguoro, Chief Instructor Flight Training Standard at Leonardo. “This flood of information imposes a lot of stress on pilots. So, for training purposes, it is very important to understand what exactly causes stress and how we can better train our pilots in flight simulators, so they can cope with this stress.”
Infrared stress monitoring system (ISMS)
To monitor stress and workload of crew members in real time, Leonardo teamed up with NEXT2U, a scientific start-up from University of Chieti- Pescara, and the Italian Army Aviation (AVES) to develop ISMS, a patent pending Infrared Stress Monitoring System.
The ISMS is a non-invasive stress analysis system that includes a thermal camera in the cockpit pointed at the pilot, and a dedicated workstation with software that can extract and analyse the collected thermal facial data. The camera will monitor facial temperature modulation of the crew while performing typical pilot tasks. The system is especially relevant for mission tasks where pilots need to process a lot of data in real time coming from multifunction displays, helmet mounted displays, and multi-function keyboards.
The ISMS helps Leonardo understand the stress imposed on pilots during missions, either in simulators or in real missions. This technology has different applications. First of all, it enables Leonardo to test and verify cockpit human machine interfaces (HMI) in an early stage and helps them to discover how information or commands should be presented to the pilot in order to keep his cognitive workload at the lowest level possible. But the system can also be used in flight simulators to help instructors evaluate the effectiveness of a training program.
Stress is thermal
Stress activates the autonomic nervous system and that system in turn affects the skin temperature. Thermal imaging is therefore an ideal tool to monitor emotional states, given the use of proper stress analysis algorithms.
“Cold sweat or feeling warm: both are indicators of stress,” said Professor Arcangelo Merla, PhD, founder and Scientific Director of Next2U. “All of these feelings are associated with the neurovegetative activity, and we capture these processes with thermal imaging in a very efficient way by looking at the temperature of specific facial regions of interest.”
Non-invasive monitoring
The NATO community defines stress levels by means of the Bedford Workload Scale, a simple, hierarchical ten-point rating scale which is presented to the pilot after a mission. However, this self-evaluation technique to assess stress and cognitive workload remains subjective. Pilots can overestimate or underestimate their performance. That is why Leonardo wanted to replace self-evaluation with a more objective, reliable, and scientifically proven technique.
“The ISMS provides real-time information about pilots’ stress,” said Marco Gazzaniga, Principal Systems Engineer at Leonardo. “There is no need to wait until the end of the mission to debrief the pilot. Now we can capture stress information in every manoeuver, so we can immediately identify where we need to improve our HMI.”
Thermal cameras are non-invasive, which means that the pilot does not experience additional stress from wearing contact probes. The cameras do not interfere with normal activity of the pilot, which is important in cockpit procedures that are highly established and certified.
Cooled thermal camera
“Choosing the right camera for the ISMS has been quite challenging, because we needed a camera that is very stable, reliable, fast and sensitive,” said Arcangelo Merla. “This is why our attention went to the cooled FLIR A6750sc. This camera is an excellent compromise between performance, dimension and weight, which is very important for flight simulator applications that have very demanding technical constraints.”
The FLIR A6750sc camera can freeze motion and perform accurate temperature measurements on moving subjects. The sensitive A6750sc can also perfectly capture subtle temperatures changes that are associated with human emotions. The camera’s high resolution (640 x 512) provides a good definition of the morphological features of the face of the pilot, which is critical for the performance of the tracking algorithms.
“We already were familiar with the high quality of FLIR’s airborne camera systems,” said Marco Gazzaniga. “For this project, we definitely appreciated the capability of FLIR Systems to listen to our needs, as well as the company’s flexibility and direct support.”
Cognitive and emotional information
Next2U has been doing a lot of pioneering work with the use of thermal cameras.T he ISMS represents the most advanced tool developed on thermal IR imaging based on computational psychophysiology.
Arcangelo Merla concluded: “The ISMS also uses Functional Near-Infrared Spectroscopy or fNIRS to monitor cognitive brain activity, while thermal imaging is used to estimate the emotional impact. As such, the ISMS manages to monitor both cognitive and emotional processes in real time during flight missions or simulations. This is truly a breakthrough. The technology has become mature for monitoring everyday human machine interactions in automotive or robotics applications.”