Digital technologies and human-machine interaction in air traffic control operations

. The aim of this study is to exploit digital technologies to develop and validate an innovative Human Machine Interface for Air Traffic Control Operations tailored based on actual visibility condition, specific phase of flight and different user working positions. The model for the interface prototype exploits multimodal interaction features and implements technologies and functions such as Extended Reality, aircraft identification and tracking labels, and safety net visualization. Moreover, it includes the future possible scenario of the Air Traffic Management comprising the integration of autonomous drones and Urban Air Mobility.


Introduction
In recent years, innovative and advanced visualization tools for Air Traffic Controllers (ATCOs) such as movement maps, conformance monitoring and conflict detection, have been designed to improve the operational safety of airports.However, the increased number of information required and displayed on the workstation terminals such as Aerodrome Traffic Zone and position and tracking of aircrafts and vehicles, lead to more time in head-down position watching at the screens rather than in a head-up one focusing on the out-of-the-window view with the consequent risk of not detecting unpredictable and potentially dangerous events [1].In addition, continuously switching from one position to another focusing on two different perspectives of the same environment would lead to a reduction in the situational awareness of controllers [2,3,4,5].To address this problem, the idea of using Augmented Reality (AR) in the control tower to overlay auxiliary information over the real external view was introduced almost 30 years ago when this technology was still in the very early stages of its industrialization [6].Over the years, a few experiments with modern hardware have been performed to test the soundness of AR technology in air traffic control towers, confirming its positive potential to benefit control tower operations [7,8,9].

Extended Reality technology in Airport Control Towers Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision
The RETINA-The Resilient Synthetic Vision for Advanced Control Tower Air Navigation Service Provision Single European Sky ATM Research (SESAR) project concept has filled the gap of using the latest advances in Augmented Reality to further investigate the application of synthetic vision tools in a real airport control tower environment (Figure 1) [10].The project consortium has investigated the potential and applicability of Virtual/Augmented Reality (V/AR) technologies for the provision of Air Traffic Control (ATC) service in conventional airport control tower [11,12].RETINA has assessed whether and how the concepts that stand behind tools such as Head-up Displays in the aircraft cockpit, Enhanced Vision Systems and Synthetic Vision Systems could be transferred to ATC with relatively low effort and considerable benefits for controllers' Situational Awareness.In doing so, two different AR systems have been investigated: Conformal-Head-Up Displays (C-HUDs) -which, potentially, can be made to coincide with the tower windows -and See-Through Head-Mounted Displays (ST-HMD).A proof-of-concept of both systems has been implemented and validated by means of human-inthe-loop real-time simulations in a laboratory environment.The external view was provided to the user through a high fidelity 4D model in an immersive environment that replicates the out-of-the tower view and additional information such as airport layout, flight tags, wind velocity and direction and warning detection were placed over the actual out of the window view.RETINA concept, therefore, has enabled the controllers to have a head-up view of the airport traffic in any visibility condition.The outcomes of the project validation, obtained through both subjective qualitative information and objective quantitative data, have proven how this concept could lead to an improvement of the human performance in the control tower, preserving safety and increasing resiliency at airports to low visibility [13].The results obtained by RETINA consortium have been exploited to feed SESAR's Digital Technologies for Tower (PJ05-W2 DTT) project [14,15].

Digital Technologies for Tower
The DTT project aims to contribute to Air Traffic Management (ATM) digitalisation objectives maturing the concept of an Augmented Reality-based interface for Air Traffic Controllers in conventional and remote airport control towers.The project is composed by three different subprojects, each of which is focusing on specific purposes to be validated and progressively matured for the benefit of the Air Traffic Management (ATM) network in terms of safety, capacity, efficiency and flexibility.The first sub-project, Multiple Remote Tower and Remote Tower Centre, proposes the development of a remote aerodrome air traffic service in which services from various aerodromes are combined in a centralised control room independent of airport location.
The second and third sub-projects, ASR at the TWR CWP supported by AI and Machine Learning and Virtual/Augmented Reality applications for Tower, address the development of new human machine interface (HMI) interaction modes and technologies at the Controller Working Position.They deal with both the current operating airport environment and future environments focusing the activities towards two main areas: Automatic speech recognition (ASR) and Virtual and Augmented Reality.ASR supported with Artificial Intelligence/Machine Learning techniques, is investigated to enable the recognition and translation of spoken language into the system reducing their workload and improving safety whilst, V/AR is expected to allow tower ATCOs to conduct safe operations under any meteorological conditions while maintaining a high taxiway and runway throughput.Within this last area, specific aspects of other features, such as Tracking Labels (TL), multimodal interaction and Attention Guidance/Safety Nets, are investigated by three of the different DTT project partners in different simulation scenarios [16,17,18].In particular, one of the validation exercises has been carried out as a real-time human-in-the-loop simulation in the Virtual and Simulation Laboratory of the University of Bologna (Figure 2) utilizing the more technically advanced version of the platform exploited to validate RETINA concept.The exercise has assessed, at different maturity levels, the introduction of an adaptive HMI encompassing different functions, namely, Virtual/Augmented Reality TL and airport layout overlays, multimodal interaction (voice and Air Gestures) to deliver not time critical clearances and Safety Nets visualisation to guide the attention of the operators towards hazardous situations.

Figure 2: University of Bologna simulation and validation platform at the Virtual and Simulation Laboratory in Forlì premises. The ATCOs are provided with a realistic and consistent scenario of the out-of-the-tower view of the aerodrome onto a CAVE-like virtual environment, by wearing the ST-HMD devices the users can simultaneously see the both the AR overlays and the out-of-the-window view (personal view of the specific ATCO in the green square).
As expected, the results of Bologna validation exercises confirmed that the proposed solution of a V/AR HMI interface in conventional control towers can support the ATC operations.The introduction of synthetic overlays stimulates the controllers in working in a head-up position and reducing the number of switching head-up/head-down with a consequent positive impact on human performances and situational awareness.Nevertheless, further improvements can still be considered to provide an even more effective interaction in airport control towers.

Human Machine Interface for Air Traffic Control Operations
Starting form RETINA and DTT technologies, devices and results, and considering the recent advancement in the aeronautical sector, the aim of this study is to exploit digital technologies to develop and validate an innovative Human Machine Interface for Air Traffic Control Operations tailored based on actual visibility condition, specific phase of flight and different user working positions.The interface should enable a more natural and effective interaction in control tower, improving, on one hand, the performance and, on the other hand, the situational awareness of the ATCOs.
The model for the interface prototype exploits multimodal interaction features and implements technologies and functions such as Extended Reality (XR), aircraft identification and tracking labels, safety net visualisation, Conflicting ATC clearances alerts, in-air gesture interaction and speech recognition.Moreover, the future possible scenario of the Air Traffic Management comprising the integration of autonomous drones and Urban Air Mobility (UAM) that requires a closer integration between vehicle and infrastructure capabilities [19] are considered to be included in the proposed project.Within the project, different simulation scenarios integrating ATM and Unmanned Aircraft System Traffic Management (UTM) are planned to be implemented and assessed in a laboratory environment to explore a wide set of possible solutions which are not yet available in the real world.More specifically, the model focuses, on one hand, on the representation of different designs and locations of UAM infrastructures (e.g.vertiports) foreseeing UAM dedicated and integrated airspace and, on the other hand, on the supply to the controllers of the necessary tools to conduct manned and unmanned air traffic control operations.

Conclusions
The proposed research can contribute to lead to a significant benefit for the future aviation system, including, but not limited to, increased safety for passengers, financial savings for carriers and Air Navigation Service Providers, better point-to-point connections and shorter travel times, and improved resilience and efficacy for the control tower IT systems.The development of such interface, indeed, could reduce the workload and increase the productivity and the situational awareness of the ATCOs.Through the use of XR technologies and multimodal interaction, controllers will be provided with high-quality information to operate in any condition of traffic, weather, airport complexity, etc., without endangering safety.Moreover, airports could use the results in planning the infrastructure of the future.As a matter of fact, the integrated traffic forecast given by the increase of autonomous advanced systems, will lead, not only to a whole new batch of actors in the aeronautical panorama, but also to change the perspective and the role of the air traffic controllers.

Figure 1 :
Figure 1: Retina Concept: an ATCO with the ST-HMD visualises the Aircraft Tracking Label through the window of the Control Tower.