ADS-B driven implementation of an augmented reality airport control tower platform

. This paper describes a real-world implementation of the solutions developed within the SESAR DTT Solution 97.1-EXE-002 project, which tested in a simulated scenario the use of Augmented Reality (AR) to assist the airport control tower operators (TWR). Following a user-centred design methodology, the requirements of a real-world live AR platform join with design concepts validated in previous projects, namely the Tracking Labels, the weather interface, and a low-visibility overlay, all used to increase the TWR situational awareness, performance and reactivity while reducing the workload. The designed AR platform performs the live tracking and visualization of real aircraft and surveillance information in the airport traffic zone. It bases on three key processes: the transmission of an ADS-B data flow to a Microsoft™ HoloLens2, the registration process of the AR platform, and the rendering of a real-time tracking system and other surveillance overlays. The concept has been first validated with the help of a TWR, preceding a technical validation to ensure the repeatability and reproducibility of the results. The results allow for defining new guidelines for the deployment in a control tower environment.


Introduction
Being an airport control tower operator (TWR) implies overseeing the aircraft in the airport manoeuvring area and departing and arriving traffic, relying on the out-of-the-window (OTW) tower's view to provide separation and clearances.In this high-risk, high-concentration and timecritical job, controllers are concerned with keeping a smooth traffic flow while ensuring the overall safety of airport operations, with the sum of these tasks resulting in a heavy workload.
While performances are a priority in a worldwide ever-growing traffic scenario, the main upgrades in TWRs' job were due to the addition of visual interfaces, which are better suited to ensure an increase in safety, with a collateral increase in cognitive workload due also to the continuous shift of focus between the outside view (head-up position) and the head-down interfaces at the work position providing surveillance and traffic information [1].
In this scenario, Augmented/Mixed Reality (AR) is suitable for helping the TWR [2], moving the surveillance information from the head-down interfaces to a collinear vision within the OTW airport traffic, promising to solve the chronic safety over performance compromise.At the University of Bologna, Solution 97.1-EXE-002 of the SESAR's funded "Digital Technologies for Tower" (DTT) project [3,4,5] tested this possibility, using a system of virtual tracking labels to pinpoint each aircraft in the user's view, providing context-related flight information, tailored to different control roles (ground vs runway) and environmental conditions.TWRs could then keep the focus on the live traffic, with increased situational awareness and reduced workload, while improving the overall safety and efficiency (temporal and thus economical) of airport operations in every condition, especially high-traffic and low-visibility.
These studies were conducted in a simulated real-time airport scenario, focusing on developing the overall AR concept in a safe, non-critical, and fully controllable environment.This paper describes the subsequent implementation of the developed concepts into the real world, resulting in a live application tracking operative aircraft and acquiring their surveillance information.The research addressed the tasks needed when dealing with a real-time physical world application.A new set of requirements have been defined, dealing with a global-coordinates registration process for AR and using live ADS-B data to retrieve the desired traffic information, as will be later described.

Methods
User-centred design (UCD) method is the obvious choice when dealing with a tricky task such as airport tower control.The design process is iterative, always considering the user needs before, during and after the development, ensuring that the final concept helps the TWR without relevant contraindications.This research aimed to increase the maturity of the solutions developed in a simulated environment, performing the critical steps to move up from a previous level 4 of the technology readiness scale, by transferring the concept into a relevant environment.For this purpose, some requirements were identified, joining outcomes of the lab validation campaign with new requirements for a real-world implementation.Table 1 summarizes all the requirements to implement and validate a real-time AR platform for the control tower.

Design of the AR-based control tower platform
Starting from the requirements, a preliminary concept has been defined, using an ADS-B receiver to detect the aircraft in real-time and retransmitting the data stream to a HoloLens2 device using a user datagram protocol (UDP).The AR application in the device processes the data, identifying aircraft position and rendering the tracking labels.Figure 1 shows the architecture of the platform.

Figure 1. Architecture of the AR platform
ADS-B data flow.The vector state information is acquired from ADS-B data, which is constantly sent in real-time by every commercial aircraft.These data are read on a pc unit over serial communication and retransmitted over a UDP to the HoloLens, where they are used to constantly track the aircraft in the virtual world and provide surveillance information.

Calibration.
A key aspect of AR applications is the registration process, meaning the continuous alignment of the virtual overlay over the intended elements in the physical world.Most AR technologies rely on a tracking system which can identify the user's position inside an environment and project the holographic overlay accordingly, with a real-time adaptation of the virtual elements' alignment over the physical world.This procedure is suited to position holograms with respect to a local frame of reference.Since aircraft positions are in global (geodetic) coordinates, a calibration procedure is required for the registration to occur.HoloLens2's applications can only fix the position of a virtual reference frame inside a room.Thus, a procedure was developed to identify the global position and orientation of this reference frame and then convert the relative position of aircraft with respect to the reference frame into cartesian coordinates using a conversion algorithm from geodetic coordinates to an East-North-Up (ENU) frame.Figure 2 shows the alignment of the reference frames and how the HoloLens 2 reference frame was positioned inside the environment.The virtual reference frame geodetic position is identified, and then the aircraft position is converted for an ENU frame with an origin coinciding with the virtual one.Finally, a rotation is applied to convert ENU coordinates into those of the virtual frame.

Figure 2. Calibration steps
Correction of position information.Some corrections had to be applied to the aircraft position data to obtain an adequate matching of the holograms.In particular, the altitude data comes from the pressure altimeter, with the uncorrected barometric altitude (HQNE) -computed by the onboard barometer with respect to the standard sea-level value of 1013.25hPa-transmitted for collision avoidance purposes.The near-ground effect is also an issue at landings.Thus, the altitude was corrected considering the current sea-level pressure at the airport location (pQNH), and a corrective term for landing gear height and ground effect (pLG).The correction formula reports the height over ground (HQFE) as the uncorrected altitude plus a corrective term for the different reference pressure, times a conversion term for the variation of pressure with altitude: Labels implementation.The tracking label design was retrieved from that already validated in the EXE-002 of Solution 97.1.The labels contain the aircraft height over ground (HQFE), velocity, heading and callsign, with other information already computable or available, such as the distance from any point of the airport, the vertical rate, and HQNE.The labels are positioned at a constant focal distance from the user, with a small sphere pinpointing the aircraft and a surmounting canvas containing the aircraft data.Colour coding is used for landing and departing traffic.
Real-time labels management.The whole purpose of the application is to track each detected aircraft keeping its position and information updated in real-time.The data processing chain starts with the ADS-B data being sent to the AR device as soon as they are detected.On the HoloLens2, the data are decoded and organized into a database containing the most updated information for each aircraft, which is then used to calculate the holograms' position, update the label content, and render the holograms in a seamless cyclic loop run at least at 60Hz, which is the rendering frame frequency.Being HoloLens2 capable of tracking the user's position relative to the virtual reference frame, the holograms stay in place while both aircraft and users move.
Additional overlays.The application was completed with additional overlays helping the controller, in particular a weather interface, using real data from METAR, and a runway overlay for low-visibility conditions, triggered by the visibility distance indicated by the METAR.

Conclusion
This paper presented an AR control tower application working in a real-world scenario.Real-time data were used, and the matching of real and virtual worlds was addressed.
Future development.Starting from this concept, it will be possible to improve the application by testing it in shadow mode in a control tower, while integrating other surveillance information and solutions coming from the SESAR programme.
Solutions developed in previous projectsRequirements for real-world implementation Use of Microsoft HoloLens2 as AR device Real-time acquisition of accurate aircraft position AR application developed with Unity3D Accurate real-time tracking of aircraft in AR layer Registration strategy using MRTK WLT Management of multiple aircraft at the same time Design of the tracking labels Render tracking holograms minimizing latency Definition of a runway overlay Evaluate and counteract possible error sources Design of the weather interface Ensure reliability and precision of tracking system Use of adaptive human-machine interface Implement additional surveillance overlays Ergonomic aspects Validate

Figure 3 .
Figure 3. Tracking Labels visualization with runway overlay, colour coding, and weather interfaceTechnical validation.As part of the UCD, a technical validation was needed to ensure the goodness of the design.After that, the fulfilment of all the requirements was verified in different environments to verify the repeatability and reproducibility of the configuration.ConclusionThis paper presented an AR control tower application working in a real-world scenario.Real-time data were used, and the matching of real and virtual worlds was addressed.Future development.Starting from this concept, it will be possible to improve the application by testing it in shadow mode in a control tower, while integrating other surveillance information and solutions coming from the SESAR programme.References[1] Pinska, E., "An investigation of the head-up time at tower and ground control positions", Eurocontrol (2006).