Skip to main content
Skip to article control options
Open AccessTechnical Notes

Applicability and Surrogacy of Uncorrelated Airspace Encounter Models at Low Altitudes

Published Online:https://doi.org/10.2514/1.D0254

I. Introduction

The National Airspace System (NAS) is a complex and evolving system that enables safe and efficient aviation. Advanced air mobility concepts and new airspace entrants, such as unmanned aircraft, must integrate into the NAS without degrading overall safety or efficiency. For instance, regulations, standards, and systems are required to mitigate the risk of a midair collision between aircraft. Monte Carlo simulations have been a foundational capability for decades to develop, assess, and certify aircraft conflict avoidance systems. These are often validated through human-in-the-loop experiments and flight testing. For example, an update to the Traffic Collision Avoidance System (TCAS) mandated for manned aircraft was validated in part using this approach [1].

For many aviation safety studies, manned aircraft behavior is represented using the MIT Lincoln Laboratory statistical encounter models [25]. The original models [24] were developed from 2008 to 2013 to support safety simulations for altitudes above 500 feet above ground level (AGL). However, these models were not sufficient to assess the safety of smaller unmanned aerial systems (UAS) operations below 500 feet AGL and fully support the ASTM F38 and RTCA SC-147 standards efforts. In response, newer models [57] with altitude floors below 500 feet AGL have been in development since 2018. Many of the models assume that aircraft behavior is uncorrelated and not dependent on air traffic services or nearby aircraft. The models were trained using observations of cooperative aircraft equipped with transponders, but data sources and assumptions vary. The newer models are organized by aircraft types of fixed-wing multi-engine, fixed-wing single engine, and rotorcraft, whereas the original models do not consider aircraft type.

Our research objective was to compare the various uncorrelated models of conventional aircraft and identify how the models differ. Particularly if models of rotorcraft were sufficiently different from models of fixed-wing aircraft to require type-specific models. The scope of this work was limited to altitudes below 5000 feet AGL, the expected altitude ceiling for many new airspace entrants. The scope was also informed by the Federal Aviation Administration (FAA) UAS Integration Office and Alliance for System Safety of UAS through Research Excellence (ASSURE). The primary contribution is guidance on which uncorrelated models to leverage when evaluating the performance of a collision avoidance system designed for low-altitude operations, such as prescribed by the ASTM F3442 detect and avoid standard for smaller UAS [8]. We also address which models can be surrogates for non-cooperative aircraft without transponders. All models and software used are publicly available under open source licenses [9].

II. Overview of the Encounter Models

Each encounter model is a set of two Bayesian networks, generative statistical models that mathematically represent aircraft behavior during close or safety critical encounters, such as near midair collisions. The initial static Bayesian network represents the aircraft’s initial position and dynamic state, whereas the transition network is a dynamic Bayesian network that represents state changes over time. In general, the initial network models initial aircraft position and kinematic state, whereas the transition network models subsequent aircraft flight in statistically realistic ways.

The original uncorrelated models from 2008 to 2013 were trained using observations collected by ground-based secondary surveillance radars from the 84th Radar Evaluation Squadron (RADES) network [24], whereas recent models have been based on crowdsourced observations of Automatic Dependent Surveillance–Broadcast (ADS-B)–equipped aircraft by the OpenSky Network [57]. Other models have also been trained based on different assumptions or data [1013], but are not discussed here.

RADES data are filtered using the Mode 3A/C code squawked by an aircraft’s transponder, which includes a nonunique identification code and a discretized MSL altitude. A Mode 3A code of 1200 indicates that the aircraft is operating under visual flight rules. In addition to Mode 3A/C, OpenSky Network ADS-B data also include an ICAO 24-bit address, a unique global identifier [14]. This address is correlated with national aircraft registries to identify aircraft type (e.g., fixed-wing multi-engine, rotorcraft, etc.) [6]. The OpenSky Network–based models are organized based on aircraft type, whereas the RADES-based model is trained using multiple aircraft types. Each aircraft type was organized into two models: the first trained with tracks associated with aircraft squawking 1200 only and the second trained with tracks not squawking 1200.

The quantity and quality of training data are dependent on the coverage and characteristics of the surveillance system. Particularly, the low-altitude coverage of the RADES and OpenSky networks widely varies across periods of time and location. The encounter models do not provide information on the density or frequency of aircraft behavior per unit time or a specific geographic region; other technologies address this [15].

Additionally, due to position uncertainties associated with ground-based radar time of arrival measurements, the RADES-based model has an altitude floor of 500 feet AGL. Conversely, ADS-B enables aircraft to broadcast the aircraft’s estimate of their own location, which is often based on precise GNSS measurements, enabling OpenSky Network–based models to have an altitude floor of 50 feet AGL.

Table 1 summarizes the different uncorrelated models. With the exception of the lowest altitude bin, the uncorrelated models have the same structure and variable discretization. The model variables are geographic domain, airspace class, altitude layer, velocity, acceleration, vertical rate, and turn rate. The first three variables model aircraft location. Geographic domain denotes operations over the USA mainland, islands, or offshore environments. Airspace class denotes operations in the FAA Class B, C, and D regulatory airspaces, or “other” airspace consisting of Class E and G airspaces; the airspace rules on aircraft behavior vary by class. The other variables correspond to assumed modeled aircraft kinematics. Refer to previous documentation [25] for details.

III. Analysis

The analysis consisted of four components. First, a brief literature review focused on rotorcraft, as previous research focused on reviews of fixed-wing aircraft. Next was a comparison of the distributions for each initial network variable to identify evident differences between models. The next two analyses were based on the sampled joint distributions of pairs of initial network variables and distributions of kinematic variables sampled from the transition network. For these two analyses, one million trajectories were generated from each model using basic rejection sampling based on altitude and airspeed [4,9]. When sampling a given bin, we assume a uniform distribution within the bin. Each trajectory was 180 s long with 1 s updates.

A. Literature Review

While rotorcraft have unique capabilities, such as the ability to hover, it is important to characterize how often these unique capabilities are used. As part of a pilot survey before prototyping a collision avoidance system on helicopters, Taylor and Adams [16] asserted that for most phases of flight, the flight profiles between a helicopter and general aviation aircraft were indistinguishable. This was due to passenger comfort and standard turn and vertical maneuvering rates. When testing a collision avoidance prototype equipped on helicopters, Harman et al. [17] and Andrews [18] discussed that civil helicopters and fixed-wing aircraft have different fuselage shapes and that helicopters generally fly slower with greater turn rates, lower to the ground, and closer to buildings. They did not assert that the flight dynamics between the aircraft types were significantly different. Matthews and Sawyer [19] noted that for terminal operations under Instrument Flight Rules (IFR), rotorcraft tend to have slower speeds and greater turn rates than fixed-wing aircraft. Because of numerous surveillance, algorithm, and policy challenges, collision avoidance systems for rotorcraft were not mandated in the 1990s. Industry-wide research into collision avoidance systems for rotorcraft subsequently waned until recently.

B. Initial Networks

First, we identified evident differences between some initial networks. Because of similar coverage bounds, the geographic domain distribution was extremely similar across all models. The distributions for the kinematic variables (acceleration, vertical rate, turn rate) were all indicative of a preference for straight and level flight. We assessed individually the models in two groups, the first group for 1200-code-only models and the second for 1200-code excluded, with the results provided in Figs. 1 and 2, respectively. An inspection of airspace class, altitude, and speed, however, revealed some significant model differences in Figs. 1 and 2. We emphasize that the model assumes inter-bin uniform distributions. For example, when training the model, two altitude observations of 600 and 1000 feet AGL would both be discretized to [500, 1200) feet AGL. When sampling that bin, both values should equally be likely.

Fig. 1
Fig. 1

Airspace, altitude, and velocity initial network distributions of 1200-code-only encounter models.

Fig. 2
Fig. 2

Airspace, altitude, and velocity initial network distributions of 1200-code-excluded encounter models.

The initial networks aligned with many of the historical assumptions. Independent of the Mode 3/AC code, rotorcraft generally fly slower and lower. There were also limited modeled instances of rotorcraft flying at 30 knots or less. The 1200-code-only and 1200-code-excluded rotorcraft models were very similar. Both fixed-wing multi-engine models skewed toward higher altitudes and faster airspaces, while sharing limited similarities with the other models. The airspace class distribution was dependent on Mode 3/AC code, with a greater relative percentage of observations in terminal airspace for the 1200-code-excluded models.

Conversely, there were significant differences between the 1200-code-only and 1200-code-excluded fixed-wing multi-engine models, with the latter trained with substantially more observations in Class B airspace and speeds of 165 knots or greater. Compared with other aircraft types, fixed-wing multi-engine aircraft were also more likely to accelerate at a magnitude greater than 0.3 knots per second. This is not surprising, as large transport aircraft have multiple engines and squawk discrete code.

Also note the similarities between the 1200-code-only RADES and fixed-wing single-engine models. For all potential pairs of models, the distribution shapes and magnitudes were the most similar. These models differed from the 1200-code excluded fixed-wing single-engine model, which had an altitude distribution skewed toward higher altitudes with slightly faster velocities

C. Pairwise Overlap

The previous analysis assessed only the independent variable distributions. Although informative for high-level trends, it was insufficient to characterize the models as it did not account for the Bayesian relationship between variables. This analysis considers pairs of variables. We calculated model overlap as the frequency that trajectories generated from different models were sampled from analogous or the same variable bins. An overlap of 85% and greater is indicative of significant similarities between the sampled trajectories, whereas 50% or less is indicative of poor overlap. We heuristically selected these thresholds, with 85% assumed to be significant overlap although accounting for minor model differences. Table 2 reports the overlap for notable variable combinations between the 1200-code-only RADES and rotorcraft models. All variable pairs were assessed but for conciseness; only notable combinations are discussed here.

For example, Table 2 reports that 50% of sampled trajectories from the 1200-code-only RADES model and both rotorcraft models were initialized in the same airspace class and same altitude layer bin. The lack of overlap was expected given the different altitude distributions in Fig. 1. More than 50% of rotorcraft observations were at [500, 1200) feet AGL, compared with less than 25% for all other models. As velocity is dependent on altitude, a lack of an overlap is observed again at 50%. However, there is better overlap when just considering the kinematics, with overlaps often greater than 85%. These results also align with the historical assumptions that rotorcraft fly similar to fixed-wing aircraft.

Likewise, trends from the previous analysis were also identified here. Fixed-wing multi-engine trajectories had little to no overlap with the RADES trajectories, with majority of pairwise combinations having less than 50% overlap. Conversely, there was consistent overlap between the fixed-wing single-engine and RADES-based trajectories. The 1200-code-only fixed-wing single-engine model has very good overlap, but even when blending the two fixed-wing single-engine models together, the minimum overlap with the RADES trajectories was 74% with many combinations greater than 85%. Comparing the 1200-code-excluded rotorcraft trajectories to the RADES or fixed-wing single-engine trajectories also drew a comparable conclusion of better overlap for kinematics than location.

D. Transition Network

The previous analysis focused on aggregate trends on how trajectories were initialized. It did not characterize how an aircraft behaved over a complete trajectory as the transition network was sampled each time step to update acceleration, vertical rate, and turn rate. We calculated the overlap between trajectories again and drew similar conclusions. For 1200-code-only RADES and rotorcraft sampled trajectories, there was 88, 93, and 95% overlap for acceleration, vertical rate, and turn rate. Compared against all models, multi-engine aircraft had greater accelerations and vertical rates more frequently. All models had greater than 90% overlap for turn rate. This was likely due to pilots consistently employing standard rate turns.

Fixed-wing single engine and the RADES trajectories were again consistently the most similar. Fittingly, a blended model of fixed-wing single engine had overlaps of 90, 98, and 99%, with both models independently also having significant overlap.

Interestingly even at slow airspeeds of 60 knots or less, there was significant overlap of greater than 90% for acceleration, vertical rate, and turn rate when comparing fixed-wing single engine and rotorcraft. This strongly supports the historical assumption that although rotorcraft can fly slower and lower than fixed-wing single engine, the underlying kinematics are similar.

IV. Guidance for Model Use

An inspection of both the initial and transition network differences indicated that the models are representative of a variety of flight operations. The relative distribution of where aircraft were instantiated, as represented by geographic domain, airspace class, and altitude, differed between rotorcraft and fixed-wing single engine. Notably, rotorcraft were more likely to be instantiated at lower altitudes. However, the physical dynamics representing how aircraft fly, as modeled by speed, acceleration, vertical rate, and turn-rate were similar across models, with stronger similarities at faster speeds. Furthermore, although aircraft may maneuver in a similar manner, the frequency and magnitude of maneuvers may differ.

A. Surrogacy for Non-Cooperative Aircraft

For conventional fixed-wing aircraft, the 1200-code-only RADES model was previously determined to be surrogate for many non-cooperative aircraft not equipped with any transponder [3]. This assumption was based on comparing the statistical model with a classifier trained using primary radar tracks with no transponder information. An identical analysis was not possible when training the OpenSky Network–based models due to concerns about using a classifier trained on one data source and applying it to another data source.

Across all analyses, there were consistent similarities between of the 1200-code-only model trained using RADES data and fixed-wing single-engine aircraft models trained using observations from the OpenSky Network. This strongly suggests that the behavior of fixed-wing aircraft is not dependent on whether the aircraft was equipped with ADS-B and observed by the OpenSky Network or if the aircraft was observed by a ground-based RADES radar. Given these similarities, we assume that the 1200-code-only fixed-wing single-engine model, trained with observations from the OpenSky Network, can represent simulated non-cooperative fixed-wing single-engine aircraft.

There was also strong evidence that fixed-wing single-engine and rotorcraft aircraft employ similar, but not identical, accelerations, vertical rates, and turn rates. Although the initial networks for rotorcraft are different, there is insufficient evidence that a rotorcraft squawking 1200 has significantly different kinematics than a fixed-wing single engine. Thus, we also assume that the 1200-code-only rotorcraft model can represent non-cooperative rotorcraft.

Lastly, although the initial networks of the 1200-code-only RADES and fixed-wing multi-engine models have similar shapes, the multi-engine models consistently generated tracks for aircraft that accelerated, climbed, and descended faster. These greater rates are not surprising when assuming that the RADES model was trained primarily based on less powerful single-engine aircraft. The analysis of initial networks also illustrated that fixed-wing multi-engine aircraft squawking 1200 behave significantly different from others. Given a reasonable hypothesis for the kinematic differences between models and the behavioral dependence on squawking 1200, we assume that the 1200-code-only fixed-wing multi-engine model can represent non-cooperative fixed-wing multi-engine aircraft. However, this assumption is made with less confidence than for the other aircraft types and subject to future refinement.

B. Applicability

We previously assumed that the RADES model was trained primarily on observations of fixed-wing single engine with some minor mix of other aircraft types, but could not validate this assumption due to insufficient information provided by the Mode 3A/C transponders. Our analyses support this assumption because the 1200-code-only model trained on RADES observations and the fixed-wing single-engine models trained based on the OpenSky Network were consistently very similar. Although these specific models were similar, we found that each of the aircraft type models trained using OpenSky Network data exhibited different behavior and is representative of particular types of traffic. Within the training data, rotorcraft were more likely to be flying lower and slower, whereas fixed-wing multi-engine were more likely to be flying higher and faster. There was significant evidence that fixed-wing aircraft, either single- or multi-engine, behaved different if squawking 1200 code or not.

Accordingly, for safety simulations that require assessments for cooperative and non-cooperative aircraft, the models are not interchangeable. For example, a simulation that only leverages the 1200-code-only RADES model would not fully represent the low-altitude environment and would likely be insufficient. Additionally, only simulating with both rotorcraft models would also be insufficient, as fixed-wing aircraft have a different distribution of speeds; the underlying kinematics would also be similar, but sufficiently different.

Thus, for safety simulations involving low-altitude non-cooperative aircraft, only the 1200-code-only models should be used. For simulating cooperative aircraft, the 1200-code-excluded models should be used and then supplemented by the 1200-code-only models. Specifically for evaluations related to ASTM F3442 [8], all of the OpenSky Network–based aircraft specific models should be leveraged and use accordingly for assessing performance with cooperative and non-cooperative manned aircraft. When sampling the models, at least the [50, 500) and [500, 1200) feet AGL altitude layers should be considered. An extensive use of all the models, along with uniform sampling within bins, supports a robust evaluation of whether a system satisfies the ASTM F3442 or similar standard. For each unique model, the appropriate level of performance requirements should be satisfied. In the absence of traffic density and surveillance coverage information, it is difficult to justify weighting one model over another.

V. Conclusions

The objective of this study to assess the similarities and differences between encounter models was satisfied. The assumptions that the models adequately represent rotorcraft and that fixed-wing single-engine flights enable safety simulations for altitudes below 500 feet AGL were validated. The similarities between 1200-code-only RADES and fixed-wing single-engine models also support that previous safety analyses were representative and valid.

This study determined that multiple models based on aircraft type are warranted and that model behavior can be dependent if an aircraft is squawking 1200-code. A consequence of this research is that multiple models should be used when assessing safety systems at lower altitudes. The recommendation for multiple models differs from assessments before 2018, where the singular RADES 1200-code model was solely leveraged for most simulations.

C. WankeAssociate Editor

Acknowledgments

We greatly appreciate the support and assistance provided by Sabrina Saunders-Hodge and Adam Hendrickson from the Federal Aviation Administration UAS Airspace Integration Office under Air Force Contract No. FA8702-15-D-0001. We also like to thank our colleagues Jim Kuchar and Rodney Cole.

References

  • [1] Espindle L. P., Griffith J. D. and Kuchar J. K., “Safety Analysis of Upgrading to TCAS Version 7.1 Using the 2008 U.S. Correlated Encounter Model,” Massachusetts Inst. of Technology, Lincoln Lab. Project Rept. ATC-349, Lexington, MA, 2009. Google Scholar

  • [2] Kochenderfer M. J., Edwards M. W. M., Espindle L. P., Kuchar J. K. and Griffith J. D., “Airspace Encounter Models for Estimating Collision Risk,” Journal of Guidance, Control, and Dynamics, Vol. 33, No. 2, April 2010, pp. 487–499. https://doi.org/10.2514/1.44867 LinkGoogle Scholar

  • [3] Kochenderfer M. J., Kuchar J. K., Espindle L. P. and Griffith J. D., “Uncorrelated Encounter Model of the National Airspace System version 1.0,” Massachusetts Inst. of Technology, Lincoln Lab. Project Rept. ATC-345, Lexington, MA, 2008. Google Scholar

  • [4] Weinert A. J., Harkleroad E. P., Griffith J. D., Edwards M. W. and Kochenderfer M. J., “Uncorrelated Encounter Model of the National Airspace System,” Ver. 2.0, Massachusetts Inst. of Technology, Lincoln Lab. Project Rept. ATC-404, Lexington, MA, Aug. 2013 [online], https://apps.dtic.mil/docs/citations/ADA589697. Google Scholar

  • [5] Weinert A., Underhill N. and Wicks A., “Developing a Low Altitude Manned Encounter Model Using ADS-B Observations,” 2019 IEEE Aerospace Conference, IEEE, New York, March 2019, pp. 1–8. https://doi.org/10.1109/AERO.2019.8741848 Google Scholar

  • [6] Weinert A., Underhill N., Gill B. and Wicks A., “Processing of Crowdsourced Observations of Aircraft in a High Performance Computing Environment,” 2020 IEEE High Performance Extreme Computing Conference (HPEC), IEEE, New York, Sept. 2020, pp. 1–6. https://doi.org/10.1109/HPEC43674.2020.9286229 Google Scholar

  • [7] Schäfer M., Strohmeier M., Lenders V., Martinovic I. and Wilhelm M., “Bringing Up OpenSky: A Large-Scale ADS-B Sensor Network for Research,” IPSN-14 Proceedings of the 13th International Symposium on Information Processing in Sensor Networks, IEEE, New York, April 2014, pp. 83–94. https://doi.org/10.1109/IPSN.2014.6846743 Google Scholar

  • [8] F3442/F3442M-20 Standard Specification for Detect and Avoid System Performance Requirements, ASTM International, West Conshohocken, PA, 2020 [online]. https://doi.org/10.1520/F3442_F3442M-20 Google Scholar

  • [9] Weinert A., Kochenderfer M. and Edwards M. W. M., “Airspace-Encounter-Models/Em-Model-Manned-Bayes: March 2021—OpenSky Updates,” March 2021. https://doi.org/10.5281/zenodo.4573957 Google Scholar

  • [10] Edwards M. W., Kochendedrfer M. J., Kuchar J. K. and Espindle L. P., “Encounter Models for Unconventional Aircraft, Version 1.0,” Massachusetts Inst. of Technology, Lincoln Lab. Project Rept. ATC-348, Lexington, MA, 2009. Google Scholar

  • [11] Griffith J. D., Edwards M. W., Miraflor R. M. and Weinert A. J., “Due Regard Encounter Model,” Ver. 1.0, Massachusetts Inst. of Technology, Lincoln Lab. Project Rept. ATC-397, Lexington, MA, Aug. 2013. Google Scholar

  • [12] Mueller E. and Kochenderfer M. J., “Simulation Comparison of Collision Avoidance Algorithms for Small Multi-Rotor Aircraft,” AIAA Modeling and Simulation Technologies Conference, AIAA Paper 2016-3674, June 2016. https://doi.org/10.2514/6.2016-3674 Google Scholar

  • [13] Underhill N., Harkleroad E., Guendel R., Weinert A., Maki D. and Edwards M., “Correlated Encounter Model for Cooperative Aircraft in the National Airspace System,” Ver. 2.0, Massachusetts Inst. of Technology Lincoln Lab. Project Rept. ATC-440, Lexington, MA, May 2018 [online], https://apps.dtic.mil/docs/citations/AD1051496 [retrieved 16 Jan. 2019]. Google Scholar

  • [14] Trim R. M., “Mode S: An Introduction and Overview (Secondary Surveillance Radar),” Electronics Communication Engineering Journal, Vol. 2, No. 2, April 1990, pp. 53–59. https://doi.org/10.1049/ecej:19900017 Google Scholar

  • [15] Serres C. and Edwards M., “Mit-ll/Traffic-Density-Database: Initial Release,” MIT Lincoln Lab., Dec. 2020. https://doi.org/10.5281/zenodo.4323626 Google Scholar

  • [16] Taylor F. and Adams R., “Helicopter User Survey: Traffic Alert Collision Avoidance System (TCAS),” Systems Control Technology for the Federal Aviation Administration Final Rept. DOT-FAA-PM-85-6, Arlington, VA, April 1985 [online], https://apps.dtic.mil/dtic/tr/fulltext/u2/a155415.pdf. Google Scholar

  • [17] Harman W. H., Welch J. D. and Wood M. L., “Traffic Alert and Collision Avoidance System (TCAS) Surveillance Performance in Helicopters,” Massachusetts Inst. of Technology Lincoln Lab. TR ATC-135, Lexington, MA, May 1987. Google Scholar

  • [18] Andrew J. W., “Pilot Evaluation of TCAS in the Long Ranger Helicopter,” Massachusetts Inst. of Technology Lincoln Lab. Project Rept. ATC-136, Lexington, MA, June 1986. Google Scholar

  • [19] Matthews R. and Sawyer B., “Rotorcraft Terminal ATC Route Standards,” Systems Control Technology, Inc. TR DOT/FAA/RD-90/18, Arlington, VA, Aug. 1991. Google Scholar

Tables

Table 1 List of analyzed encounter models

Aircraft typeSourceSquawk criteriaAltitude floor
UnknownRADES1200-code only500 feet AGL
Fixed-wing multi-engineOpenSky Network1200-code only50 feet AGL
Fixed-wing single engineOpenSky Network1200-code only50 feet AGL
RotorcraftOpenSky Network1200-code only50 feet AGL
Fixed-wing multi-engineOpenSky Network1200-code excluded50 feet AGL
Fixed-wing single engineOpenSky Network1200-code excluded50 feet AGL
RotorcraftOpenSky Network1200-code excluded50 feet AGL

Table 2 Overlap between 1200-code-only RADES and rotorcraft sampled trajectories

Dependent variableIndependent variableOverlap percentageNote
AltitudeGeographic domain90Location
AltitudeAirspace class50Location
VelocityAirspace class76Location/kinematic
AccelerationAirspace class93Location/kinematic
Vertical rateAirspace class90Location/kinematic
Turn rateAirspace class90Location/kinematic
VelocityAltitude50Location/kinematic
AccelerationVelocity78Kinematics
Vertical rateAcceleration88Kinematics
Turn rateAcceleration86Kinematics
Turn rateVertical rate93Kinematics