Introduction
When we talk about the advancement in systems automation, whether it is a smart phone, computer system or an aircraft system, the rate of change is so quick that any failure or problem in these sophisticated systems can take a long time to resolve..
During an aviation occurrence, Investigators are involved to identify the root cause when a failure occurs in an aircraft that had an impact on the safety of passengers or the operation of the aircraft.
To identify the root cause of problems or failures of complex systems that sometimes are interdependent and may function simultaneously investigators must have a good understanding of the characteristics of such systems to enable the identification of the root causes of problems. Modern techniques and tools are being developed to support training in investigation of complex systems.
The aviation industry is utilizing augmented reality, virtual reality and artificial intelligence for systems failure analysis and for training proficiency skills in various sectors such as Air Traffic Services and Management, Aircraft Simulator training, flight crew safety training and training in aircraft maintenance. The sophisticated training methods are not limited to these areas.
Virtual Reality and Augmented Reality
Virtual Reality (VR) technology creates an environment in which the user feels and senses that they are moving inside a computer-created virtual world in the same way that people move inside the natural environment; while immersed in the virtual world, the user cannot perceive the real world which still surrounds him.
Virtual reality is currently used for aircraft cockpit, cabin and ground handling safety training. This technology can be utilized for aircraft occurrence investigations training and to convert aspects of historical accidents into virtual reality to illustrate lessons learned to investigators.
For example, the TWA Flight 800 accident involved a Boeing 747-100 that exploded and crashed into the Atlantic Ocean on 17th July 1996. The reconstruction of the aircraft structure and components from the accident wreckage was done with intense detail. If such data and visual imagery is converted into a virtual reality training module investigators can utilize the lessons learned and also use such accident data as a reference for different aircraft types involved in major accidents.
Augmented Reality (AR) allows the user to see the real world, augmenting it with superimposed virtual objects. In other words, while VR replaces reality, AR supplements it, creating an environment in which real and virtual objects harmonically coexist. AR exploits users’ perceptual-motor skills in the real world, creating a special type of human-machine interaction.
Augmented Reality mixes virtual and actual reality, making available to the user new tools to ensure efficiency in the transfer of knowledge for several processes, and in several environments.
If investigators are equipped with technology such as AR during accident or serious incident investigations, such tools can be used for identification of aircraft parts or aircraft systems. In a situation where aircraft parts and debris are scattered AR technology can be used to identify parts and it can also be used to search the aircraft database using the AR system to recover the part number.
For Example: The Air France AFR4920 accident at Roissy, Charles de Gaulle Airport, on 25th July 2000, the aircraft crashed shortly after take-off as the fuel tank ruptured due to tire failure caused by debris on the runway. The wreckage was spread over a large area. AR technology can be utilized during crash site investigation or on collected wreckage in the hanger.
How does it work?
The operation of both VR and AR systems involves four stages; User Interface, Inference Engine, Knowledge Base (rules) and Working Memory (Facts).
The user, or operator, interacts with the machine through a User Interface by providing information about a particular problem to be solved. Based on the rules in the Knowledge Base the Inference Engine gives commands to the Working Memory to fetch the problem-specific data, and it then sends information back to the operator by way of the User Interface.
Augmented Reality mechanisms are used in the User Interface to enhance the system’s capability. The mobility of the system is achieved by using light and portable devices.
Augmented Reality in Aircraft Maintenance or Inspections
Virtual Reality mixed with actual reality is called Augmented Reality. It provides users with new tools to execute complex operations like aircraft maintenance efficiently. There are proposals by researchers for improvements in existing operations and there are flaws that undermine its implementation. Aircraft maintenance personnel can utilize this technology as an additional aid in rapid identification of locations and elements currently being serviced or repaired.
Lockheed-Martin›s application is used in the F-35 and F-22 fighter programs for final fighter assessment and repair. An inspector looking at the fighter through the glasses sees part numbers and plans projected over the physical aircraft. The inspector then uses a handheld device to enter any defects or repairs. This technology is replacing the checklist and clipboard method with the inspectors walking around the aircraft and logging the areas for repair., This reduces the time, effort and errors since the maintenance personnel access the same system to identify parts, locations and procedures.
Depending on the state of the post-accident wreckage this technology could be utilized by aircraft accident investigators at the accident site to identify aircraft parts, provide the history of the maintenance records and also details of the functions of specific parts during the operation of an aircraft.
Augmented Reality for Air Traffic Controllers
Air Traffic Controller operations are challenging during low-visibility conditions. The visual situational awareness of Controllers can be impaired, leading to a reduction in efficiency. As part of the Single European Sky ATM Research (SESAR) program a European company conducted research called Resilient Synthetic Vision for Advance Control Tower Air Navigation Service Provision (RETINA). The concept was demonstrated to the potential stakeholders in January 2018.
Using synthetic vision and augmented reality technologies, RETINA has developed goggles through which the controllers can see synthetic information overlaid on the actual “out-of-the-window” view.
Using the goggles, the controllers can have a head-up view of the airport traffic, with the aircraft call sign and type, supplemented by additional information, such as wind velocity and direction, airport layout and runway status superimposed on the view, even during low-visibility procedures.
There are many two and three dimensional software ATC Virtual reality applications available that provide Tower Simulation to facilitate Air Traffic Control monitoring. Advanced ATC Training for example, developed a 3D Tower Simulator for training purposes with a complete control console of features, including radio, intercom, meteorological and NAVAID simulations. Emergency procedures training, often dangerous and impractical to do in real life, is easily performed in 3D simulators. This package could lead to creating new methods of ATC data presentation, exploiting 3D technology using real-time airport database visualization.
Mixed Reality Proto-space
NASA’s Jet Propulsion Laboratory (JPL) and the SpaceX company are in the process of developing an augmented reality application utilizing Microsoft HoloLens AR headsets to assist JPL engineers, in a virtual world context, in the construction of a spacecraft for future Mars missions. The innovators develop their creations in the virtual world before it is produced in reality in factories or production plants.
The Proto-space project is presented to scientists with a virtual model of the Mars rover. The virtual model can be interacted with in full scale for size and construction details in ways which cannot be achieved on a 2D computer screen.
The scientists and engineers can interact with the model by walking around the rover, accessing the interior and opening virtual panels to closely inspect the internal parts. These types of virtual reality mockups help engineers to fill the gaps and find hidden problems which would not be possible using traditional design tools.
The physical construction phase of traditional designs may differ in dimensions from the design blue prints and potentially slow down production progress and can cost significant amounts due to delays in project completion. AR produces an entire product design and construction process and product virtually, avoiding such losses.
The mainstream adoption of technology such as virtual reality, augmented reality and mixed reality for innovators and enthusiasts is just beginning and the future of such technology is very promising.
Conclusions
Every major aircraft accident is a unique experience for aviation safety investigators. An aircraft accident investigator may not experience investigation of a large accident during their professional career, but they must be constantly prepared for such an eventuality.
The use of aircraft accident investigation training incorporating innovative aspects of virtual reality or augmented reality for onsite activities is an attractive thought For instance, it would allow the integration of historical major accident data as a training tool to develop investigator skills. It may be anticipated that virtual reality tools will be developed as training aids and on-site aids for safety investigators in the near future…
Written by: Mohammed Abdul Bari, Air Accident Investigator, GCAA-AAIS
Published on: The Investigator Magazine, Volume 1, Issue 11, October 2018