How can augmented reality help first responders in critical situations? Examples from the War in Ukraine

Katherine Quezada-Tavárez and Rune Vanleeuw

At the end of February 2022, peace was shattered in Europe due to the Russia-Ukraine conflict, a situation that has been described as “the West’s Biggest Test Since World War II”. Since the invasion of Ukraine, NATO is witnessing atrocities of war at its doorstep. Beyond the obvious military and defence operations taking place, the world also watches the emergency management efforts by first responders on the battlefield. Building on the research conducted within the H2020 DARLENE project, we examine how the ability of first responders to handle this type of post-combat emergency could be improved. In the midst of Europe’s worst security crisis in decades, we would like to flag the possibilities for augmented reality as an emerging solution to enhance first responders’ situational awareness and support tough decision-making during emergencies.

Crisis management in a war zone

War is, unfortunately, back in Europe. After almost three decades of relative peace on the continent, Russia invaded Ukraine on 24 February 2022, causing significant casualties and devastation. In the wake of major incidents, first responders such as firefighters, emergency medical workers and police play a crucial role in dealing with the immediate aftermath. When an emergency strikes, situational awareness and information exchange are crucial for an effective response by all team members, especially in areas of active warfare. Amidst ongoing violence and destruction, with the danger of attacks looming practically anywhere in the territory, all first responders can benefit from quick assessments of the situation and seamless exchange of information: from incident commanders, who need an overall understanding of the state of affairs, to staff on the ground who need to understand the right time/place to act. This is where emerging technologies, such as Augmented Reality (AR) and Artificial Intelligence (AI), come into play.

Enhanced situational awareness with AR 

Top trending technologies, like AR and AI, provide humans with improved capabilities and facilitate the performance of tasks. For instance, AR, combined with AI and 5G technology, can provide first responders with rapid situation assessment and accurate information exchange between incident commanders and local staff, thus helping in the prevention and management of crises. A case in point is the current security crisis, as novel AR solutions can help with healthcare provision after missile attacks and shelling on towns and cities, aiding in the prompt search and rescue missions in shelled environments. In this context, smart glasses can provide ground personnel with an “augmented” sight of their surroundings, highlighting silhouettes of injured persons, thus enhancing first responders’ ability to quickly locate individuals for the timely delivery of care and assistance. The glasses’ visor can also make potentially dangerous objects or armed adversaries more noticeable, thereby facilitating timely action and threat neutralisation.

These technologies can help in other ways too. Adaptive visualisation features would allow for the provision of tailored information based on the wearer’s stress level and situation context. This would not only result in improved user experience but can also boost first responders’ capability to deal with high-stress situations. Furthermore, 5G capabilities can facilitate the remote collaboration between teams, providing field personnel and their command staff with streamlined exchange of information to better coordinate their actions, while minimising incidents caused by the neglection or late delivery of vital information. This way, such a system can contribute to the situational awareness of the team, increasing chances of survival and protection of civilians, while reducing or minimising threats.

Building law and ethics into the technology design

Such novel technological applications come with great promise, but one may not lose sight of potentially adverse consequences. From the beginning of the DARLENE project, for example, it became apparent that while the proposed solution may indeed be a powerful tool providing unique opportunities for its end-users, there are, likewise, inherent risks (legal, ethical and otherwise) associated with it. Anyone familiar with AI-enabled systems would immediately pinpoint major ethical issues, including human agency and oversight, technical robustness and safety, privacy and data protection, transparency, non-discrimination and fairness.

When considering the applications outlined in this contribution, a particularly important issue is technical robustness and safety. If the software systems are not protected against vulnerabilities, that could allow adversarial attacks. For example, software vulnerabilities could result in unauthorised access to the system, thus enabling adversaries to ‘see through the eyes’ of ground personnel. Another example entails exploiting system functionalities to introduce false data (e.g. misleading information) into the visualisation. Both situations would ultimately facilitate the aggressors’ actions and entail potential avenues for safety harms. This explains the importance of having AR with an augmented focus on law and ethics and risk analysis from the technology design. In particular, complex socio-technical systems call for systematic impact assessments and building mitigation measures into the technology from the start.

Concluding remarks

At the time of writing, nobody knows when and how the war in Ukraine will end. What is clear is that first responders are on the front lines risking their lives, and they must be equipped with the tools that best allow them to efficiently and successfully perform their duties. This article is an attempt to show that, when dealing with crisis management, like in the aftermath of the deadly shelling, firefighters, emergency medical services, and police can benefit from enhanced situational awareness, efficient coordination and reliable information exchange. In particular, we articulate the importance of enhanced sight for first responders and the potential of AR to enable increased protection and more agile response in life-threatening scenarios. In that way, AR has the potential to provide tremendous amounts of information that remain largely inaccessible in crisis management, which may foster personnel safety, efficient response to dangerous incidents and, eventually, save more lives.

The research for this article has been conducted within the DARLENE project. This project has received funding from the European Union’s Horizon 2020 Research and Innovation programme under grant agreement No 883297.

Curious to read more about the situation Ukraine from an insider perspective? Then make sure to check David Mendelsohn’s interview with a foreign fighter on the ground via this link!

Share this article


Join over 150,000 marketing managers who get our best social media insights, strategies and tips delivered straight to their inbox.