On the trail of crime with the “holodeck”

Written by Christian Steiner on 4. March 2024

At the Bavarian State Office of Criminal Investigation, science fiction becomes reality: In virtual, reconstructed crime scenes, investigations can be carried out with entire teams long after the crime has been committed.

At the BLKA, several people can view a digitized crime scene virtually at the same time in the “Holodeck”.

Summary:

Together with the Bavarian State Office of Criminal Investigation (BLKA), INCAS Training und Projekte GmbH & Co. KG realized the development of a VR application for virtual crime scene inspections.

The application is used in practice in the so-called “holodeck”, a specially equipped VR room. It can be used by up to eight people in different roles at the same time.

INCAS developed various modules for the software used and also provided training for BLKA employees.

The application is used to support the administration of justice and is currently in the proof of concept phase. Continuous further development is planned.

The starting position:

The BLKA is an innovative police authority of the Free State of Bavaria with interfaces to the federal and state governments and is responsible for combating serious crime, organized crime and terrorism.

For years, the BLKA has been using state-of-the-art technologies to efficiently document and analyze crime scenes. An extensive forensic team is entrusted with the digital recording of crime scenes, using advanced methods such as photogrammetry, laser scanning and drone photography. By using these cutting-edge tools, the BLKA can create detailed 3D reconstructions of crime scenes, enabling precise and comprehensive analysis.

These technologies help to optimize investigations and further improve the solving of crimes.

The next step is to combine these different forensic methods into a comprehensive holodeck solution. This will allow the BLKA to expand its capabilities to include virtual crime scene inspections using the latest virtual reality technologies.

The development of comprehensive and cross-media software for virtual crime scene inspections was a completely new challenge. A competent partner had to be brought on board for the implementation. The development of this application was to lead to an open framework that could be further developed and individually adapted to specific cases.

INCAS was able to cover employee training, the development of software modules and consulting.

Ralk Breker has a clear vision for the implementation of his “Holodeck” project

The development:

Ralf Breker, Head of Forensic Media Technology at the Bavarian State Office of Criminal Investigation and the main person responsible for the “Holodeck” project, approached INCAS with a clear vision. His many years of experience in the field of forensics provided him with a precise idea of how meaningful software for virtual crime scene inspections should be structured.

Together with INCAS, the decision was made to use the Unreal Engine, as it offers the optimum range of functions both for visual representation and for mapping essential workflows, such as the processing of very large 3D data, as is the case when scanning crime scenes.

Thanks to INCAS’ experience in the field of virtual reality development, the software was designed with open standards from the outset, meaning that the choice of specific VR glasses only played a subordinate role. For example, the BLKA decided to use HTC Vive Pro Eye at the start and was able to switch seamlessly to HP Reverb G2 and HTC Vive Elite over the course of the development period.

Another challenge that could only be solved together was Ralf Breker’s desire to develop the application in such a way that the entire operation can be carried out uniformly in VR as well as on the screen.

“Virtual reality is a future technology with great benefits for us. However, it will take a little time before the technology is used across the board. In order to give other colleagues in forensics access to it, using it on a screen is essential for the acceptance of our software,” says Breker.

From an ergonomic point of view, this requirement was and is particularly demanding. On a PC, users operate the crime scene application with a mouse and keyboard. In the virtual world, however, it is the user’s own hands that become the input medium. Thanks to creative teamwork, small UI prototypes were quickly developed, tested together and ultimately found to be suitable and implemented. As a result, the application can now also be operated by people unfamiliar with VR on the screen or in the virtual world using natural hand interaction.

This is what a virtual crime scene looks like in the application.

The application

At the BLKA site in Munich, it was possible to find a large room with sufficient space to physically implement the holodeck. New scenarios and new updates can be tested there during the project. It enables the protagonists to carry out virtual crime scene inspections on site at any time.

Several VR goggles, connected either wirelessly or by cable to high-performance PCs, operate synchronized within a large multiplayer application. An operator can monitor the status of all users on a screen at any time and provide assistance with operation. The VR moderator controls the entire process of the crime scene inspection with functions that are only available to him and can, for example, influence the time sequence of all animations, the display of a 3D scene or the display of photos, videos or 3D objects, such as a murder weapon or other evidence.

As the role of the viewer, it is possible to be passively guided through the crime. It is suitable for judges, victims or even perpetrators and can be expanded to include special functions such as eye-tracking to record a “gaze heat map”.

The application has already been used to reconstruct a train accident in Burgrain or the case of an S-Bahn pusher in Munich.

In these cases, it is used to check the plausibility of statements made by suspects or witnesses. The application is particularly useful in large, complex criminal cases.

All users are in the same application at the same time. In different roles.

The future:

The development team at BLKA has grown and is continuing the development. In addition to project development, INCAS also offers training on the Unreal Engine used.

Ralf Breker says about the future development of the application: “We are now gaining a lot of experience from the first uses in the Holodeck and the feedback is extremely positive. We have already come up with some useful ideas for expansion. I can hardly wait to continue developing the application with our team over the next few years thanks to the feedback from all our colleagues.”

We at INCAS would like to express our thanks for this extremely productive collaboration on this meaningful project and look forward to continuing to make a contribution to tracking down crime in the future.

Many thanks to Ralf Breker for his forward-looking ideas and the trust he has placed in us to implement them. Thanks also to his professional team for the great cooperation.