Augmented Reality and 3D Printing for Surgery

A group led by the University of Bremen is combining virtual reality, augmented reality and 3D printing in order to improve the planning and realization of surgeries. The technologies can also be used for the training and continuing education of doctors.

New technologies can help to provide surgeons in surgical theaters with important information and thus significantly increase the chances of success. A research group led by the Center for Computing Technologies (TZI) at the University of Bremen is currently developing test processes and applications. The use of virtual reality (VR), thus artificial realities created with the help of computers, augmented realities (AR), i.e. computer-supported expanded reality, and 3D printing are to improve the planning and realization of surgeries. The results are to be additionally employed for training purposes and for patient information. The Federal Ministry of Education and Research is funding the “Vielseitiger Immersiver Virtueller und Augmentierter Tangible OP” (Multifaceted, Immersive, Virtual and Augmented, Tangible Surgery) (VIVATOP) with a total of 2.2 million euros over a period of three years.

Preoperative Planning

The planning of an operation with 3D models has significant advantages over the current method with 2D pictures. Doctors have a notably more realistic and comprehensible overview of the area on which they wish to operate. Alongside the 3D computer models, the VIVATOP researchers are testing the use of lifelike physical models of the affected organs. The picture data is analyzed and prepared for this so that it can be imported to a 3D printer, in order for a lifelike organ, for example a patient’s liver, to be made.

The stated printed models make it possible for doctors to gain haptic experience with each organ, thus the use of their sense of touch, but can also be applied together with the virtual model. It is therefore possible for someone to hold the physical model in their hands and to explain with gestures, whilst other participants can see the gestures in connection with the highly detailed computer model through a VR headset. This method of use correlates more with our natural habits than the use of controllers in the way that they are used for the controlling of movement in games consoles.

The 3D prints, which can be either transparent or multicolored, do not only simplify the planning but also the consultation with patients. They allow the team of doctors to explain the procedures in a clearer manner.

Support during Surgery

During surgery, picture data and models of the organ help doctors to orient themselves and to make decisions. Additionally, the VIVATOP team is expanding an available augmented reality application, which allows for the relevant information to be shown in the field of view of a pair of glasses. Surgeons are able to control the unit with voice controls and without using their hands. At the same time there is also the possibility of presenting surgeries live in virtual reality so that experts outside of the surgical theater can be consulted (remote, collaborative, immersive surgery consultation). The hand movements of doctor teams and the procedures on the operated organ can be recorded using depth cameras, sensors and further technologies. These can then be shown in virtual reality. The planning details can also be displayed in virtual reality at the same time. Using telepresence, the system supports worldwide cooperation of specialists during a surgery.

Training and Further Education

Furthermore, vocational training, professional development and further education of young doctors can be designed in a more efficient manner by integrating AR, VR and 3D printing. Specific cases are able to be discussed using the 3D print data and can be precisely comprehended in virtual reality. The printed 3D models allow young doctors to gain valuable first experience. They can, for example, learn to examine tumors at varying stages by palpation (haptic feedback) and to subsequently judge the location in virtual reality. Individual surgery steps can also be trained in virtual reality.

A Network of Science, Economics and Healthcare

The working group Digital Media Lab (Prof. Dr. Rainer Malaka) at the TZI, University of Bremen coordinates the VIVATOP project. The working group Computer Graphics and Virtual Reality (Prof. Dr. Gabriel Zachmann supports the project. These teams are primarily responsible for the VR interactions, or rather the VR algorithms. The Pius Hospital Oldenburg/University Hospital for Visceral Surgery provides the clinical expertise and generates the picture data. The Frauenhofer MEVIS: Institute for Digital Medicine uses that data to produce virtual reality organ models for AR/VR and the 3D prints, whilst apoQlar GmbH and cirp GmbH are involved as specialists in the fields of augmented reality and 3D printing, respectively. The areas of training and further education are led by szenaris GmbH. The project is intended to be completed in September 2021.