Mixed Reality in Medicine: Providing More Insights to Advance Surgery
Just a decade ago, the idea of incorporating spatial computing systems created primarily for gaming, like virtual reality (VR), augmented reality (AR) and mixed reality (MR), into medicine seemed like a futuristic dream. Today, these highly advanced tools for visualization are already in clinical use around the globe. They have a variety of uses, helping medical professionals with anything from teaching and training new clinicians to providing patients with a closer look at their diagnosis and treatment.
Mixed reality vs. augmented reality
The terms “mixed reality” and “augmented reality” are sometimes used interchangeably. While the technologies both merge the real and the virtual world, creating a hybrid of both physical and digital realities, the main difference lies user interaction. Augmented reality is mostly associated with enhancing the real world with static or animated virtual overlays. In mixed reality, there is a focus on immersive technology that allows the user to interact with digital objects as if they are real. In Mixed Reality vs. Augmented Reality vs. Virtual Reality: Their Differences and Use in Healthcare, we discuss these differences and their application in medicine in more detail.
Surgical planning is one area specifically in which mixed reality is becoming a staple for viewing and interacting with patient scans and data. The outlook for the future is that these systems will also be used intraoperatively to provide a digital overlay of the patient—with all relevant information about anatomy and pathologies—so the surgeon can see “inside” the patient even before making an incision. Since currently the application of most mixed reality systems focuses on preoperative surgical plan review, this article will focus on that use case.
How mixed reality in surgery works
To enable viewing a case in mixed reality, the surgeon begins by creating a plan using surgical planning software. A 3D model of the patient’s anatomy is generated from available scans, like MRI and CT. The software then identifies and outlines the various anatomical structures and the surgical target, like a tumor or an arteriovenous malformation (AVM), on the image.
With these structures identified in the 3D model, the surgeon can begin to plan the optimal approach to the surgical target, avoiding any critical anatomy. Once this plan is created, the surgeon can then put on a mixed reality head-mounted display and bring it from the screen into the room. They can walk around it, assessing the plan from all angles, and even invite other clinicians to join in real-time to review and discuss the plan together.
Even though this technology is considered the pinnacle of surgical plan review today, the concept of assessing the surgical situation virtually actually dates back more than fifty years. The first hints of using spatial computing in surgery began in orthopedics when Robert Mann proposed using VR to allow surgeons to test multiple approaches for orthopedic cases in 1965.[i] Fast forward to 2020, this technology is more real than ever as it expands into a wide range of surgical subspecialties.
Mixed Reality in Neurosurgery
The close proximity of critical structures in the brain makes neurosurgery all the more complex. One small deviation from a surgical plan could drastically affect the surgical outcome and even the patient’s quality of life. Technology, like mixed reality, that allows the surgeon to review the plan in detail before surgery are crucial advancements in ensuring the treatment is carried out exactly as intended.
“I do every aneurysm case using mixed reality during the planning process. It helps me easily determine if I need to do clipping or coiling. Viewing in mixed reality is especially helpful for clipping. The technology shows you exactly where the aneurysm and aneurysm neck are located. Not only is it extremely helpful when planning the approach, it’s also fascinating to virtually view and interact with the anatomy.” —Prof. Veit Braun, MD, Head of Neurosurgery at Diakonie Klinikum Jung-Stilling, Siegen, Germany
Another clear advantage of planning with spatial computing is that from the first cut, the surgeon knows exactly what they are going to do and therefore the surgery can be more quickly completed with a decreased amount of complications. Since Prolonged time under anesthesia increases the risk of complications during and after the procedure[ii], more efficient surgeries benefit the patient immensely as the time under anesthesia is reduced.
Interdisciplinary planning for CMF cases with mixed reality
For craniomaxillofacial (CMF) surgery, incorporating the use of mixed reality into the planning process supports clinicians in their decision-making. Since multiple clinicians can join a session and observe the same patient images at the same time—even remotely—mixed reality technology actually facilitates interdisciplinary planning.
“I believe that mixed reality applications will be the next big step in virtual surgical planning, patient education, as well as medical education. Mixed reality, combined with planning tools, allows craniomaxillofacial surgeons to segment out and visualize complex anatomy in a three-dimensional environment, prior to stepping into the operating room. The fact that multiple users can interact with these virtual objects in the same environment is a game changer. We are very excited about the opportunities this will provide at our institution.” — Prof. Bradley Strong, MD, Otolaryngologist at University of California – Davis Medical Center
Surgeons, radiologists, and radiation oncologists, for example, can discuss resection with adjuvant radiotherapy during a tumor board meeting to determine the best approach[iii]. Since the surgeon knows and has seen the extent of resection in advance, the surgery can progress quickly as they know when to stop and in turn, move the patient to radiation therapy.
Functional Stereotactic Surgery
Functional neurosurgery applications, such as deep brain stimulation (DBS) and stereo-electroencephalography (SEEG), require interdisciplinary coordination in many steps of the clinical workflow. Neurosurgeons, epileptologists and neurologists need to work together from preoperative planning all the way to postoperative patient management.
Mixed reality technology supports functional neurosurgeons and epileptologists in combining their knowledge to create an optimal trajectory plan before an SEEG case. These types of cases consist of the implantation of up to 20 electrodes to identify epileptic foci in drug-refractory epilepsy patients. Collaboration between these experts can contribute to improved outcomes.
«Many SEEG cases are three-dimensionally complex implantations that are planned in an interdisciplinary team. Since the implantations are tailored according to clinical, imaging and EEG findings, each scheme is different. Especially for this use-case the visualization in mixed reality turns out to be very helpful for the team discussion during planning.» — Peter Reinacher, MD, Consultant, Department of Stereotactic and Functional Neurosurgery, Medical Center – University of Freiburg, Germany, and Group Leader (ATTRACT), Fraunhofer Institute for Laser Technology, Aachen, Germany.
Hours or days after deep brain stimulation surgery, the electrodes are programmed by neurologists to ensure they stimulate the patient’s brain as planned. Mixed Reality contributes to their understanding of the patient’s anatomy on a new level, supporting visualization-based programming in place of traditional trial and error approaches.
The 3D reconstructions the clinician can view in mixed reality greatly assist in their understanding of very complex anatomy. An example from vascular surgery would be a deep-seated kidney artery aneurysm, in which a true-to-life representation provides a much clearer picture into the case than two dimensional scans alone[iv].
In the future when mixed reality and related technologies come into regular use intraoperatively, the surgeon will have access to a range of data, without ever having to look away from the surgical situation. Bloodwork results, stent graft measurements, EKG, preoperative images, or even the radiation exposure to the surgical team could be displayed, turning the surgeon’s field of view into a “virtual cockpit”[v].
Mixed Reality in surgery is both the future and the present
In neurosurgery, CMF, functional and vascular surgery today, mixed reality is already in use and increasingly becoming an essential tool to better understand anatomy and ensure surgical plans are the best they can be. Since spatial computing technologies are adaptable to changes and new developments, they will continue to play a significant role in surgery in the future. Incorporating these advanced visualization tools into surgical workflows is already an easy step clinicians can take today to ensure their patients have great surgical outcomes.
[i] Rosen JM. Virtual Reality and Medicine-Challenges for the Twenty‐First Century. Wiley Online Library. https://onlinelibrary.wiley.com/doi/abs/10.1002/0471216690.ch4. Published October 29, 2001. Accessed October 20, 2020.
[ii] Phan K, Kim JS, Kim JH, et al. Anesthesia Duration as an Independent Risk Factor for Early Postoperative Complications in Adults Undergoing Elective ACDF. Global spine journal. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5721997/. Published December 2017. Accessed October 20, 2020.
[iii] CMF PAPER BEING PUBLISHED
[iv] Böckler, D., Geisbüsch, P., Hatzl, J. et al. Erste Anwendungsoptionen von künstlicher Intelligenz und digitalen Systemen im gefäßchirurgischen Hybridoperationssaal der nahen Zukunft. Gefässchirurgie (2020). https://doi.org/10.1007/s00772-020-00666-9
[v] Böckler, D., Geisbüsch, P., Hatzl, J. et al. Erste Anwendungsoptionen von künstlicher Intelligenz und digitalen Systemen im gefäßchirurgischen Hybridoperationssaal der nahen Zukunft. Gefässchirurgie (2020). https://doi.org/10.1007/s00772-020-00666-9