During complex surgeries, having access to specialized clinical experts can make all the difference. But what if the expert is on another continent? A new joint study by researchers from the Cluster of Excellence CeTI and clinical partners showcases how Mixed Reality (MR) can replace flat 2D video feeds with an immersive, real-time 3D workspace—allowing remote surgeons to guide local procedures with unprecedented precision.

The paper, titled “Surgical Telementoring in Mixed Reality”, has been published in IEEE. This highly interdisciplinary work was co-authored by Mats Ole Ellenberg, Katja Krug, Yichen Fan, Jens Krzywinski, Raimund Dachselt, Rayan Younis, Martin Wagner, Jürgen Weitz, Ariel Rodriguez, Gregor Just, Sebastian Bodenstedt, and Stefanie Speidel—uniting leading experts from the Interactive Media Lab Dresden, the Chair of Industrial Design Engineering, and the surgical departments of the University Hospital Carl Gustav Carus Dresden.

The Problem: The Visual Limits of Flat Video Telementoring

Surgical telementoring—where an experienced specialist remotely guides a local surgeon through an operation—historically relied on traditional 2D video streams. However, flat screens fail to capture the spatial depth, complex physical anatomy, and physical layout of a real operating room.

Remote experts could only watch, point, or talk on a 2D monitor, making intuitive, real-time spatial guidance incredibly difficult. Under high-pressure surgical conditions, this lack of spatial depth can lead to communication lag and precision errors.

Research Highlights: Transforming the Operating Room into a 3D Canvas

To bridge the physical distance, the research team designed an immersive telementoring framework that leverages the power of Mixed Reality:

  • From 2D to 3D Workspaces: Instead of looking at a flat video, remote experts are placed in a fully interactive 3D digital duplicate of the surgical environment, allowing them to perceive depth, angles, and spatial relationships intuitively.

  • Immersive Annotations: The system enables the remote mentor to place precise 3D visual markers, guides, or virtual tools directly into the local surgeon’s field of view via MR headsets, facilitating instantaneous, clear guidance.

  • User-Centered Design: By combining the expertise of human-computer interaction (HCI) researchers, industrial designers, and practicing clinical surgeons, the platform was tailored directly to the ergonomic and cognitive demands of highly intense operating rooms.

The CeTI & Digital Health Impact

In high-stakes environments like digital medicine and telesurgery, the human-in-the-loop concept is central. This research demonstrates how human expertise and high-tech spatial computing can merge to bypass geographical borders.

By allowing top-tier surgeons to virtually step into any operating room worldwide and guide procedures in real time, this technology democratizes specialized healthcare, optimizes patient outcomes, and lays a vital foundation for the future of global, connected medicine.

The research is published in the IEEE library. Dive into the system architecture, clinical evaluations, and design methodologies here.

Authors: Mats Ole Ellenberg, Katja Krug, Yichen Fan, Jens Krzywinski, Raimund Dachselt, Rayan Younis, Martin Wagner, Jürgen Weitz, Ariel Rodriguez, Gregor Just, Sebastian Bodenstedt, and Stefanie Speidel

This work represents a breakthrough in digital health and spatial computing, developed collaboratively at the Technische Universität Dresden.