Training for keyhole surgery (minimal invasive surgery like laparoscopy) is complex and the learning success is not yet objectively measured. One prototype has already been built for the purpose of optimizing surgical traning which has delivered promising results in small user studies. The prototype uses fast machine-learning-based scene analysis which analyzes learning progress and compares with experts. Therefore a large dataset was collected that was used to generate haptic feedback via the instrument based on forces and instrument visibilty.

The demonstrator EndoMersion is an immersive robotic laparoscope guidance with 3D perception. The goal here is to provide computer-assisted camera guidance during keyhole surgery using patient and sensor data. The navigation is based on Augmented Reality and Artificial Intelligence. We acquire/ generate/ annotate large data sets of real surgical videos with the surgical department VTG which are processed by a neural net to create realistic simulated environments showing, for example, optimal cutting lines to protect nerves. The methods are being developed to bridge the gap between robotics, sensors, and data science in surgery.

The CoBot assistance system is to be used for tumor operations in the rectum. The device relieves the surgeon of the direct holding and moving of instruments and translates larger hand movements, which the surgeon executes via two joystick-like handles, into tiny tremor-free incisions. Since surgeons depend on visual information, camera images of the laparoscope are shown with additional information displayed at the right time, such as the location of important nerves. However, the surgeon makes the decision himself at any time. The system only supports him, similar to a navigation system in a car.

The purpose of intraoperative navigation is to provide information of hidden risk and target structures based on preoperative patient planning data. The visualization can be done via Augmented Reality (AR), either by overlaying the information directly in the endoscopic view or by using glasses. In soft-tissue navigation, this process is substantially complicated due to the organ deformation, induced by forces applied to the organs, cutting of the tissue, different patient poses and the patient’s breathing. The estimation of organ-internal deformation and movement of risk/target structures from intraoperative sensor data (e.g. surgical video streams) and pre-operative data in real time is an open research topic. Our team is using novel machine learning methods for many of the tasks involved.

In the surgical domain, obtaining ground-truth data for computer vision tasks is almost always a major bottlenack. This is especially the case for ground truth which would require additional sensors (e.g. depth, camera poses, 3D information) or fine-grained annotation (e.g. semantic segmentation). To this end, the aim is to render photorealistic image and video data from simple surgical simulations (e.g. laparoscopic 3D scenes).

There are almost no public datasets available which show the liver of a single patient in multiple deformed states. To drive our machine learning algorithms for navigation, we are developing simulation pipelines to generate synthetic deformations of both real and non-real organs. We build upon physical biomechanical models to generate a vast amount of semi-realistic data which is then used to train fast and powerful neural networks.