Surgical Gesture Recognition in Robot-Assisted Surgery using Machine Learning Methods on Kinematic Data

Abstract

This diploma thesis focuses on training a machine learning model to recognize gestures during robot-assisted surgical procedures in real-time, using exclusively kinematic data from the patient-side manipulators. The JIGSAWS dataset, specifically the suturing tasks, serves as the evaluation benchmark. Our goal was to achieve state-of-the-art performance, ensuring the model operates in real-time with a maximum delay of 1 second and is trained solely on kinematic data. We experimented with various neural network architectures, using an LSTM architecture as foundation, in order to effectively capture temporal dependencies within the data sequences. Visualization tools like graphs, confusion matrices, and transition matrices were employed to identify areas for improvement. Challenges arising from imbalanced data led to difficulties in recognizing underrepresented classes. We expanded the feature set, creating a new feature based on gripper angles. To further enhance performance, we implemented two hybrid approaches: one integrating an attention layer and another combining an LSTM with a Conditional Random Field (CRF) to leverage the sparse transition matrix. Our efforts culminated in a hybrid LSTM - Self Attention model, achieving an accuracy of 81.56%, demonstrating improvements and meeting the constraints set for real-time operation and exclusive use of kinematic data.