Skip to main content

SD-Net: joint surgical gesture recognition and skill assessment.

Zhang, J., Nie, Y., Lyu, Y., Yang, X., Chang, J. and Zhang, J., 2021. SD-Net: joint surgical gesture recognition and skill assessment. International Journal of Computer Assisted Radiology and Surgery, 16, 1675-1682.

Full text available as:

Zhang2021_Article_SD-NetJointSurgicalGestureReco.pdf - Published Version
Available under License Creative Commons Attribution.


DOI: 10.1007/s11548-021-02495-x


PURPOSE: Surgical gesture recognition has been an essential task for providing intraoperative context-aware assistance and scheduling clinical resources. However, previous methods present limitations in catching long-range temporal information, and many of them require additional sensors. To address these challenges, we propose a symmetric dilated network, namely SD-Net, to jointly recognize surgical gestures and assess surgical skill levels only using RGB surgical video sequences. METHODS: We utilize symmetric 1D temporal dilated convolution layers to hierarchically capture gesture clues under different receptive fields such that features in different time span can be aggregated. In addition, a self-attention network is bridged in the middle to calculate the global frame-to-frame relativity. RESULTS: We evaluate our method on a robotic suturing task from the JIGSAWS dataset. The gesture recognition task largely outperforms the state of the arts on the frame-wise accuracy up to [Formula: see text] 6 points and the F1@50 score [Formula: see text] 8 points. We also keep the 100% predicted accuracy for the skill assessment task using LOSO validation scheme. CONCLUSION: The results indicate that our architecture is able to obtain representative surgical video features by extensively considering the spatial, temporal and relational context from raw video input. Furthermore, the better performance in multi-task learning implies that surgical skill assessment has a complementary effects to gesture recognition task.

Item Type:Article
Uncontrolled Keywords:Self-attention ; Surgical gesture recognition ; Surgical skill assessment ; Temporal convolutional network
Group:Faculty of Media & Communication
ID Code:36142
Deposited By: Symplectic RT2
Deposited On:25 Oct 2021 15:40
Last Modified:30 Mar 2022 13:56


Downloads per month over past year

More statistics for this item...
Repository Staff Only -