Skip to main content

Context-Aware Mixed Reality: A Learning-based Framework for Semantic-level Interaction.

Chen, L., Tang, W., John, N.W., Wan, T.R. and Zhang, J. J., 2020. Context-Aware Mixed Reality: A Learning-based Framework for Semantic-level Interaction. Computer Graphics Forum, 39 (1), 484-496.

Full text available as:

Chen_et_al-2020-Computer_Graphics_Forum.pdf - Published Version
Available under License Creative Commons Attribution.

[img] PDF
Computer_Graphics_Forum-compressed.pdf - Accepted Version
Restricted to Repository staff only
Available under License Creative Commons Attribution Non-commercial.


DOI: 10.1111/cgf.13887


Mixed Reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic-based interactive MR framework that is beyond current geometry-based approaches, offering a step-change in generating high-level context-aware interactions. Our critical insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object-specific behaviors, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real-world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material-aware prototype system for context-aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real-time semantic level interactions.

Item Type:Article
Uncontrolled Keywords:interaction techniques; interaction; methods and applications‐computer games; methods and applications; augmented reality; virtual environments
Group:Faculty of Science & Technology
ID Code:32880
Deposited By: Symplectic RT2
Deposited On:14 Oct 2019 08:46
Last Modified:14 Mar 2022 14:18


Downloads per month over past year

More statistics for this item...
Repository Staff Only -