VRySmart

Motivation:

• Smart devices are ubiquitous

• Smart devices are everyday objects

• Smart devices can act as fully-fledged controller

• Smart devices have rich functionality that can be utilized in VR

• Users are familiar with the interaction style and affordances

The framework aims to improve experience in following aspects:

• Haptic feedback

• Interaction

• Notifications handling

The framework handles following functionality:

• Networking

• Marker based Positional tracking of the device

• Translation of touch-events from the device to the main app

• Vibration calls from the main app to the device

• Camera streaming (based on Unity Render Streaming)

• Hands/body Segmentation (based on SRWorks)

• Body segmentation (ML-based)

Future work:

• SLAM-based tracking

• Screen content streaming