Virtual Reality 101 - input to a spatial computer.
While we have transitioned to immersive workspaces, the input mechanisms are still stuck with the era of QWERTY layout keyboards in thin air. Typing on these virtual keyboards is really hard and takes a long time. The transition to typing with touch also took time, but it worked since there’s a tactile response to every keystroke, and the swipe was genius. This made the T9 dictionary obsolete, but recommendations started with T9 and it’s lived on in this form since then. It’s a different story when typing in thin air, there’s no tactile response and muscle memory doesn’t help since the tracker doesn’t track all finger-movement well.
The next phase of typing in immersive worlds is electromyography, it essentially eliminates dependency on video feeds recognising every subtle finger movement and replaces it with recognising electrical signals passing through your hand muscles.
There’s a fundamental approach problem here, we are still trying to just replace our screen with a virtual reality experience while keeping everything else the same - keyboards (type in the air), mouse arrow + click/tap (pointing fingers and click gesture), and the most important of all, square windows (9:16 aspect ratio windows for the adventurous). We are not utilising the new dimension of depth at all, and this is a brand new career for user experience designer in this decade and possibly the next. As an example, I don’t need multiple browser windows, I need multiple virtual spaces that I can switch between easily. But I digress, that’s a conversion for another topic.
Coming back to inputs, I tried speech-to-text on Hololens and it works beautifully (when it does). Just getting it up-and-running was a bit painful. But glad to see Microsoft thinking through this, feedback from early adopters. That also indicates the feedback from early Oculus users was not prioritised on speech-to-text - likely that users were not expected to work using Oculus. It is primarily used for entertainment with limited training and simulation use-cases.
Nonetheless, speech-to-text seems to be the way forward for the next generation of VR headsets until a stronger input framework is developed. For the vast majority of Indian languages, I have not seen/used keyboards since it’s a learning curve and it’s easier to type in English. I’ve occasionally used hacks to type in English and that verbatim converts to Kannada, and I make corrections as needed. But speech-to-text for spatial computing spaces is an excellent opportunity to support all Indian languages in tandem with English and other latin based languages. Everyone is starting from scratch and keyboards are out.
The long-term vision for inputs into virtual worlds is likely BCI, possibly a non-invasive version of BCI that enables users to type in virtual worlds without saying things out loud or move their fingers in thin air. It is definitely not easy to move away from keyboards considering keyboards/mice have worked for soo long, or likewise rectangular screens in virtual reality, but that’s the path forward as we adopt spatial computers.
Can’t wait to see how it is handled on Vision Pro, even with the cameras facing your hands at rest, you cannot type long sentences on Vision Pro - this is a speech-to-text future. The good news for Oculus is it’s just a software upgrade, so I’m sure they will have an update on this soon.
Keyboards and mice are for screen-based devices, not fully-immersive experiences. What will work is a free-flowing GPT-style conversational AI to interact with the device.
After being comfortable seated in virtual reality, the next logical step is to interact with the apps to figure out what can be done. If I want to work on my spatial computer exclusively, I should be able to do all the things I do with my smartphone, to begin with. Not Excel, VSCode, etc., but replying to emails, setting notifications/reminders, blocking calendars, etc. without the need to level up on my fine motor skills.