Stream VR from PC to NReal over WiFi (wip)

XREAL MR Apps
1

So this is a side project I’ve been working on for over a month now. I was hoping I would finish it by now, but I’m only half way through.

The project isn’t spacificaly exclusive to NReal, but I’m building it to use it with my NReal glasses. Basically It would work with any device that support WebXR.

The idea is simple: Make an OpenVR driver that let you communicate both tracking and view through a web browser using WebXR.

So basically all you have to do is enter your PC private IP in your WebXR supported device and you can stream VR to it :grin:

I have never been THIS FAR OFF from my comfort zone :cry: Literally every single step was new an required days of reading and testing just so I can wrap my head around it.

The project involved:

  • OpenVR API to produce the DLL files for the driver
  • websocket (JavaScript) to winsock (C++) communication, for both HTML request for the page, and socket to socket connection.
  • OpenSSL for encryption.

I mentioned that I was half way through. I managed to get OpenVR to send a WebXR page by accessing it’s IP. And I managed to send the tracking data to SteamVR.

The next half is getting the image to the device. I already have access to the frames. All I need now is figure out how to stream it to the headset.

Devices in the video I tested the tracking with:

  • My phone Asus ROG 3 using “Samsung Internet” browser. Chrome worked too but it was lagging for some reason.
  • NReal glasses using the FarePlay app
  • Oculus Quest 1 using their newly WebXR supported web browser.

Here’s the video of the devices being test it:

I honestly hope this eventually works :sweat_smile: I’m hoping this would open a few more doors for those who would like to push their glasses further.

Personally I’m just hopping I can get to enjoy Bigscreen and VRChat in NReal. Hell, maybe even play a bit of Elite Dangerous too :yum:

1 Like
2

Hey Mr.FarePlay you are fabulous. This is in my opinion not the half of the way - the hardest part is working tracking data. This is awesome!!

If it is possible to open 2 browsers in your app and the tracking works even if this windows is not selected, there are a lot of possibilities.

I guess with ur app it was already possible switch to sbs (3D mode).

Maybe with your video player, … you can open the desktop video stream using vlc or other low latency destop streaming solution.

Thanks a lot for your work and please keep on doing such great projects.

1 Like
3

Please check the winodws executable from this project.

Possible to stream entire desktop or a window to a webbrowser (local IP)

4

Thank you for your kind and encouraging words :heart:

Yes, you can have other browsers open while having one separately for WebXR. I do that all the time while using Mozilla Hub. I open a separate browser to google videos or image to share them in the Hubs.

Thank you for your suggestion about streaming the mirror VR screen to the headset. That could actually speed up the process for testing although it may not work if you start an actual game. Some games will have their own screen when started and they lose tracking focus if they are in background.

Plus, I’m kinda stuck not using any webrtc based streaming services on the FarePlay app for now due to an issue with my phone. So that’s on hold till I figure it out or find another phone :frowning: