[Unity] Virtual Displayer showing in the HMD rather than on Phone

Development & Help
,
1

Hello! I seem to be running into a problem whenever we try to use a Unity Virtual Display. Opening our app causes the user to see the virtual display flickering in their HMD rather than their phone, in addition to what they’d normally see in the app. This doesn’t occur if we delete the virtual display prefab from the scene and the default android native one is used instead. However, we’d like to use a custom controller which requires us to use an in-scene display. Does anyone have any idea what could be causing this or at the least what the right scripts to look at may be? We are using URP which might be a factor, but we didn’t have any issues like this until upgrading to 1.7.0.

Some observations:

  • It seems like the view from the Virtual Display camera is being rendered to the HMD
  • The “flicker” is a result of frames from the actual HMD and the Virtual Display overwriting each other
  • You can still use the phone as a controller, tapping where the touchpad should be still picks up a button press
  • This doesn’t seem to occur with the native android controller, only the Unity version. This includes the VirtualDisplayer prefab included in the NRSDK

Unity Version: 2020.3.22fa
NRSDK Version: 1.7.0
Nebula Version: v2.02.00.2111081602
Nreal Serial: A939FD0412

I’ve also run a logcat, though I couldn’t see any error messages. This ran from the start of opening the app, then waiting a few seconds, then closing. This log is filtered to Unity-tagged messages only. If it would help I can also get a full logcat.
Filtered - 01-02 23:49:27.328 17491 32349 I Unity : XRGeneral Settings awakening...01-0 - Pastebin.com

I have seen the other community thread with a similar title here: Virtual display causes the flicker on the device screen - #4 by kamihiro. Unfortunately their solution did not work, and they seem to have run into a seperate problem than ours as far as I can tell.

2

Was digging through the full logcat and found these getting output during the app’s runtime, could be related. Going to see if I can’t find what’s causing these errors

01-03 13:21:02.265 32245 20358 I Unity   : [XR] [NrealSDK] Initialize Render.
01-03 13:21:02.266 32245 20405 I NREAL   : [2022-01-03 13:21:02.266] [INFO] [NRSDK] NRHMD START!
01-03 13:21:02.266 32245 20405 I NREAL   : [2022-01-03 13:21:02.266] [INFO] [NRSDK] NRHMD RUN!
01-03 13:21:02.266 32245 20405 W NREAL   : [2022-01-03 13:21:02.266] [WARN] [NRSDK] Not supported command(61)
01-03 13:21:02.266 32245 20405 E NREAL   : [2022-01-03 13:21:02.266] [ERROR] [NRSDK] GetSupportedDevices fail 1 
01-03 13:21:02.266 32245 20405 W NREAL   : [2022-01-03 13:21:02.266] [WARN] [NRSDK] Not supported command(62)
01-03 13:21:02.266 32245 20405 E NREAL   : [2022-01-03 13:21:02.266] [ERROR] [NRSDK] GetSupportedDisplays fail 1 
01-03 13:21:02.266 32245 20405 I NREAL   : [2022-01-03 13:21:02.266] [INFO] [NRSDK] InitSetRenderingFlags: 1
01-03 13:21:02.266 32245 20358 I Unity   : [XR] [NrealSDK] Start Rendering.
01-03 13:21:02.266 32245 20405 I NREAL   : [2022-01-03 13:21:02.266] [INFO] [NRSDK] NRDisplay START!
01-03 13:21:02.266 32245 20405 I NREAL   : [2022-01-03 13:21:02.266] [INFO] [NRSDK] NRDisplay RUN!
01-03 13:21:02.267 32245 20405 I NREAL   : [2022-01-03 13:21:02.267] [INFO] [NRSDK] NRRendering START!
01-03 13:21:02.269 32245 20405 I NREAL   : [2022-01-03 13:21:02.268] [INFO] [NRSDK] hardware is:qcom
01-03 13:21:02.269 32245 20405 I NREAL   : [2022-01-03 13:21:02.269] [INFO] [NRSDK] NRRendering RUN!
01-03 13:21:02.275 32245 20405 I NREAL   : [2022-01-03 13:21:02.275] [INFO] [NRSDK] Display interrupt type=1.
3

Just documenting progress in case anyone runs into this issue in the future. I was able to recreate the issue with just the NRSDK package in an empty project. After adding in URP + the NRSDK XR Plugin, the issue started occurring again. Since we want post processing in the project I’m going to be looking for a workaround. But for anyone running into this that doesn’t need PP, you may be able to avoid this issue by removing the plugin

4

Last update, we found that the URP post processing stack was still working without the XR Plugin, however HDR emission values were not picked up by the bloom. This can be resolved by changing some code in the NRSDK.

There are a couple places where the SDK creates a render texture that eventually gets displayed on the HMD. The SDK initializes this render texture with the format RenderTextureFormat.Default, all we need to do is change this to RenderTextureFormat.DefaultHDR so that the color space has HDR support.

Hopefuly this helps out who runs into anything similar!