Hi everyone, I’m using the Nreal Light RGB camera to get frames and find objects in them using YOLO. Once I get the 2D points corresponding to the positions of objects in a particular frame, I want to tranform this point back in Unity 3D space. I’m currently doing so with the method ScreenToWorldPoint(), but the x and y coordinates position are not so accurate (the z is estimated by me). Does NRSDK provide an alternative way?
Thank you in advance!
1 Like
Hi, developer. Maybe the attached post can be helpful to you.
Hi, thanks for the response. Unfortunately this doesn’t seem to fix my problem. More precisely, I have 2 variable x and y, both are normalized in the range (0,1). This coordinates are associated to the center position of one object that was recognized in screen space. I want to track it in the 3D world. This is what I tried so far:
transform.position = Camera.main.ViewportToWorldPoint(new Vector3(x, y, z));
I previously used ScreenToWorldPoint() (after getting those coordinates in pixels), but its the same result.
Maybe there is some sort of matrix that I could use, provided by NRSDK as utility. If so could you please explain to me I to use it?
Dont use the Camera.main
For the Camera from the glasses you might want to use CameraCenter
Join us on discord <3
The RGB cam corresponds to the far plane of the camera, so converting your viewport to world point you use that for the z axis, to augment objects on any detected yolo object.
Hi, have you had a chance to look into the following explanation in another thread? As whatever camera you call ScreenToWorldPoint() with, this issue will surely happen as the physical RGBCamera position is not identical with either center/left/right camera. Quoted from the reply of the thread mentioned above:
The offset comes from the different pose between RGBCamera and virtual camera (leftCamera, rightCamera or centerCamera you call ScreenpointToRay with). Since you get the texture from RGBCamera, you also need to virtualize a RGBCamera whose pose corresponds to the physical position on glasses and whom you call ScreenpointToRay with.
Please inspect NRKernal.Record.NRCameraInitializer, where the position of “virtual RGBCamera” gets initialized:
transform.localPosition = eyeposFromHead.RGBEyePose.position;
transform.localRotation = eyeposFromHead.RGBEyePose.rotation;
You could create a new camera (RGBCamera, for example) under NRCameraRig. Then add “NRCameraInitializer.cs” as a component. Choose RGB_Camera as device type:
Call ScreenpointToRay on this camera when needed.
Hope this will help! Thank you.