If you want to use that parameters with that Unity OpenCV-Asset from Enox, I converted them like that, to be usable with Aruco.estimatePoseSingleMarkers() to get the rvec and tvec, which are by the way the Rotation-Vector and Translation-Vector (last one: for positioning).
private Mat convertToMat(NativeMat3f input)
Mat output = Mat.zeros(3, 3, CvType.CV_64FC1);
Vector3 columnVectors = new Vector3;
columnVectors = input.column0.ToUnityVector3();
columnVectors = input.column1.ToUnityVector3();
columnVectors = input.column2.ToUnityVector3();
for (int i = 0; i < 3; i++)
for (int j = 0; j < 3; j++)
output.put(j, i, columnVectors[i][j]);
private Mat convertToMat(NRDistortionParams input)
Mat output = new Mat(1, 4, CvType.CV_64FC1);
output.put(0, 0, input.distortParams1, input.distortParams2, input.distortParams3, input.distortParams4);
To get an Quaternion, I used the code from this answer to translate the rvec.
I added the tvec to my CenterCamera.position to get the position in front of me, but it is still not that accurate - can someone give me a hint?
Jan, did you get anywhere with this? I am also trying to accurately place objects detected in RGB frames, using the intrinsic and distortion parameters, along with the head pose and RGB from head pose transformations… Positions are always a little off…
No worries - I worked on this some today, and now have pretty good positioning. Ignoring the distortion parameters was a part of it, but also I wasn’t quite storing the values in the projection matrix correctly - some reading up helped me out!
Hey - I tried to compress it into one method - not tested, but this is all the steps to go from a 2D pixel position to a 3D world position. I have used Vectors and Matrices in the System.Numerics namespaces rather than Unity’s own…
bool TryGetWorldPosition(Vector2 pixelPosition, float distance, out Vector3 worldPosition)
var glassesPose = UnityEngine.Pose.identity;
ulong timestamp = 0;
// Assume you are using the very latest frame so the image and
// glasses position are not too different! :)
if (NRFrame.GetFramePresentHeadPose(ref glassesPose, ref timestamp))
// Get the projection transform.
var projectionTransform = NRFrame.GetEyeProjectMatrix(out var _, 0.3f, 100f).RGBEyeMatrix.ToNumerics();
// Get the projection intrinsic values from the projection transform
var focalLengthX = projectionTransform.M11;
var focalLengthY = projectionTransform.M22;
var centerX = projectionTransform.M13;
var centerY = projectionTransform.M23;
var normalFactor = projectionTransform.M33;
// Normalize the center.
centerX = centerX / normalFactor;
centerY = centerY / normalFactor;
// Get the pixel coords on a scale between -1 and 1.
var pixelCoordinates = (new Vector2(pixelPosition.X / 1280f, 1 - (pixelPosition.Y / 720f)) * 2f) - new Vector2(1, 1);
// Create a directional ray using the principal point and the focal length.
var dirRay = new Vector3(
(pixelCoordinates.X - centerX) / focalLengthX,
(pixelCoordinates.Y - centerY) / focalLengthY,
// Multiple the ray by the distance you want.
var position = dirRay * distance;
// Get the RGB camera transform relative to the glasses.
var cameraToGlassesTransform =
// Get the glasses transform relative to the world.
var glassesToWorldTransform =
// Add these transform to create the full camera view transform
var cameraViewTransform = cameraToGlassesTransform * glassesToWorldTransform;
// Transform the position we have relative to the camera to make it relative to the world.
worldPosition = Vector3.Transform(position, cameraViewTransform);
worldPosition = Vector3.Zero;
Let me know if this works for you, or if it doesnt, and we can look again.