Spatially anchoring a postion based on Camera position

@neerajwadhwa
I am making an AR short film that plays a movie taking into account the objects in my surrounding. For eg spawning a hologram onto a table, wall,etc. It is composed of multiple Unity scenes.

The idea is to use spatial understanding in the first scene, store all the positions globally so that the value is accessible in all the further scenes and then disable the spatial understanding and spatial mapping once all the required data (position of wall, floor, etc) is saved.

The placement of holograms work perfectly fine based on the saved positions provided I am standing at the same orientation and position at the time the spatial understanding was disabled. The moment I move away and start seeing here and there, the positions where the holograms get spawned become haywire. The reason that I see is because all the positions that were saved were based on the orientation of Camera (the origin) just at the time of disabling the Spatial Understanding. Now that it has changed it spawns objects relative to that.

I thought of using World Anchor for solving this issue but World Anchor just anchors a Hologram to a point in space whereas in my case I need multiple holograms to spawn at that point based on my changing scene.

Can I use the World Anchor logic somehow to tackle this issue or is there a better way to handle this?