Identify real world objects defects and differentiate between them

ctsholo

Hello guys,
How to identify between several real world objects which looks alike through Gaze? . For example - say there are 100 monitors placed adjacent or with some space between each other. So how to differentiate between Monitor 1 and and Monitor 2? Also, say I have a data of defects in the backend, so when I gaze at Monitor 1 how to highlight the defect in that particular area on Monitor 1?

jbienzms

Computer vision is amazing technology, but it does require significant resolution for doing things like defect detection. Especially in a scenario like @ctsholo describes when there are 100 nearly unique items in a row. The resolution of the HoloLens camera is pretty good, but probably not up to this task using a CV approach.

Instead, I would recommend an approach where a specialist uses the HoloLens to do the initial defect identification. They could simply look at the equipment, find the defect using gaze and mark it using voice or gesture. When a defect is found, its position would be captured in relation to the nearest World Anchor, and new World Anchors would be created roughly every 30 feet down the line (per the recommended guidance).

Later, anyone could come back and load the World Anchors and the defects related to them. This would align the defects correctly in the world and allow the follow-up users to see the same locations and defects on the equipment line.

I don't know if that meets your scenario needs, but that's one possible implementation.

jbienzms

Yes this is possible, you just need to translate the coordinates. If I'm understanding you correctly, your defect locations are relative to the object itself, correct? So we could say that the defect coordinates are in object space. We just need to get from object space to world space.

The way I would do this is to create "work areas" for each location that a monitor can sit on in the assembly line. These work areas would be 3D volumes (or bounding boxes in Unity terms). I would probably create these by walking down the assembly line and using the HoloLens to mark the corners around each area and just use a predetermined maximum height. In Unity, each work area would be a GameObject and would act as a parent container where child holograms could live. Each work area would have its own name or ID and each work area would be added as a child of the nearest World Anchor.

It's important to understand how World Anchors work and test them in your space. I would probably create one about every 30 feet or so down the assembly line. Once they are created they can be saved on the device and just loaded the next time the application runs. They don't need to be created every time.

Adding each work area as a child of the nearest World Anchor ensures that every time the application loads, the work areas will be aligned correctly with the physical space.

Now we have individual "islands" of space in the physical world that represent a working volume for each monitor. From here, all you need to do is load the data from your database, create holograms, and place them as a child in one of the work spaces. Because we have correctly setup islands of "local space" in the physical world, we can position holograms in object space and see them correctly in world space.

Hope that helps!

utekai

Check out computer vision APIs like Microsoft's Cognitive Services or Vuforia.

jbienzms

Computer vision is amazing technology, but it does require significant resolution for doing things like defect detection. Especially in a scenario like @ctsholo describes when there are 100 nearly unique items in a row. The resolution of the HoloLens camera is pretty good, but probably not up to this task using a CV approach.

Instead, I would recommend an approach where a specialist uses the HoloLens to do the initial defect identification. They could simply look at the equipment, find the defect using gaze and mark it using voice or gesture. When a defect is found, its position would be captured in relation to the nearest World Anchor, and new World Anchors would be created roughly every 30 feet down the line (per the recommended guidance).

Later, anyone could come back and load the World Anchors and the defects related to them. This would align the defects correctly in the world and allow the follow-up users to see the same locations and defects on the equipment line.

I don't know if that meets your scenario needs, but that's one possible implementation.

ctsholo

@jbienzms Thank you for your response. You wrote

They could simply look at the equipment, find the defect using gaze and mark it using voice or gesture

Seems I did not write it clearly.The specialist is not marking the defects. Instead the defect position(x,y) is already recorded through some other platform during the manufacturing of the product(Monitors in this example) so when the Specialist looks at the monitor the defect position is pulled from that platform and the exact (X,Y) position is highlighted on the real world monitor. Is this implementation possible? How can HoloLens translate the defect (X,Y) to real world monitor's (X,Y)

jbienzms

Yes this is possible, you just need to translate the coordinates. If I'm understanding you correctly, your defect locations are relative to the object itself, correct? So we could say that the defect coordinates are in object space. We just need to get from object space to world space.

The way I would do this is to create "work areas" for each location that a monitor can sit on in the assembly line. These work areas would be 3D volumes (or bounding boxes in Unity terms). I would probably create these by walking down the assembly line and using the HoloLens to mark the corners around each area and just use a predetermined maximum height. In Unity, each work area would be a GameObject and would act as a parent container where child holograms could live. Each work area would have its own name or ID and each work area would be added as a child of the nearest World Anchor.

It's important to understand how World Anchors work and test them in your space. I would probably create one about every 30 feet or so down the assembly line. Once they are created they can be saved on the device and just loaded the next time the application runs. They don't need to be created every time.

Adding each work area as a child of the nearest World Anchor ensures that every time the application loads, the work areas will be aligned correctly with the physical space.

Now we have individual "islands" of space in the physical world that represent a working volume for each monitor. From here, all you need to do is load the data from your database, create holograms, and place them as a child in one of the work spaces. Because we have correctly setup islands of "local space" in the physical world, we can position holograms in object space and see them correctly in world space.

Hope that helps!