Barcode/QR code detection

holodoctor · July 2016

Hi,

I will be working on a project where QR/barcode scanning is needed. I used barcode scanning library before (on Windows Phone). But, the problem I see is that, when you use HoloLens, how do you even know where the barcodes are.

In mobile apps, typically the device is in close proximity to the barcode image. (A typical app displays a "window" through which a barcode can be viewed/read, which constrains the barcode reading space.)

In a use case I'm thinking about, that may not be generally feasible. A user wearing HoloLens may view a barcode(s) in a distance, and the app needs to recognize it as a barcode first, and subsequently scan it.

Is there a way to accomplish this without resorting on a full-blown CV approach?

Any thoughts?

Jimbohalo10 · July 2016

See this thread for a relevant discussion Is it possible to "see" other apps and use their information for my own?

Have a look at QR software in Unity Asset Store, there is a lot, try and avoid samples with *.dll files as these cannot transfer to HoloLens also Windows Universal Program Examples library has sample barcode and OCR reader.
The hardest part and killer is Locate Camera code and forcing it to a specific place. Trying to make it focus did not work on webcam or HoloLens

Jarrod1937 · July 2016

If you're using Unity and you find a good 1D and 2D barcode solution, or roll your own, let me know. I haven't progressed to that point in my work application yet, but it would be a time saver. If you roll your own I can probably even workout a payment.
The best bet is this one from the Unity asset store:
https://www.assetstore.unity3d.com/en/#!/content/16941

Supposedly is non-native and works with UWP.

In regards to your main question, a good library usually allows you to disable all barcode types you're not interested scanning, these can help reduce false positives. From here, you can try an optimized brute force method. That is, grab maybe 2-5 frames a second, attempt to process them via the barcode library, if you get any results back, verify they somewhat meet the expected scan data format, if so good, if not reject. Then you can only enable this scanning in contexts where it needs to be. I am going to experiment with this method, as it is brute force, but may be cheaper than complex CV computations.
In regards to CV, luckily the Hololens provides some data already that is normally computationally complex. That is, you have the spatial data that you can use for doing perspective correction on the grabbed frames. Most barcode libraries either do little perspective correction or are expensive to do it. You should be able to map a pixel/texel to the spatial mesh, maybe average a few points due to the meshes lower resolution, and correct the frame for barcode processing.
Again, I haven't got around to this portion yet, so if you develop something I'd be highly interested in it.

Jimbohalo10 · July 2016

@Jarrod1937 @holodoctor How about this free code Generate Barcode and QR code in Windows Universal app

holodoctor · July 2016

Hi @Jarrod1937 @Jimbohalo10 Thanks for the input. My primary concern was how to locate a barcode from an unmarked scene. After giving some thought, I realized it might be a bit too ambitious at this point to try to do that. (For example, a barcode image might be at an angle at a distance.) I am still thinking about it, but for now, instead, I will just direct the user to view a barcode in a particular way so that the app can easily identify/detect the barcode image. Not ideal, but definitely much easier.

I am just starting my HoloLens dev, and I haven't tried this LocatableCamera API yet, but I did use one of those barcode scanning libraries on WinRT before (WP8.1 not UWP). Either ZXing.Net or ZXing.Net.Mobile, cannot remember at this point. Thanks for the pointers to Barcode libraries on Asset Store. I quickly looked at some of them, and they all say they support iOS and Android. But, no explicit mention of Universal Windows.

In any case, I'll definitely keep you guys posted if there is any development.

Thanks,
~h

Jarrod1937 · July 2016

@Jimbohalo10 said:
@Jarrod1937 @holodoctor How about this free code Generate Barcode and QR code in Windows Universal app

Yep, I've used ZXing before, it's not too bad though requires a bit more tweaking than a commercial library (but it's free!). I primarily need to scan 1D barcodes due to the Hololens fitting in with our current workflow and more redundancy in industrial environments. QR codes have some parity I believe but 1D basically has the entire vertical region as redundancy, which helps read even after dust/dirt/scratches/tearing...etc.
The only thing is that I want to avoid figuring out how to program a Unity plugin that integrates the barcode reading, so I can more quickly get out a prototype. However, I will if need be.

Jarrod1937 · July 2016

@holodoctor said:

Thanks for the pointers to Barcode libraries on Asset Store. I quickly looked at some of them, and they all say they support iOS and Android. But, no explicit mention of Universal Windows.

In any case, I'll definitely keep you guys posted if there is any development.

Thanks,
~h

Just want to make sure you read this in the link I provided, "WP8, Windows Universal Apps," most of the assets target Android and IOS, but that one also targets Windows. I definitely look forward to your progress.

holodoctor · July 2016

@Jarrod1937 Yes, it does mention Windows Universal. I'll give it a try. Thanks, ~h

Monkey · July 2016

@holodoctor : Did you have any luck with this? I'm also interested in finding a QR or barcode reading solution for a HoloLens app. However I have no experience with QR or barcode reading in the past, so I will be starting fresh. Did the Unity plugin that was recommended above work out for you or did you find another solution?

Jarrod1937 · July 2016

@Monkey said:
@holodoctor : Did you have any luck with this? I'm also interested in finding a QR or barcode reading solution for a HoloLens app. However I have no experience with QR or barcode reading in the past, so I will be starting fresh. Did the Unity plugin that was recommended above work out for you or did you find another solution?

This was not that long ago (only a few days). I would give him a chance to work on it first, haha. I think barcode scanning is common, so hopefully the community will share as they develop. I'm still working on some software for Win CE6 (some outdated mobile scanners...), after that I'll be putting more time forward toward the Hololens. If some solutions aren't available at that time, I will certainly share what I find and/or develop.

holodoctor · July 2016

@Monkey @Jarrod1937 I haven't had a chance to work on it yet (It's been "only a few days" after all ), but I will keep you guys posted. ~h

sergiy · August 2016

Is any progress with barcode reading? I'm interested too in scanning 1D&2D barcodes by using HL.
@Jarrod1937, if you need to do proof of concept, you can look at some wearable on finger bluetooth scanner which can be paired with HoloLense. I understand that it's not perfect solution and it's little bit out of current topic, but it can be workaround for you.

Anders · August 2016

I'm currently working on an app using AR Toolkit for HoloLens. This means I am able to locate and track 2D markers using the camera. I only tried it with square markers but I guess using it with barcodes would be quite similar: https://artoolkit.org/

Generalkidd · August 2016

I've tried most of the pre-existing barcode plugins on the Unity store but none of them have worked for me so far. They work fine if I deploy to a smartphone but I'm having trouble getting them to work on the HoloLens. I'm not sure if I'm implementing it wrong or something, but so far I've been unsuccessful getting a basic barcode reader to work on the HoloLens. Has anyone else had more success yet?

avatar · August 2016

@Generalkidd did you try this plugin https://www.assetstore.unity3d.com/en/#!/content/56083 ? looks promissing
@Anders could you add some clues how to use artoolkit marker detection in HoloLense?

Anders · August 2016

Hi,
The existing dll:s in AR Toolkit didn't work for me, the error messages weren't very helpful for me but I think they are only compatible with 8.1. Basically I recompiled the toolkit from scratch, including the libraries it depended on, such as libjpeg.

When you have the toolkit up and running, you also need to adapt it for Hololens positioning and calibrate it, otherwise it won't work too well.

wilfal · August 2016

@Anders said:
Hi,
The existing dll:s in AR Toolkit didn't work for me, the error messages weren't very helpful for me but I think they are only compatible with 8.1. Basically I recompiled the toolkit from scratch, including the libraries it depended on, such as libjpeg.

When you have the toolkit up and running, you also need to adapt it for Hololens positioning and calibrate it, otherwise it won't work too well.

I am starting my Master Thesis project and I am looking for ways to make Hololens detect a stationary object and then place a hologram on that object. Your work sounds very applicable to what I want to accomplish. My Thesis does not focus on the technical aspects but rather on the testing/verifaction of the implementation I am about to create Would you care to share your work in any way Anders? We could talk more in PM

mark_grossnickle · August 2016

@wilfal Check out Vuforia... it was released for hololens last week: https://developer.vuforia.com/downloads/beta

It can do VuMarks (which are QRCodes) and images. It won't be able to do stationary object recognition though. But perhaps you could put a VuMark on your object?

stephenhodgson · August 2016

I believe the HoloToolkit also has april tags (QR codes) built into it.

DocStrange · August 2016

Has anyone implemented QR code reading successfully? If so, would you be willing to share code/tutorial?

chirag_icpl · November 2016

I am also looking for way to scan QR/Barcode using hololens. Please provide sample code if anybody has been succeded to do so.

Jimbohalo10 · November 2016

I did make a similar project that works in HoloLens Emulator and UWP executable for webcams
QR Scan in UWP with USB2.0 Webcam and Kinect 2

Kinect 2 is the nearest relative to HoloLens.

Although there is Locate Camera in Unity . You only get static input as one picture. Even in the new Windows.MediaCapture library the HoloLens camera does not seem to be able to stream like a webcam. HoloLens camera is reading Mixed Reality and Augmented Reality.

Jimbohalo10 · November 2016

Hi @holodoctor @DocStrange @Jarrod1937 @stephenhodgson et all,
There is a new app in the Windows store , which has the compatible with HoloLens sign.
It would be great if someone could test this kwiQR on a real HoloLens device

The author sometimes works for Microsoft UK https://twitter.com/mtaulty and the app is FREE!

CaptainMM · November 2016

Yes, it does work on Hololens. The app is currently processing around 6-8 frames per second but does scan from a pretty far distance. This leads me to assume they aren't using the webcam and are rather taking async photos then processing those. I'm definitely interested in what the used to create this as well.

@Jimbohalo10 said:
Hi @holodoctor @DocStrange @Jarrod1937 @stephenhodgson et all,
There is a new app in the Windows store , which has the compatible with HoloLens sign.
It would be great if someone could test this kwiQR on a real HoloLens device

The author sometimes works for Microsoft UK https://twitter.com/mtaulty and the app is FREE!

KingPepito · December 2016

Hi, I'm new to the forum and I'm currently working on a HoloLens project which involved Qr code detection. I achieved to use ZXing library with the HoloLens, I used a class to take a capture from the camera and then detect the code in it. So no continuous detection, but concerning the QR code reading what I did do the job quite easily.

For that mean I built a class "QRDecoder" which is used while taking a capture with a CaptureManager class (like in this locatable Camera tutorial https://developer.microsoft.com/en-us/windows/holographic/locatable_camera_in_unity).

Here is the CaptureManager class:

`
using UnityEngine;
using UnityEngine.VR.WSA.WebCam;
using System.Linq;
using System.Collections.Generic;

public class CaptureManager {

// Use this for initialization
PhotoCapture photoCaptureObject = null;

QRDecoder Decoder = new QRDecoder();

int cameraWidth;
int cameraHeight;

public string folderPath;

public void takeCapture()
{
    Debug.Log("MRC captured");
    PhotoCapture.CreateAsync(false, OnPhotoCaptureCreated);
}

protected void OnPhotoCaptureCreated(PhotoCapture captureObject)
{
    photoCaptureObject = captureObject;

    Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();

    CameraParameters c = new CameraParameters();
    c.hologramOpacity = 0.0f;
    c.cameraResolutionWidth = cameraResolution.width;
    cameraWidth = cameraResolution.width;
    c.cameraResolutionHeight = cameraResolution.height;
    cameraHeight = cameraResolution.height;
    c.pixelFormat = CapturePixelFormat.BGRA32;

    captureObject.StartPhotoModeAsync(c, false, OnPhotoModeStarted);
}

void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
    Debug.Log("Stopping");
    if (photoCaptureObject != null)
    {
        photoCaptureObject.Dispose();
        photoCaptureObject = null;
    }   
}

private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
{
    if (result.success)
    {  
        photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToByte);
    }
    else
    {
        Debug.LogError("Unable to start photo mode!");
    }
}

void OnCapturedPhotoToByte(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{

    if (result.success)
    {
        List<byte> imageBufferList = new List<byte>();

        // Copy the raw IMFMediaBuffer data into our empty byte list.
        photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);


        // In this example, we captured the image using the BGRA32 format.
        // So our stride will be 4 since we have a byte for each rgba channel.
        // The raw image data will also be flipped so we access our pixel data
        // in the reverse order.
        int stride = 4;
        float denominator = 1.0f / 255.0f;
        List<Color> colorArray = new List<Color>();
        for (int i = imageBufferList.Count - 1; i >= 0; i -= stride)
        {
            float a = (int)(imageBufferList[i - 0]) * denominator;
            float r = (int)(imageBufferList[i - 1]) * denominator;
            float g = (int)(imageBufferList[i - 2]) * denominator;
            float b = (int)(imageBufferList[i - 3]) * denominator;

            colorArray.Add(new Color(r, g, b, a));
        }

        colorArray.Reverse();

        // Converting the List<byte> to a bute array
        byte[] imageBufferArray = imageBufferList.ToArray();

        // Now we could do something with the array such as texture.SetPixels() or run image processing on the list

        #if !UNITY_EDITOR
        // Here use the Decoder class to decode the Qr code
        Decoder.DecodeFromByteArray(imageBufferArray, cameraWidth, cameraHeight);
        #endif

    }
    // Stopping the photo mode
    photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}

}
`

KingPepito · December 2016

Here is the QRDecoder class:

`using UnityEngine;
using System;
using ZXing;

public class QRDecoder : MonoBehaviour {

if !UNITY_EDITOR

// create a barcode reader instance
BarcodeReader barcodeReader = new BarcodeReader { AutoRotate = true };

 //Used to detect a QR code in a byte array image, return the decoded text
 public void DecodeFromByteArray(byte[] arrayByte, int Width, int Height)
 {

     try
     {
        // Locatable camera from the HoloLens use BGRA32 format in the MRCManager class 
        Result TextResult = barcodeReader.Decode(arrayByte, Width, Height, RGBLuminanceSource.BitmapFormat.BGRA32);
        //If QR code detected 
        if (TextResult != null)
        {
            Debug.Log("Result decoding: " + TextResult.Text);

            //
            // Do what you want with the result here
            //
        }
        //If QR code not detected
        else
        {
            Debug.Log("No QR code detected");
        }

    }
        //If error while decoding
     catch(Exception e)
     {
        Debug.Log("Exception:"+e);
     }

 }
#endif

}`

I hope it will help, this is certainly improvable but atm it's working well.

mtaulty · December 2016

The kwiQR app mentioned above is a fairly basic one that I built for Windows 8.0 back in 2012 and then ported to Windows 10 (PC and Phone) last year. I made notes about doing that port at the time and you can see how the camera capture code works (sitting on top of ZXing) in this blog post and it essentially takes video preview frames from the camera.

I've yet to try this app on HoloLens so glad to hear if it works but it sounds like the processing rate is quite slow. If it helps someone to have the code for what I have then I'm happy to share it (mail me and let me know) and I could probably turn it into a library although I haven't thought about it from the point of view of Unity at all (yet! :-)).

KingPepito · December 2016

Please let us know when you have try on the HoloLens Device, sounds good to access a solution with better performance .

mtaulty · December 2016

Sure, I wrote up my efforts around this [here](https://mtaulty.com/2016/12/28/windows-10-uwp-qr-code-scanning-with-zxing-and-hololens/) - not sure if it's what people were looking for but if it's helpful in any way then feel free to make use of it.

Jimbohalo10 · December 2016

Hi @mtaulty Its an excellent write-up, good videos as well. The adding of voice control is a brilliant idea. This gets round many of the problems of having a menu and buttons

baskievits · December 2016

Hi, perhaps someone here knows why I'm getting a FileLoadException when trying to use ZXing on a HoloLens device? The error I get is that it can't load file or assembly system.core version 3.5.0.0. I'm using the unity build of the .NET ZXing project with .NET 3.5 as a target platform.

Barcode/QR code detection

Answers

if !UNITY_EDITOR