Hello everyone.

The Mixed Reality Forums here are no longer being used or maintained.

There are a few other places we would like to direct you to for support, both from Microsoft and from the community.

The first way we want to connect with you is our mixed reality developer program, which you can sign up for at https://aka.ms/IWantMR.

For technical questions, please use Stack Overflow, and tag your questions using either hololens or windows-mixed-reality.

If you want to join in discussions, please do so in the HoloDevelopers Slack, which you can join by going to https://aka.ms/holodevelopers, or in our Microsoft Tech Communities forums at https://techcommunity.microsoft.com/t5/mixed-reality/ct-p/MicrosoftMixedReality.

And always feel free to hit us up on Twitter @MxdRealityDev.

ZXing Unity Barcode DLL Issue

Jarrod1937Jarrod1937 ✭✭✭
edited November 2016 in Questions And Answers

Hello,
I was happy to see that Jimbohalo10 was able to get ZXing to work on the Hololens in some form via:
https://forums.hololens.com/discussion/2966/qr-scan-in-uwp-with-usb2-0-webcam-and-kinect-2

I am in the middle of attempting to write my own barcode reader, partly to customize it more, and partly to provide a codebase for the community that doesn't rely on a paid plugin. However, while portions of my code is working well, the second I add the ZXing Unity plugin I am unable to build the exported VS2015 project.
I am downloading the DLL's from the main ZXing website, dragging the "zxing.unity.dll" into Assets/Plugins folder in the Unity Editor to import the DLL. I build and export the project to VS2015. However, here is the build message I get:

C:\barcode_reader_test\buildf\barcode_reader_test\barcode_reader_test.csproj(285,9): error MSB3073: The command ""C:\barcode_reader_test\buildf\Unity\Tools\AssemblyConverter.exe" -platform=uap -lock="C:\barcode_reader_test\buildf\barcode_reader_test\project.lock.json" -bits=32 -configuration=Release -removeDebuggableAttribute=False -path="." -path="..\Players\UAP\x86\Release" "C:\barcode_reader_test\buildf\barcode_reader_test\Assembly-CSharp.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\Assembly-CSharp-firstpass.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\UnityEngine.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\UnityEngine.UI.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\UnityEngine.HoloLens.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\zxing.unity.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\UnityEngine.Networking.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\UnityEngine.VR.dll" "C:\barcode_reader_test\buildf\barcode_reader_test\Vuforia.UnityExtensions.dll"" exited with code 1.

Above that I get:

Failed to fix references for attribute System.ComponentModel.BrowsableAttribute
Failed to fix references for property System.Collections.Generic.IDictionary`2<ZXing.EncodeHintType,System.Object> ZXing.Common.EncodingOptions::Hints()
Failed to fix references for type ZXing.Common.EncodingOptions

This is using Unity 5.4.0f3 and the latest Vuforia for the Hololens.

«1

Answers

  • Jarrod1937Jarrod1937 ✭✭✭
    edited December 2016

    Figured it out. After importing the DLL into the project, in the asset manager navigate to it and click on it. Here there is an import dialog in the inspector, change this to the Editor and Standalone for the platforms, and change the OS to Windows. Under the standalone icon uncheck all options except 'Windows x86). Clear the build folder, rebuild and it seems to compile fine. No idea if it functions yet, will work on that next.

    Edit: At least in the editor via webcam, the barcode scanning is working.

  • Jarrod1937Jarrod1937 ✭✭✭
    edited December 2016

    Alright, after experimenting here are some insights:
    1.) Real time, constant barcode scanning is not likely. It was my hope to have a constant scanning system so that it can look for certain barcodes and maybe see them before the person does to better direct them. However, even with multi-threading, the ZXing library creates too big of a process hit and causes a large FPS drop.
    2.) Grabbing frames to process is also a bit too intensive for the system, and results in a lower frame rate and sometimes hitching/freezing for a very short period. I found the locatable camera to be too slow. I found the Unity's WebCamTexture to work better. I set the resolution to 1280x720 for testing (reason for higher res explained below), and set the requested frame rate to 1. The 1 is not proper, but Unity forces the webcam to match the nearest FPS supported, thus the 1 defaults the webcam to the lowest FPS it supports.
    3.) The barcode library, ZXing, works pretty well but it does not help that it seems that it largely focuses on the center of an image, which is an optimization that makes sense on their part. However, if the user is in a scanning mode, and their head is moving around doing some task while trying to scan this may be more difficult for them to achieve. My code takes the webcam texture, and splits it into quadrants, and each of these are scanned. This helps with the peripheral scanning.
    4.) Doing #3 I noticed the system now recognized barcodes from further away. This is because you're taking the larger image, cropping it, and essentially zooming in to a section for scanning. This zooming allows it to work from further away. Thus, I created a zoom section as well.
    5.) #3 and #4 result in multiple images to scan, so I abstracted out the barcode scanning to lists and arrays and abstracted that out to it's own class. You instantiate the class, passing a callback function in the constructor, and then call a barcode scanning function passing the list of images to scan.
    6.) Most importantly, I figured out how to get the darn thing to compile. You get the Unity ZXing DLL from their repository. From here, drag it into your project, then click on the DLL in the asset manager, under the Windows Store App Settings check the "Don't process" checkbox. Best I can tell, without this setting (which is off by default) Unity actually alters the DLL to what it thinks is needed to run on the platform... Sadly, it doesn't do a good job. Once this is checked, the exported codebase compiles perfectly for x86.

    My plans are to:

    • Convert all of my support code to use the barcode scanning as more of a manual scanner. The user will indicate they want to scan and it will enter scan mode and exit the scan mode when it gets a good scan.
    • The current code requires you to be horizontally aligned with the barcode. In my use case, this would be inconvenient. I have already experimented with a proof of concept perspective correction. You take the webCamTexture, apply it to a plane, then render to texture with a separate camera, and use this texture for the scanning operations previously mentioned. The goal of this extra step is that you can assume an average height of 5 feet 5 inches, assume the scanning surface is 90 degrees to the floor, then do a little bit of trig math to get the viewing angle from the headset viewing vector. Then you can simply shift render to texture plane angle to the complement of this, and the result should be a shifted perspective that is better for scanning.

    Here is my current basic barcode scanning class. Please note, in my case, I am doing 1D barcodes, QR codes may have their own tweaks that are required.

    public delegate void BarcodeCallB(Result data);
    `
    public class JCbarcodeReader
    {
    private List<Color32[]> imagesToScan = new List<Color32[]>();
    private List<int[]> imagesToScanRes = new List<int[]>();
    private BarcodeCallB callback;

    public JCbarcodeReader(BarcodeCallB callbackDelegate)
    {
        callback = callbackDelegate;
    }
    
    public void setScanData(List<Color32[]> imagesToScanS, List<int[]> imagesToScanResS)
    {
        imagesToScan = imagesToScanS;
        imagesToScanRes = imagesToScanResS;
    }
    
    public void barcodeDecode()
    {
        BarcodeReader codeReader = new BarcodeReader();
    
        codeReader.Options.PossibleFormats = new List<BarcodeFormat>();
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.CODE_128);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.CODE_39);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.UPC_A);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.UPC_E);
    
        codeReader.AutoRotate = true; // Rotates barcode image to try to align with vertical lines for proper scanning
        codeReader.TryInverted = false; // I believe this is for brightness inversion, so if you have inverted barcodes change to yes
        codeReader.Options.TryHarder = true; // Forces system to try harder, unclear what attempts are made, but does help
    
        Result data = null;
    
        for (var i = 0; i < imagesToScan.Count; i++)
        {
            data = codeReader.Decode(imagesToScan[i], imagesToScanRes[i][0], imagesToScanRes[i][1]);
    
            if(data != null)
            {
                break;
            }
        }
    
        if (callback != null)
        {
            callback.Invoke(data);
        }
    }`
    

    }

  • The callback function is defined as:
    Currently it's just setup for testing, but you can do what you want with the return data.
    public static void ResultCallback(Result data) { if (data != null) { GameObject txt = GameObject.Find("barcodeR"); txt.GetComponent<Text>().text = data.Text; } else { GameObject txt = GameObject.Find("barcodeR"); txt.GetComponent<Text>().text = ""; } }

    The above code requires the following includes:

    using ZXing; using ZXing.Common; using System.Collections.Generic;

    Class usage:
    Defined as a member of your main class, this way you only need to instantiate this once in your code.
    JCbarcodeReader readerInstance = new JCbarcodeReader(new BarcodeCallB(ResultCallback));

    Within your main image data manipulation function you pass the data to the class and attempt a scan via:
    The lists are added to at the same time so the data between each index is correlated. The res is an int array with 0 index being the width and 1 being height.
    ` List<Color32[]> imagesToScan = new List<Color32[]>();
    List<int[]> imagesToScanRes = new List<int[]>();

        readerInstance.setScanData(imagesToScan, imagesToScanRes);
        readerInstance.barcodeDecode();
    

    `

    My code for quadrant scanning:

    ` imagesToScan.Add(tex.GetPixels32());
    imagesToScanRes.Add(new int[] {tex.width, tex.height });

        int halfresW = (int)Mathf.Floor(resW / 2);
        int halfresH = (int)Mathf.Floor(resH / 2);
    
        /*
        Separate the texture into quandrants for individual scanning.
        3 | 2
        1 | 4
        */
    
        int cQ = 1, qMax = 4;
    
         while (cQ <= qMax)
         {
             Texture2D tempTex = new Texture2D(halfresW, halfresH);
             Color[] pixArr;
    
              int startX = 0;
              int startY = 0;
    
              if (cQ <= 2)
              {
                 startX = (cQ - 1) * halfresW;
                 startY = (cQ - 1) * halfresH;
              }
              else if (cQ == 3)
              {
                  startX = 0;
                  startY = halfresH;
              }
              else
              {
                  startX = halfresW;
                  startY = 0;
              }
    
              pixArr = tex.GetPixels(startX, startY, halfresW, halfresH);
    
              tempTex.SetPixels(pixArr);
              tempTex.Apply();
    
              imagesToScan.Add(tempTex.GetPixels32());
              imagesToScanRes.Add(new int[] { tempTex.width, tempTex.height });
               cQ++;
          }`
    

    I will share the center zoom code once I fully test it.
    The return data also includes image location data associated with where the barcode was found. Theoretically, even without the locatable camera data, you should be able to go from webcam image to worldspace coordinates by recording some data from the viewing ray at the time of image capture.

    Some more advanced things I tried:

    • I messed around with Vuforia. The software is extremely easy to use within Unity, so I would say they did an excellent job. I tried experimenting with image targets. The goal here is to get the software to recognize that, at least in my case, a barcode is a 4x2 or 6x4 white label with start and stop code 128 characters. If this could work, Vuforia would tell you where a barcode actually is on camera rather than you having to do a brute force search like the code above. The issue is that the image target platform is meant for matching specific images, not generic ones. So essentially I was playing around trying to get a specific image recognition system to specifically recognize a generic label... Not the easiest task. The initial results are promising, but the more specific detail it has the better it works, so I stripped out as much specific detail so it would be generic for all of our labels, but specific enough it could at least look for some sort of pattern. This is still in testing, I'll let you know if I can get this to work... or if any of you have some ideas for this, I'm all ears.
  • This forum just deleted a rather long and time consuming post... Can any mod look to see if it can be recovered?

  • Well, annoyingly, here is a summary of the post deleted:
    To get the thing to compile, get the Unity ZXing DLL from their site, drag to your project, and select it in the asset manager, under the windows store section check, 'Don't process' which will keep Unity from altering the DLL. The exported project then will compile for x86.

  • Here is my code for the barcode scanning class:

    `using ZXing;
    using ZXing.Common;
    using System.Collections.Generic;
    using System.Linq;
    using System.Threading;

    public delegate void BarcodeCallB(Result data);

    public class JCbarcodeReader
    {

    private List<Color32[]> imagesToScan = new List<Color32[]>();
    private List<int[]> imagesToScanRes = new List<int[]>();
    private BarcodeCallB callback;
    
    public JCbarcodeReader(BarcodeCallB callbackDelegate)
    {
        callback = callbackDelegate;
    }
    
    public void setScanData(List<Color32[]> imagesToScanS, List<int[]> imagesToScanResS)
    {
        imagesToScan = imagesToScanS;
        imagesToScanRes = imagesToScanResS;
    }
    
    public void barcodeDecode()
    {
        BarcodeReader codeReader = new BarcodeReader();
    
        codeReader.Options.PossibleFormats = new List<BarcodeFormat>();
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.CODE_128);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.CODE_39);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.UPC_A);
        codeReader.Options.PossibleFormats.Add(BarcodeFormat.UPC_E);
    
        codeReader.AutoRotate = true; // Rotates barcode image to try to align with vertical lines for proper scanning
        codeReader.TryInverted = false; // I believe this is for brightness inversion, so if you have inverted barcodes change to yes
        codeReader.Options.TryHarder = true; // Forces system to try harder, unclear what attempts are made, but does help
    
        Result data = null;
    
        for (var i = 0; i < imagesToScan.Count; i++)
        {
            data = codeReader.Decode(imagesToScan[i], imagesToScanRes[i][0], imagesToScanRes[i][1]);
    
            if(data != null)
            {
                break;
            }
        }
    
        if (callback != null)
        {
            callback.Invoke(data);
        }
    }
    

    }`

  • I am a bit annoyed, as it deleted a lot of explanation of the code and various tips... I also see the forum software can't handle large post sizes (which is why it needed to be split into two) and the code parsing of the tick marks is not correct as you can see. Now and in the future I'd like to give a clearer and more comprehensive explanation, so it would be helpful if these items could be fixed.

  • edited December 2016

    @Jarrod1937 Which version of Zxing are you using? I'm getting System.Io.FileLoad exceptions when trying to use the ZXing 0.14 unity build.
    I can get it to compile just fine by the way, it only breaks during runtime.

  • I get the same runtime error. The System.Core for .NET 3.5 can not be loaded.

    Has anyone any progress on that?

  • keljedkeljed ✭✭
    edited February 2017

    Thanks to this blog post I got it working:
    https://mtaulty.com/2016/12/28/windows-10-uwp-qr-code-scanning-with-zxing-and-hololens/

    Solution in short words: The usage of ZXing has to be wrapped in an own UWP class library.

  • Wow! Kudos @Jarrod1937, very nice work. Yes, the plugin system is a bit confusing the first time you try to do it and sometimes we have to do conditional compile statements to access the UWP code but still have behaviors compile under the editor. There is a lot of good knowledge up there. Any chance you're thinking about turning it into a blog post? Or maybe even a wrapped NuGet package if it makes sense? I can't promise, but if you write it up I might be able to ask for a retweet or something.

    Again, really nice work! I'm bookmarking this for future reference.

    Our Holographic world is here

    RoadToHolo.com      WikiHolo.net      @jbienz
    I work in Developer Experiences at Microsoft. My posts are based on my own experience and don't represent Microsoft or HoloLens.

  • MattFedoCSGMattFedoCSG ✭✭
    edited March 2017

    The DLLs are really the issue here and the way they are called. I have it working just fine in the Hololens and will write up a blog post that is a bit more specific and maybe cap a video or two and show the results. A HUGE thanks @mtaulty for that blog post which led me straight to the fix! Here is a hint, use the ZXing.winmd dll for sure. Ill be back soon!

  • At the beginning thank you @Jarrod1937 very much for that blog post. It took me several days to get my Qr-Scanner running on my hololens but sadly without success. Now I am very close to it. Thank you for that.

    @baskievits said:
    @Jarrod1937 Which version of Zxing are you using? I'm getting System.Io.FileLoad exceptions when trying to use the ZXing 0.14 unity build.
    I can get it to compile just fine by the way, it only breaks during runtime.

    I have exactly the same problem as @baskievits descripted. Has anyone found a solution for that?
    @MattFedoCSG I would be very happy about a more specific blog post from you. Do you have any results yet?
    Greetings from Germany :smile:

  • I recently commented on the main issue with @mtaulty's awesome wrapper. It is the fact that there is an Access Violation that happens when running on the device. I can get one scan done with a clean success, but if I try to scan a second time, I get a crash and the general Access Violation error. Its kind of odd, but I feel like something is trying to access something else that is not currently in memory.

  • I wonder if we switch to using the UnityEngine.VR.WSA.Webcam model to grab the texture, then use the ZXing.unity.dll would be any better on the device. Also, I need to try it with the ZXing.winmd dll. Remember to wrap your code in #if !UNITY_EDITOR to get it to compile. And on the Import Manager for the DLL in Unity, make sure you check "Dont Process" for the WSAPlayer settings.

  • I found a solution for my problem.
    I downloaded the QRcodeScanUWp54f3.rar from this blogpost:
    https://forums.hololens.com/discussion/2966/qr-scan-in-uwp-with-usb2-0-webcam-and-kinect-2
    Then analysed how he got ZXing to work on HoloLens.
    Just add the zxing.unity.dll twice with the following settings in the inspector.
    One to get access in your script (so #if !UNITY_EDITOR is not necessary):

    And then I added a second:

    This setting is deploying right and no runtime errors appear. To tick the "Don't process"-option is obviously not necessary in my case.
    To tell the truth i absolutely don't get it. If someone has an explanation why my Hololens can handle zxing when I add it twice, I would be thankful.
    Have a nice day :smile:

  • @Felix_96 : I will test this out. When you use the subfolders in the plugins folder it compiles differently. I never tried it with WSAPlayer as a folder and add the dll twice. The one you added in the general plugins folder is probably why the !Editor is not needed. Ill test and get back to you! Thanks!

  • MattFedoCSGMattFedoCSG ✭✭
    edited March 2017

    @Felix_96 : I have confirmed that the link you provided has the answer. I was able to scan multiple QRCodes with the Hololens, on the device. :+1: We are good to go! @Jimbohalo10 : thanks for the info to start this up.

  • MattFedoCSGMattFedoCSG ✭✭
    edited March 2017

    Ok peeps. Here you go. Multiple QR Code scanning with the holo. Please note that I had to cap this on a phone because the packages uses the cameras on the Holo, therefore you cannot screen cap, or stream from the hololens. BTW, I put down my phone so that I can do the tap gesture on the menu to start the scanner. I have it turing on and off with a button on the UI. I will get a UnityPackage together soon of all this. For now, enjoy : https://youtu.be/QSNYiUmwnNU

  • I managed to get some time to look at the code that's getting referenced here and I could reproduce the Access Violation which was happening down deep in Windows.Media.dll and relates to disposing a MediaFrameReader and a MediaCapture at the same time. I would find on HoloLens that this would crash every 2-3 scans. I could reproduce this on PC as well as HoloLens but less frequently.

    I've modified the code to try and workaround the issue as best I can as the crash is outside of my code.

    If people have success/failure with this, let me know and I'll take a look at it again.

  • @mtaulty I tested your updated version and I couldn't get to reproduce the AV exception, so everything seems fine. Thanks again I really couldn't figure out what was causing the issue.

  • Sorry for the late response, I didn't get any notifications and was busy with school and other work.
    I used the ZXing.Unity.DLL provided by ZXing on their repository. After telling Unity not to process the DLL I stopped getting those compilation errors.
    With the code I provided I was able to achieve 1D barcode scanning in a test application on the Hololens. The issue is that couldn't get it to work fast enough to be running in the background all the time.

    P.S. Thanks to which moderator brought my deleted post back. Maybe the repeated editing triggered some sort of spam filter... not sure.

  • @MattFedoCSG Since I am currently stuck on the multiple-barcode-scanning-thingy: I was wondering if you already managed to get a UnityPackage together for this?

  • Hi,
    I have downloaded a QRproject from git, but i am getting a error from "XR" framework. Since i am new for hololens development i am not able to figure it out why it is throwing a error.
    Here is the Link for the project: https://codeload.github.com/mtaulty/QrCodes/zip/master

  • Hi !
    I've been trying to use Zxing on the Hololens but as soon as the app starts on the device, I get a fileloadException :

    FileLoadException: Could not load file or assembly 'System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
    at ZXing.BarcodeReader..ctor()
    at Decoder.Start()
    at Decoder.$Invoke1(Int64 instance, Int64 args)
    at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64 args, IntPtr method)
    (Filename: Line: 0).

    I have the Unity 2018.1.0b5 beta version with Visual Studio 2017.
    The Building works perfectly.
    I tried the @Felix_96 configuration, as well as @Jarrod1937 's idea to tick the "don't process" on the dll here's a sight of the last way I tried :

    I also tried to use the Zxing.winmd but the dll can't be found then.
    Did I miss a step ? A file to download maybe ?

    Thanks for your help !

  • After many configurations tried, I finally managed to use Zxing on Hololens by switching the scripting Backend in Player Settings to Il2Cpp. I was using .net (for the scripting backend) which caused me most of the errors.

    If you don't have the IlCPP proposition, you might need to relaunch the Unity installer, making sure that "Windows Store Il2CPP " is checked.

    I also downgraded my version of unity to 2017 in order to use the Holotoolkit which wasn't working on my 2018 beta version.

    This way I don't need any specific settings to import Zxing in the Assets. Also, I didn't use Zxing.winmd but Zxing.unity.

    Tips : don't forget to unable the webcam and to deploy it in Visual Studio with "Release", otherwise it's going to slow down your app.
    If Visual Studio doesn't find your Zxing.unity file, right click on your project in the Solution's Explorer (of Visual Studio) : Add->Reference->Find your Zxing.unity.

  • @Holoholo Nice! Is it possible for you to share your unity package? I can't for the life of me get this to work on the HoloLens emulator (not even sure if it's possible to run it on the emulator at this point).

  • Hi @EleumLoyce ! I can't use it on the emulator either. I'm sorry I can't share my package as I'm working for a company.

  • Try opening the XdeCleanup.exe, in the emulator folder (microsoft Xde). It worked for me.
    Have a good day

  • Hello, I know that this topic is quite old but I have a solution to share to all that have matter with ZXing and Unity for scanning QRCode.

    It was that dumb that I nearly depress when I realize that we could do this in this way. ZXing already exist in Nuget, the packet manager for .Net application. So far, the matter is that Nuget package are not handle at all by Unity. So, we could do both solution in same time, and it work perfectly :

    1) Import zxing.unity into your Unity assets package under the folder Plugins, and (assuming that you use the recommanded version of Unity for hololens, 2017.4 LTS) set it import parameter for both WASP and Editor, and set UWP on the WASP parameter. In that way, both Editor and Visual Studio won't yield error and you could run simple test in the Editor.

    2) When times come to deploy, you'll experienced crash due as explain before. In order to get rid of that, you could just delete zxing.unity in the list of reference of your Visual project (compile with Unity before, make sense), and import ZXing.net from Nuget repo to replace it. It work pretty well on my project, and avoid wrapper and every stuff like that.

    I'm really sorry for my crappy english, I hope I was clear about.
    Hope it could help you.
    If you want more detail, contact me to thibaud.vegreville@gmail.com. I don't have finish my actual project, so I can't send you sources, but I'll be glad to make a proper blog post about it, and a full tutorial about how to get things work with ZXing.

    Have a good day.

Sign In or Register to comment.