How to Use OpenCV in Unity (Example Project)

This is a post that many people have requested recently. I’m going to describe how you can use OpenCV in Unity. Of course we’ll be using official OpenCV libraries and not any assets or existing plugins.  For those of you who aren’t familiar with the subject, Unity is a very popular game engine which allows building games, apps and so called experiences with much ease. Unity allows some modified form of JavaScript and also C# for its scripting. In this example project I’ll use C# since that’s the language I’m familiar with but it shouldn’t be hard to adapt this to JavaScript (but you’re gonna do it yourself if needed, sorry) so let’s start.

Here’s an screenshot of the project shared in this post. Hopefully if you follow all the steps correctly you’ll have it running on your PC and Mobile.





First and foremost, as you might already know, OpenCV or at least standard OpenCV uses C++ and not C# so we’ll need to somehow call OpenCV C++ functions from C#. Think about it like this, if you already know how to use OpenCV then you can create a very simple library file (.dll, .so, .a and so on) in C++ which uses OpenCV the normal way. Then export those functions from the library and call them from C#. But before that we need to define a struct that we’ll use to receive images and video frames from Unity C# code.

struct Color32
{
uchar r;
uchar g;
uchar b;
uchar a;
};
And here’s how we need to define our exported function in the library. In its header file:
extern “C”
{
__declspec(dllexport) void processImage(Color32* raw, int width, int height);
}
Note that this is the case for Windows. If you use some cross platform tool like Qt Creator you will have something like CVTESTSHARED_EXPORT instead of “__declspec(dllexport)” which will automatically adapt to Linux, macOS, iOS and so on.

And in source file we’ll have this which is the actual function and image processing part of the project:

extern “C”
{

void processImage(Color32* raw, int width, int height)
{
using namespace cv;
using namespace std;

Mat frame(height, width, CV_8UC4, raw);

// Process frame here …
// Try imshow(“frame”, frame); for example …
}

}

Notice how we’re creating an OpenCV Mat using an array of Color32 struct we created. It’s basically a 32-bit image with R, G, B and Alpha channels because that’s how unity will send over images and video frames:

Mat frame(height, width, CV_8UC4, raw);

Now build your library and get your library file. (That would be .DLL for Windows, .SO for Android and so on)

You can also grab the source code for library from here. (It needs OpenCV and is created using Qt Creator but it doesn’t use any Qt library. Let me know if you face any difficulties building/modifying the library to fit your needs)





And next for the Unity part of the project, we’re going to use WebcamTexture class to easily access camera on almost any platform. (Note that from here on codes are in C# not C++ though they’re very similar)

public WebCamTexture webcam;
Now the most important question is “How to pass images from Unity to OpenCV?” Well WebcamTexture provides a very simple method to get the raw image data. Similar functions exist for single images and video and so on. As you saw above we’ll use that raw data to create a Mat in OpenCV. Here’s how we’re going to get frames from the camera and pass it to our C++ code and library:
if (webcam.isPlaying) {
Color32[] rawImg = webcam.GetPixels32 ();
System.Array.Reverse (rawImg);
processImage (rawImg, webcam.width, webcam.height);
}
Of course we’re just calling it now. We need to actually define our project in the C# code in Unity, here’s how:
[DllImport(“CVTest”, EntryPoint = “processImage”)]
public static extern void processImage(Color32[] raw, int width, int height);
If you get a compiler error for not knowing what DllImport is, you should add this to the top of your C# code:
using System.Runtime.InteropServices;
You can find the whole Unity project along with all source codes mentioned from this link.
Here are a few final (and very important) notes about using OpenCV in Unity.
1. You need to copy your C++ library to Plugins folder in Unity project.
2. You need to deploy OpenCV libraries used by your project along with the library itself.
You might want to read about Plugins in Unity and also make sure to comment below if you face any issues with the source code or have any questions.




13 Replies to “How to Use OpenCV in Unity (Example Project)”

  1. Hi, thanks very much for the tutorial, it is very usefull.

    Could you tell me how to convert from Mat image to Color32 struct so I can send an image from OpenCV to Unity?

    1. The answer to your question is actually in the post. I’ll explain it a little bit more.

      We have the following:
      Mat frame(height, width, CV_8UC4, raw);

      This is how we created a Mat using Color32 struct.

      One way of passing any other Mat to Color32 is to make sure 1. Your Mat has the type 8UC4 and 2. Pass the data of Mat over to Unity again as described above.

      Note that you need to be careful about memory management here, so I would really suggest (if it’s possible) just send an empty Color32 (with the required width and height from Unity to OpenCV and then fill it in your OpenCV code.
      Since you have access to its pointer, the operation is not going to be too slow anyway.

  2. Hi Mr. Ahmadi,
    Thank you very much for this useful tutorial,
    I want to use exported frame from OpenCV as background of the unity scene in android platform, in other word, I want to give a frame from Unity and modify it in openCV and then use it in a Augmented reality application in Unity engine.
    I guess if I pass the frame via pointer, all of opencv changes will be effected in the output background, but I need some more guidance about it.
    Do you think there is a better way to achieve better result?

    1. Hi Mohammad,
      There’s really nothing else worth mentioning, except the fact that you need to be careful with memory management especially when casting data types to each other, for use in OpenCV or Unity.

  3. Hi, thank you a lot for your work.
    I am having trouble using your solutions, Unity crashes when I run it in play mode.
    I have this error in the crash repport :
    Unity Editor [version: Unity 5.3.0f4_2524e04062b4]

    opencv_highgui2413d.dll caused an Access Violation (0xc0000005)
    in module opencv_highgui2413d.dll at 001b:34f77648.

    Error occurred at 2017-11-15_124706.
    C:\Program Files\Unity\Editor\Unity.exe, run by Nathan.
    74% memory in use.
    3326 MB physical memory [845 MB free].
    0 MB paging file [2942 MB free].
    2048 MB user address space [857 MB free].
    Read from location 00220065 caused an access violation.

    If anyone has a solution, I will be please to have it.
    Thank you

      1. Hi,
        Actually, I was using some functions implemented within the same cpp file that the final function that I exported. When I rewrote the function to use only opencv function inside the function I export, it was running. I think that I shoulde export every function that I wrote even if it is just an intermediate one. I was the first time I build my own dll so I am not sure how it really works. Any way, it is running now. Again thank you for making that open.

  4. Thanks for the great tutorial. I tried this approch to make a simple imshow() function. but the result is completely black. I did exactly as per ur tutorial…

    1. I would suggest you to try the source codes which I have put there in the post. Download and see if they work then we can check out the issue. Or be more specific about the issue you have faced.

      1. Thanks for the reply @Amin,
        Yes that code worked well. The reason that resulted a complete black image for me is, I call my dll fuction in side strat(){} function of unity, which only execute once. So I found in my case, that the typical 1st image acquired to dll will mostly a black one. So I created another function inside dll to check if the true image is acquired before processing by checking for feature points. Now that is perfectly resolved. Thank You very much for this tutorial.

        And I’d like to ask another question. Currently I’m trying to write an augmented reality application by processing the image inside a dll and render the webcamtexture frames to a game object simultaneously. When I implemented, it is too slow. Is there any way to run the webCameraTexture feeds on to a game object without making a significant slow effect for such an application.

        1. The way you are doing it should be the fastest with the logic provided in this tutorial and the way OpenCV and Unity interact. I would say first try to see if the processing and displaying codes are performant or not. Try benchmarking the image processing code separately from Unity. Maybe your image processing is what makes the whole thing slow. Try to narrow down the issue first. Let me know if this helps.

          1. Thanks for the reply again @Amin,

            I’m actually using the dll to compute the pose of camera using feature points. When I run that code in my IDE, it runs with 20 FPS, which is faster. And I’m using the results of transformation to move the camera in unity space.

Leave a Reply

Your email address will not be published. Required fields are marked *