How to Use OpenCV in Unity (Example Project)

Posted on Posted in OpenCV Tutorials, Tutorials

This is a post that many people have requested recently. I’m going to describe how you can use OpenCV in Unity. Of course we’ll be using official OpenCV libraries and not any assets or existing plugins.  For those of you who aren’t familiar with the subject, Unity is a very popular game engine which allows building games, apps and so called experiences with much ease. Unity allows some modified form of JavaScript and also C# for its scripting. In this example project I’ll use C# since that’s the language I’m familiar with but it shouldn’t be hard to adapt this to JavaScript (but you’re gonna do it yourself if needed, sorry) so let’s start.

Here’s an screenshot of the project shared in this post. Hopefully if you follow all the steps correctly you’ll have it running on your PC and Mobile.





First and foremost, as you might already know, OpenCV or at least standard OpenCV uses C++ and not C# so we’ll need to somehow call OpenCV C++ functions from C#. Think about it like this, if you already know how to use OpenCV then you can create a very simple library file (.dll, .so, .a and so on) in C++ which uses OpenCV the normal way. Then export those functions from the library and call them from C#. But before that we need to define a struct that we’ll use to receive images and video frames from Unity C# code.

struct Color32
{
uchar r;
uchar g;
uchar b;
uchar a;
};
And here’s how we need to define our exported function in the library. In its header file:
extern “C”
{
__declspec(dllexport) void processImage(Color32* raw, int width, int height);
}
Note that this is the case for Windows. If you use some cross platform tool like Qt Creator you will have something like CVTESTSHARED_EXPORT instead of “__declspec(dllexport)” which will automatically adapt to Linux, macOS, iOS and so on.

And in source file we’ll have this which is the actual function and image processing part of the project:

extern “C”
{

void processImage(Color32* raw, int width, int height)
{
using namespace cv;
using namespace std;

Mat frame(height, width, CV_8UC4, raw);

// Process frame here …
// Try imshow(“frame”, frame); for example …
}

}

Notice how we’re creating an OpenCV Mat using an array of Color32 struct we created. It’s basically a 32-bit image with R, G, B and Alpha channels because that’s how unity will send over images and video frames:

Mat frame(height, width, CV_8UC4, raw);

Now build your library and get your library file. (That would be .DLL for Windows, .SO for Android and so on)

You can also grab the source code for library from here. (It needs OpenCV and is created using Qt Creator but it doesn’t use any Qt library. Let me know if you face any difficulties building/modifying the library to fit your needs)





And next for the Unity part of the project, we’re going to use WebcamTexture class to easily access camera on almost any platform. (Note that from here on codes are in C# not C++ though they’re very similar)

public WebCamTexture webcam;
Now the most important question is “How to pass images from Unity to OpenCV?” Well WebcamTexture provides a very simple method to get the raw image data. Similar functions exist for single images and video and so on. As you saw above we’ll use that raw data to create a Mat in OpenCV. Here’s how we’re going to get frames from the camera and pass it to our C++ code and library:
if (webcam.isPlaying) {
Color32[] rawImg = webcam.GetPixels32 ();
System.Array.Reverse (rawImg);
processImage (rawImg, webcam.width, webcam.height);
}
Of course we’re just calling it now. We need to actually define our project in the C# code in Unity, here’s how:
[DllImport(“CVTest”, EntryPoint = “processImage”)]
public static extern void processImage(Color32[] raw, int width, int height);
If you get a compiler error for not knowing what DllImport is, you should add this to the top of your C# code:
using System.Runtime.InteropServices;
You can find the whole Unity project along with all source codes mentioned from this link.
Here are a few final (and very important) notes about using OpenCV in Unity.
1. You need to copy your C++ library to Plugins folder in Unity project.
2. You need to deploy OpenCV libraries used by your project along with the library itself.
You might want to read about Plugins in Unity and also make sure to comment below if you face any issues with the source code or have any questions.




2 thoughts on “How to Use OpenCV in Unity (Example Project)

  1. Hi, thanks very much for the tutorial, it is very usefull.

    Could you tell me how to convert from Mat image to Color32 struct so I can send an image from OpenCV to Unity?

    1. The answer to your question is actually in the post. I’ll explain it a little bit more.

      We have the following:
      Mat frame(height, width, CV_8UC4, raw);

      This is how we created a Mat using Color32 struct.

      One way of passing any other Mat to Color32 is to make sure 1. Your Mat has the type 8UC4 and 2. Pass the data of Mat over to Unity again as described above.

      Note that you need to be careful about memory management here, so I would really suggest (if it’s possible) just send an empty Color32 (with the required width and height from Unity to OpenCV and then fill it in your OpenCV code.
      Since you have access to its pointer, the operation is not going to be too slow anyway.

Leave a Reply

Your email address will not be published. Required fields are marked *