How to Pass Images Between OpenCV and Unity

A question I got a lot after writing this post quite some time ago was “How to pass a processed image from OpenCV back to Unity?!” Obviously, that tutorial described just the way to pass images from Unity to C# (because of the nature of what it needed to do) and my answers in the comments section didn’t quite clarify things for most people, so I thought why not write another tutorial and address all those questions in one go. This article is the result, so I hope it helps you understand not just how images are passed from Unity to OpenCV and vice versa, but also why it needs to be done like this.



First things first, the environment I used for the screenshots and testing the codes I have shared here in this post:

  • Windows 10 Pro 64-bit
  • Unity 2019.1.4f1 (Installed using Unity Hub 2.0.0)
  • Visual Studio 2017 Professional Edition
  • OpenCV 4.1.0 for MSVC 2017 x64

A couple of notes before going on. You can adapt this tutorial to any other operating system and development environment if you understand what the differences are between libraries in macOS and Windows and so on, but if you still face issues making it work, you can post your questions in the comments section down below. Another thing you should note, is that you can use any other preferably recent version of OpenCV as well, such as OpenCV 3.4.6 and so on.

Well, to begin, let’s assume we’ll create a simple user interface to view the camera feed and see the results of the OpenCV processed image in real time. Here is how our UI will look like:

To clarify, we’ll be creating a 2D Unity project with a simple UI seen above. It includes an Image and a Button to exit the application when clicked.

Next, we’ll create a C# Script to bring our UI to life. Let’s assume this Script is called MainScript.cs and it is added to an Empty Component called Script in our UI and initially it’s filled with the following code to handle the Exit Button and other necessary assignments of components in UI to members in our C# class:

using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.UI;

public class MainScript : MonoBehaviour
{
    public Image CameraImage;
    public Button ExitButton;

    void Start()
    {
        ExitButton.onClick.AddListener(OnExitButtonClick);
    }

    void Update()
    {
    }

    private void OnExitButtonClick()
    {
        Application.Quit();
    }
}

For the Script to work, you must first assign the Image and Button components as seen in the following picture:

Just drag and drop objects from the Hierarchy panel on the left and into the relevant fields in the Inspector panel, while Script object is selected.

Now we need to use WeCamTexture class in Unity to access live frames read from the camera or webcam in our computer. For this reason we define it as a member in our MainScript class:

private WebCamTexture _webcam;

The other class we need is called Texture2D. We’ll use this class to display images modified by OpenCV. SO let’s define that as well as a member of our class:

private Texture2D _cameraTexture;

The next and probably the most important thing we need to define in our class is the following:

[DllImport("ComputerVision")]
private static extern void ProcessImage(ref Color32[] rawImage, int width, int height);

This corresponds to the library and function we’ll create in a C++ library that uses OpenCV to process an image read by Unity classes in our C# code. In case you’re interested about the details of the code above, you can read more in this page. The part we need to pay attention to is the first parameter passed to the ProcessImage function:

ref Color32[] rawImage

If you are familiar enough with both C++ and C# languages, you can probably guess what we’re trying to achieve and what will be the corresponding definition for this parameter in our C++ code. But to clarify, we use this to pass a reference to an array of pixels from C# to C++. In other words, we’re trying to pass pixels from C# to C++ but at the same time we want our C++ code to be able to modify the contents of the pixels and consequently pass back pixels from OpenCV and C++ to Unity and C#. In C++, this corresponds to the following:

Color32 **rawImage

Again, if you’re not new to C/C++, you already know that this is a pointer to a pointer of pixel data, or in other words a pointer to an array of pixel data. For more curious C/C++ pros, you’ll see why we must use pointers instead of proper arrays or vectors and references later on.



Back to where we were, we need to initialize the members we defined in our C# code, so let’s modify the start() method of MainScript class to make it look like this:

void Start()
{
    ExitButton.onClick.AddListener(OnExitButtonClick);
    _webcam = new WebCamTexture();
    _webcam.Play();
    _cameraTexture = new Texture2D(_webcam.width, _webcam.height);
    CameraImage.material.mainTexture = _cameraTexture;
}

And update() method needs to look like this:

void Update()
{
    if (_webcam.isPlaying)
    {
        var rawImage = _webcam.GetPixels32();
        ProcessImage(ref rawImage, _webcam.width, _webcam.height);
        _cameraTexture.SetPixels32(rawImage);
        _cameraTexture.Apply();
    }
}

It’s nothing complicated if you are at least armed with the basics. In the start() method, we initialize _webcam and _cameraTexture members of MainScript whereas in update() method we do the following:

  • Check if the camera is actually providing frames (isPlaying)
  • Read pixels of the current frame read from the camera (GetPixels32)
  • Pass the image/pixels from Unity to OpenCV and then from OpenCV back to Unity, since it’s defined as a reference (ProcessImage)
  • Display the results (SetPixels32 and Apply)

Now that we have everything we need in our C# code and Unity project, it’s time to create a library and actually write the code responsible for ProcessImage function above.

Start by creating Dynamic-Link Library (DLL) project in Visual Studio 2017. Make sure you name it ComputerVision, since that’s the name we used when we defined it using DllImport in MainScript above:

After the project is created, what I’d suggest is to remove all automatically created header and source files and add just a single *.cpp file called ComputerVision.cpp and modify it to look like this:

#include <opencv2/opencv.hpp>

struct Color32
{
	uchar red;
	uchar green;
	uchar blue;
	uchar alpha;
};

extern "C"
{

	__declspec(dllexport) void ProcessImage(Color32 **rawImage, int width, int height)
	{
		using namespace cv;

		// create an opencv object sharing the same data space
		Mat image(height, width, CV_8UC4, *rawImage);

		// start with flip (in both directions) if your image looks inverted
		flip(image, image, -1);

		// start processing the image
		// ************************************************

		Mat edges;
		Canny(image, edges, 50, 200);
		dilate(edges, edges, (5, 5));
		cvtColor(edges, edges, COLOR_GRAY2RGBA);
		normalize(edges, edges, 0, 1, NORM_MINMAX);
		multiply(image, edges, image);

		// end processing the image
		// ************************************************

		// flip again (just vertically) to get the right orientation
		flip(image, image, 0);
	}
}

Notice that we need to first define the Color32 structure as seen below, which corresponds to the type of pixel data Unity provides:

struct Color32
{
	uchar red;
	uchar green;
	uchar blue;
	uchar alpha;
};

In OpenCV this corresponds to CV_8UC4 type, or in other words 32-bit RGBA. The next important piece of code is to wrap all of our functions in extern “C”, as seen here:

extern "C"
{
...
}

The comments I’ve inserted inline should be enough, but again just to further clarify, the following part can be replaced with any other code that modifies the original image:

// start processing the image
// ************************************************

Mat edges;
Canny(image, edges, 50, 200);
dilate(edges, edges, (5, 5));
cvtColor(edges, edges, COLOR_GRAY2RGBA);
normalize(edges, edges, 0, 1, NORM_MINMAX);
multiply(image, edges, image);

// end processing the image
// ************************************************

A couple of other notes if you are facing issues building your library using Visual Studio 2017. Make sure you add OpenCV include paths and lib files to your project, and disable pre-compiled headers if you remove all initial files that were created by Visual Studio. Other than that, if you’re still facing any issues, just post it as a question in comments section below.



Now build your library and copy the created library file into the Plugins folder in Unity project. Make sure you also copy over any dependencies, such as OpenCV in this case:

Before running the project in Unity, make sure the libraries are configured correctly by choosing them and setting the required values in Inspector. In my case I used OpenCV 4.1.0 64-bit to build my library, that’s why I used the following configuration:

If you haven’t missed anything, you should be able to safely build and run your project now. Below is an screenshot of the project running on my PC showing me in my room with a Canny effect applied to the live images:



9 Replies to “How to Pass Images Between OpenCV and Unity”

  1. This is what I was trying to do in the .cpp:


    #include

    using namespace cv;
    extern "C"
    {
    std::vector FindCircles(uchar **rawImage, int width, int height, double dp, double minDist, double param1, double param2, int minRadius, int maxRadius)
    {
    Mat image(height, width, CV_8UC1, *rawImage);
    std::vector circles;
    HoughCircles(image, circles, 3, dp, minDist, param1, param2, minRadius, maxRadius);

    return circles;
    }
    }

    ^This fails to build the way I described in the previous comment, so I modified it to the one below, it builds, but does not seem to work:


    #include

    using namespace cv;
    extern "C"
    {
    std::vector FindCircles(uchar **rawImage, int width, int height, double dp, double minDist, double param1, double param2, int minRadius, int maxRadius)
    {
    Mat image(height, width, CV_8UC1);
    image.data = *rawImage;
    std::vector circles;
    HoughCircles(image, circles, 3, dp, minDist, param1, param2, minRadius, maxRadius);

    return circles;
    }
    }

    And this is my C# script in Unity to pass the image as byte[]:


    [DllImport("OpenCVOBject")]
    private static extern Vector3[] OpenCVFindCircles(byte[] image, int width, int height, double dp, double minDist, double param1, double param2, int minRadius, int maxRadius);

    public Vector3[] FindCircles(byte[] image, int width, int height, double dp, double minDist, double hiThreshold, double lowThreshold, int minRadius, int maxRadius)
    {
    System.Array.Reverse(image);
    Vector3[] circles = OpenCVFindCircles(image, width, height, dp, minDist, hiThreshold, lowThreshold, minRadius, maxRadius);
    return circles;
    }

    Thank you for your aid 🙂

    1. Please pay attention to how the data is passed from Unity to OpenCV:
      Mat image(height, width, CV_8UC4, *rawImage);

      You can’t just copy a 4 channel image (hence the Color32 structure) into a single channel and expect things to somehow work out.
      Use the exact code from the tutorial and then try to convert it from CV_8UC4 to CV_8UC1.
      I suggest you to try doing the exact same thing as in this tutorial first and see if you’re successful.
      If you are, then you can try and convert the 4 channel to single channel using cvtColor or any similar method, and then perform any image processing task you need.

      1. Sorry, I forgot to mention I was trying to pass a *single channel* 8 bit image.
        But you are right, I should try to copy your method first, and going jackass only afterwards 🙂

  2. Hi,

    I am trying to achieve this for Android library, I have set up the project based on your next article, my SDK is the OpenCV 4.1.0.

    Now, whenever I try to define a “Mat” by encoding the byte[] (CV_8UC1 or CV_8UC1) or I try to create that from a given file location, I get build error: identifier “__fp16” is undefinied.

    As far as I understand, ARM processors can not work with half precision floats, which Mat seems to be using.
    However, for image processing I need to define a Mat (again, as far as I understand). Any idea to resolve this?

    Thanks in advance,
    Richard

    1. Are you trying to initialize a CV_8UC1 using the image from Unity? (Not sure if I understood your issue correctly)
      Anyway, if that’s the case, or something similar to that, then just define a CV_8UC1 and then try to use cvtColor to convert the contents of the Unity image from RGBA to Grayscale and into your CV_8UC1.
      Let me know if this helps.

    2. You can also share the exact lines of code you’re writing and the line your compiler is complaining about. It would help identifying the issue.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.