Overview

Machine vision Python modules are programmed as a single class that exposes a specific interface expected by the JeVois Engine. See Programming machine vision modules with Python for an introduction.

As an initial (slightly over-simplified) idea, the overall workflow for JeVois vision modules is that they implement a processing function that will receive an image captured by the camera sensor, and a pre-allocated output image that will be sent to the host computer over USB. The task of the processing function is to fill in the output image with results that arise from processing the input image.

In this tutorial, we first show you how to program a few very simple modules, to bring in the necessary concepts. Towards the end of this tutorial, we will point you to further detailed reading.

JeVois-Pro: The tutorials below create modules that operate in "legacy mode" on JeVois-Pro, i.e., the module receives one image as input and outputs a new image that contains processing results. For examples of modules that use the new "Pro/GUI" mode of /jvpro, check out the source code of the many examples in jevoisbase: User guide to bundled vision modules and demos

Before you start, you should understand Concepts used throughout this documentation.

Pixel formats and video mappings

As a reminder, the JeVois smart camera can capture images in the following camera pixel formats: YUYV, BAYER, or RGB565. These formats are the ones that are supported by the camera sensor chip.

JeVois can send a wider range of pixel formats to a host connected over USB: YUYV, GREY, MJPG, BAYER, RGB565, and BGR24.

For explanations about these formats, please see User guide to video modes and mappings for details.

Camera to USB video mappings

A module is invoked when a particular image resolution and pixel format is selected by a host computer over USB on JeVois-A33 or by th euser on JeVois-Pro. A list of video mappings associate a given output resolution and pixel type to the corresponding camera resolution and pixel type that should be used and to the machine vision module that should be invoked. Again see User guide to video modes and mappings for details.

Because an output video format is selected by the host computer on JeVois-A33 or the user on JeVois-Pro, it is not negotiable. If a video mapping has been specified in videomappings.cfg that invokes a particular machine vision module, that module just must perform (or throw an exception), and process the images that it receives from the camera to generate the desired output images.

JeVois-Pro: for JeVois-Pro, replace import libjevois as jevois in the examples below by the following:

import pyjevois
if pyjevois.pro: import libjevoispro as jevois
else: import libjevois as jevois

Getting started: an edge detection module

Here we program a simple edge detection module in Python using OpenCV's Canny edge detector. We will grab a color video frame, convert it to grayscale, compute the edge map (also grayscale), and send that over USB to the host computer.

Here is the complete, working code. We will look at it step-by-step below:

import libjevois as jevois
import cv2
import numpy as np
 
## Simple example of image processing using OpenCV in Python on JeVois
#
# This module by default simply converts the input image to a grayscale OpenCV image, and then applies the Canny
# edge detection algorithm. Try to edit it to do something else (note that the videomapping associated with this
# module has grayscale image outputs, so that is what you should output).
#
# @author Laurent Itti
# 
# @displayname Python Tutorial 1
# @videomapping GRAY 640 480 20.0 YUYV 640 480 20.0 JeVois PythonOpenCV
# @email itti\@usc.edu
# @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
# @copyright Copyright (C) 2017 by Laurent Itti, iLab and the University of Southern California
# @mainurl http://jevois.org
# @supporturl http://jevois.org/doc
# @otherurl http://iLab.usc.edu
# @license GPL v3
# @distribution Unrestricted
# @restrictions None
# @ingroup modules
class PythonTutorial1:
    # ###################################################################################################
    ## Process function with USB output
    def process(self, inframe, outframe):
        # Get the next camera image (may block until it is captured) and convert it to OpenCV GRAY:
        inimggray = inframe.getCvGRAY()
        
        # Detect edges using the Canny algorithm from OpenCV:
        edges = cv2.Canny(inimggray, 100, 200, apertureSize = 3)
        
        # Convert our GRAY output image to video output format and send to host over USB:
        outframe.sendCvGRAY(edges)

Explanations:

line 1: The JeVois core exposes a library called libjevois to Python. This allows Python modules to use some of the core functionality programmed in JeVois. See Programming machine vision modules with Python for details. Here, we import this library into our module under the alias name jevois.
line 2: Likewise, OpenCV (which is written in C++) exposes bindings that allow Python programs to call the image processing functions of OpenCV. Here we import that library as well.
line 3: Another library, numpy, is useful for manipulating arrays, including images. Indeed, OpenCV uses numpy ND-arrays to represent images.
Lines 5-24: We use doxygen and custom tags for JeVois to document the code so that a simple documentation page will be created automatically from the comments in the code. The page for this module is here and it is generated automatically from the comments in the code. See Programmer SDK and writing new modules for details.
line 25: We declare a class. Its name must exactly match the file name (without .py extension) and the directory name under which this module is stored.
Line 28: The JeVois Engine expects a function called process() which it will call on every video frame. The Engine will give us an InputFrame here named inframe (a wrapper over C++ class jevois::InputFrame, which is a proxy to a camera sensor frame), and an OutputFrame here named outframe (wrapper over C++ class jevois::OutputFrame, which is a proxy to a video buffer that can be sent over USB). Our task in the process() function is to create an output image that we will send over the USB link using the outframe wrapper, from the input image that we are getting via the inframe wrapper.
Line 30: In this tutorial, we want a grayscale input image. We extract it from the inframe. This call may block until the camera sensor has finished capturing the next frame. See jevois::InputFrame for more details.
Line 33: We detect edges in our gray image using OpenCV, see http://docs.opencv.org/trunk/da/d22/tutorial_py_canny.html for very nice explanations of how it works.
Line 36: We send our output image to the host computer over USB. Note how sendCvGRAY() is used here because our edge map is grayscale. sendCvGRAY() will convert from grayscale to whatever the host is expecting (which depends on the videomapping that invoked our module).

Try it yourself: a sandbox module

Here we program a simple color image filtering module using OpenCV's Laplacian detector. We will grab a color video frame, apply the filter to it, and send the color result USB to the host computer.

Here is the complete, working code. We will look at it step-by-step below:

import libjevois as jevois
import cv2
import numpy as np
 
## Simple example of image processing using OpenCV in Python on JeVois
#
# This module is here for you to experiment with Python OpenCV on JeVois.
#
# By default, we get the next video frame from the camera as an OpenCV BGR (color) image named 'inimg'.
# We then apply some image processing to it to create an output BGR image named 'outimg'.
# We finally add some text drawings to outimg and send it to host over USB.
#
# @author Laurent Itti
# 
# @displayname Python Tutorial 2
# @videomapping YUYV 352 288 30.0 YUYV 352 288 30.0 JeVois PythonSandbox
# @email itti\@usc.edu
# @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
# @copyright Copyright (C) 2017 by Laurent Itti, iLab and the University of Southern California
# @mainurl http://jevois.org
# @supporturl http://jevois.org/doc
# @otherurl http://iLab.usc.edu
# @license GPL v3
# @distribution Unrestricted
# @restrictions None
# @ingroup modules
class PythonTutorial2:
    # ###################################################################################################
    ## Constructor
    def __init__(self):
        # Instantiate a JeVois Timer to measure our processing framerate:
        self.timer = jevois.Timer("sandbox", 100, jevois.LOG_INFO)
        
    # ###################################################################################################
    ## Process function with USB output
    def process(self, inframe, outframe):
        # Get the next camera image (may block until it is captured) and here convert it to OpenCV BGR by default. If
        # you need a grayscale image instead, just use getCvGRAY() instead of getCvBGR(). Also supported are getCvRGB()
        # and getCvRGBA():
        inimg = inframe.getCvBGR()
        
        # Start measuring image processing time (NOTE: does not account for input conversion time):
        self.timer.start()
 
        # Detect edges using the Laplacian algorithm from OpenCV:
        #
        # Replace the line below by your own code! See for example
        # - http://docs.opencv.org/trunk/d4/d13/tutorial_py_filtering.html
        # - http://docs.opencv.org/trunk/d9/d61/tutorial_py_morphological_ops.html
        # - http://docs.opencv.org/trunk/d5/d0f/tutorial_py_gradients.html
        # - http://docs.opencv.org/trunk/d7/d4d/tutorial_py_thresholding.html
        #
        # and so on. When they do "img = cv2.imread('name.jpg', 0)" in these tutorials, the last 0 means they want a
        # gray image, so you should use getCvGRAY() above in these cases. When they do not specify a final 0 in imread()
        # then usually they assume color and you should use getCvBGR() here.
        #
        # The simplest you could try is:
        #    outimg = inimg
        # which will make a simple copy of the input image to output.
        outimg = cv2.Laplacian(inimg, -1, ksize=5, scale=0.25, delta=127)
                
        # Write a title:
        cv2.putText(outimg, "JeVois Python Sandbox", (3, 20), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255,255,255),
                    1, cv2.LINE_AA)
        
        # Write frames/s info from our timer into the edge map (NOTE: does not account for output conversion time):
        fps = self.timer.stop()
        height, width, channels = outimg.shape # if outimg is grayscale, change to: height, width = outimg.shape
        cv2.putText(outimg, fps, (3, height - 6), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255,255,255), 1, cv2.LINE_AA)
 
        # Convert our BGR output image to video output format and send to host over USB. If your output image is not
        # BGR, you can use sendCvGRAY(), sendCvRGB(), or sendCvRGBA() as appropriate:
        outframe.sendCvBGR(outimg)

Explanations:

lines 1-27: Same as in the previous tutorial above.
Line 30: Constructor. In this module, we will use a persistent member object of our class, a Timer which is provided by JeVois (see jevois::Timer). It will measure how much time it takes to process each frame, and it will report the average frame rate every 100 frames. Because the timer will remain across many frames, we declare it as a member variable of our class and we initialize it in the constructor of our module.
Line 32: We add the timer as a new member variable of our class and we initialize it.
Line 36: The JeVois Engine expects a function called process() which it will call on every video frame. Same as in the previous tutorial above.
Line 40: In this tutorial, we want a color input image. We extract it from the inframe, as 'BGR' color because, for historical reasons, this is the preferred color format of OpenCV (though RGB is nowadays more popular in other machine vision libraries). This call may block until the camera sensor has finished capturing the next frame. See jevois::InputFrame for more details.
Line 43: Start measuring time for the current run through process()
Line 60: We compute the Laplacian in our color image using OpenCV, see http://docs.opencv.org/trunk/d5/db5/tutorial_laplace_operator.html for very nice explanations of how it works (note that they run it on a grayscale image, while we here run it on our color image). As written in the commends, this module is an invitation for you to experiment and try some other image filters (pick those that take a color image in, and output a color image of the same size). In fact, this tutorial also exists as module Python Sandbox in jevoisbase, and is pre-installed on the latest microSD images. See the JeVois interactive tutorials for examples of how to play with it.
Lines 63-64: Write a title near the top of our output image. See http://docs.opencv.org/3.2.0/d6/d6e/group__imgproc__draw.html for details about putText()
Line 67: Tell our timer to stop measure time for this frame. As we stop it, it also returns a string that contains average frames/s, CPU usage, CPU temperature, and CPU frequency and which gets updated every 100 frames. Here we store that info into a string fps which we will display on top of our results.
Line 68: Get the image width, height and number of channels. Here we need this so we can compute the Y coordinate of the bottom of the frame, which we will use to display text near the bottom.
line 69: Write the fps information near the bottom of the frame.
Line 73: We send our output image to the host computer over USB. Note how sendCvBGR() is used here because our output image has BGR color. sendCvBGR() will convert from BGR to whatever the host is expecting (which depends on the videomapping that invoked our module).

More fine-grained control using JeVois raw images

In the above two tutorials, we used the easiest route of converting the input image to OpenCV, then working with OpenCV images, and converting the output image from OpenCV to a raw image buffer that will be sent over USB. While this is great to get started, we do incur some cost (i.e., time) converting the images. The core JeVois library also exposes lower-level access to the camera sensor raw image data and to the raw output buffer that will be sent over USB.

Here we write a module that does not use OpenCV at all. It uses JeVois drawing functions that work directly with raw YUYV images. This avoids having to convert from YUYV (camera sensor raw format) to BGR and then back to YUYV (format sent natively over the USB link).

Here is the complete, working code. We will look at it step-by-step below:

import libjevois as jevois
 
## Simple test of programming JeVois modules in Python
#
# This module by default simply draws a cricle and a text message onto the grabbed video frames.
#
# Feel free to edit it and try something else. Note that this module does not import OpenCV, see the PythonOpenCV for a
# minimal JeVois module written in Python that uses OpenCV.
#
# @author Laurent Itti
#
# @displayname Python Tutorial 3
# @videomapping YUYV 640 480 15.0 YUYV 640 480 15.0 JeVois PythonTutorial3
# @email itti\@usc.edu
# @address University of Southern California, HNB-07A, 3641 Watt Way, Los Angeles, CA 90089-2520, USA
# @copyright Copyright (C) 2017 by Laurent Itti, iLab and the University of Southern California
# @mainurl http://jevois.org
# @supporturl http://jevois.org/doc
# @otherurl http://iLab.usc.edu
# @license GPL v3
# @distribution Unrestricted
# @restrictions None
# @ingroup modules
class PythonTTutorial3:
    # ###################################################################################################
    ## Constructor
    def __init__(self):
        jevois.LINFO("PythonTest Constructor")
        jevois.LINFO(dir(jevois))
        self.frame = 0 # a simple frame counter used to demonstrate sendSerial()
 
    # ###################################################################################################
    ## Process function with no USB output
    def processNoUSB(self, inframe):
        jevois.LFATAL("process no usb not implemented")
 
    # ###################################################################################################
    ## Process function with USB output
    def process(self, inframe, outframe):
        jevois.LINFO("process with usb")
 
        # Get the next camera image (may block until it is captured):
        inimg = inframe.get()
        jevois.LINFO("Input image is {} {}x{}".format(jevois.fccstr(inimg.fmt), inimg.width, inimg.height))
 
        # Get the next available USB output image:
        outimg = outframe.get()
        jevois.LINFO("Output image is {} {}x{}".format(jevois.fccstr(outimg.fmt), outimg.width, outimg.height))
 
        # Example of getting pixel data from the input and copying to the output:
        jevois.paste(inimg, outimg, 0, 0)
 
        # We are done with the input image:
        inframe.done()
 
        # Example of in-place processing:
        jevois.hFlipYUYV(outimg)
 
        # Example of simple drawings:
        jevois.drawCircle(outimg, int(outimg.width/2), int(outimg.height/2), int(outimg.height/2.2), 2, 0x80ff)
        jevois.writeText(outimg, "Hi from Python!", 20, 20, 0x80ff, jevois.Font.Font10x20)
        
        # We are done with the output, ready to send it to host over USB:
        outframe.send()
 
        # Send a string over serial (e.g., to an Arduino). Remember to tell the JeVois Engine to display those messages,
        # as they are turned off by default. For example: 'setpar serout All' in the JeVois console:
        jevois.sendSerial("DONE frame {}".format(self.frame));
        self.frame += 1
 
    # ###################################################################################################
    ## Parse a serial command forwarded to us by the JeVois Engine, return a string
    def parseSerial(self, str):
        jevois.LINFO("parseserial received command [{}]".format(str))
        if str == "hello":
            return self.hello()
        return "ERR Unsupported command"
    
    # ###################################################################################################
    ## Return a string that describes the custom commands we support, for the JeVois help message
    def supportedCommands(self):
        # use \n seperator if your module supports several commands
        return "hello - print hello using python"
 
    # ###################################################################################################
    ## Internal method that gets invoked as a custom command
    def hello(self):
        return "Hello from python!"
        

Explanations:

Lines 1-24: Same as above.
Lines 27-30: Constructor. We print a couple of messages using the JeVois LINFO() logging function. These messages will be sent either to the hardware serial port (if you type setpar serlog Hard in the JeVois console), serial-over-USB port (if setpar serlog USB) or both (if setpar serlog All). See Command-line interface user guide for details about serlog.
Lines 34-35: JeVois also supports modules which process video frames from the camera sensor but do not produce any video output. These modules would typically only product text outputs over serial ports. See Concepts used throughout this documentation. In C++, we have two overloads of the process() function, but this is not possible in Python, so both functions take different names: process() and processNoUSB(). Here we do not implement the processNoUSB() function that only takes a camera image in, but we provide a skeleton for it. Examples of how this function is implemented ate provided in several of the jevoisbase C++ modules. Look for modules that claim to support mappings with NO USB output under User guide to bundled vision modules and demos for details (for example, DemoArUco).
Line 39: Here is our process() function with video output to USB.
Line 43: We get the raw video frame from the camera sensor. Given our recommended videomapping (see comment on line 13), we expect that this will be YUYV. See User guide to video modes and mappings for a refresher on pixel formats and video mappings. The object returned by get() here is a Python-wrapped jevois::RawImage object. It contains a direct pointer to the memory buffer allocated in the Linux Kernel and into which the camera sensor stuffs pixel information using direct memory access (DMA). Hence, jevois::RawImage is a lightweight object, which just holds a shared pointer to the pixel data as supposed to owning a copy to that data. So, unlike the tutorials above, so far we have made zero copy or conversion of the raw pixel array that is coming from the camera sensor. This is very fast.
Line 47: We also get the output video buffer as raw YUYV jevois::RawImage. Here again, RawImage directly points to a raw pixel buffer allocated by the Linux kernel, which will be used directly to stream data over the USB link using DMA. Very efficient zero-copy access.
Line 51: In this tutorial module, we start by copying the pixels from the camera sensor into our output raw buffer, so that users can see what the camera sees. In more complex scenarios, the input buffer could be copied to a small window within the output buffer (e.g., as in DemoSaliency), or may not be copied at all, just processed in some way to yield a computed output image that will be sent out (e.g., computing an edge map from the input image). Thus, we here just paste inimg into outimg starting at location (0,0), which is the top-left corner, in outimg.
Line 54: From this point on, we will work with the output image only, so we can release the input image, so that its buffer will be recycled to the camera sensor kernel driver, to be used for subsequent frame capture. It is generally a good idea to mark the input frame as done as early as possible, so that its buffer becomes available to the camera sensor as early as possible.
Line 57: The JeVois core provides a number of operations that can work directly on YUYV images. Here, for example, we flip the output image horizontally in place. See Raw zero-copy / memory-mapped video images and support functions for more operations.
Lines 60-61: The JeVois core also provides simple drawing functions that can work directly with YUYV images. Again, see Raw zero-copy / memory-mapped video images and support functions for details. Here, we just draw a circle and write a text message.
Line 64: Once we are done and the output image is finalized, we can send it to the host computer over USB. This call to send() on the output frame is optional, it would be called anyway when process() ends.
Line 68: Here is an example of sending a text string to the serout serial port, for example intended for an Arduino connected to JeVois. The Engine parameter serout determines where that string will go (hardware serial port, serial-over-USB, none, or both). Note that by default it will go to no port. You need to issue a setpar serout Hard in the JeVois console (or in a config file) to send output to the hardware serial port. See Command-line interface user guide for more information about serout.
Line 69: We increment our internal frame counter and we are done for this frame. The JeVois Engine will call process() again on the next frame.
Lines 73-77: Modules can optionally support custom commands, which add to the commands already interpreted by the JeVois engine. This is achieved by implementing a parseSerial() function in the module. The JeVois Engine will forward to that function any commands received over the command-line interface that it does not recognize. Here, we support a new command hello, which will call the hello() member function of our module (Lines 87-88). The parseSerial() function should return a string with form either "OK", or "ERR some error message", or it could throw an exception (which the Engine will catch and report with a message that starts with "ERR "). The Engine will forward any returned string here to the appropriate serlog serial port (see Command-line interface user guide for details on serlog).
Lines 81-83: If we support custom commands, we should let user knows by describing them in the help message. This is achieved by implementing a member function supportedCommands(self) in the module. The Engine will call it when users type help in the command-line interface. This function should return a string with the command name and a brief description. If the module supports multiple commands, return only one string, separating the different commands with a newline \n character.
Lines 87-88: This is the module member function that gets invoked when users type hello on the command-line interface.

For more Module tutorials and examples

For more, see:

the documentation of the JeVoisBase collection of modules, look for those written in Python (the language field towards top of the documentation page of each module).
JeVois tutorials