JeVois + Arduino: blink for X

JeVois v1.9.0

In this tutorial, we program an Arduino connected to JeVois to make it turn on its LED when a particular object is in sight. This is the JeVois version of the Arduino blink LED tutorial.

Getting started

• Setup Jevois and JeVois Inventor using the other tutorials in this series
• Download and install the Arduino software from http://arduino.cc

Approach

• We will use any of the JeVois modules that can recognize 1000 different types of objects using deep neural networks trained on the ImageNet dataset. This includes the following JeVois modules: TensorFlowEasy, DarknetSingle, TensorFlowSingle
• These modules send standardized serial messages that describe what they have detected.
• These modules only detect objects at the center of the field of view. Hence the messages are only about what was detected and contain no information about where it was detected. Other modules like TensorFlowSaliency or DarknetYOLO send more complex messages that describe both what (category) and where (bounding box) for each detected object. Those are covered in another tutorial, JeVois + Arduino: Decoding object detection boxes
• We will write two Arduino programs to decode these messages:
  • First a simple one that matches the whole message sent by JeVois to a desired message,
  • Then a more elaborate one that parses the message sent by JeVois into tokens, using a state machine.

Hardware setup

• We use an Arduino compatible Pro Micro with Atmega 32u4 since this is available at jevoisinc.com
• First let's review Serial port user guide

• Then, let's find pin-out information for our Arduino on the web, for example at https://cdn.sparkfun.com/assets/9/c/3/c/4/523a1765757b7f5c6e8b4567.png and reproduced here:

  
• Finally, we solder the micro-serial cable that came with our JeVois camera to the Arduino, as follows:

Wire color	Arduino Pro Micro 32u4 pin
White	RX
Yellow	TX
Black	GND (3 GND pins are available)
Red	VCC (this Arduino has no IOREF pin)

Test the Arduino and LED

On the Pro Micro 32u4, there is no LED on the standard pin 13 (this is the LED of other models). However, there are 2 LEDs we can use. One possible LED pin is number 17, labeled 'RX LED'.

• In the Arduino software, go to File, Examples, 01.Basic, and open Blink

• Replace all instances of LED_BUILTIN (on recent Arduino software) or of 13 (on older software) in there by 17

  
• Connect your Arduino to your computer using a micro-USB cable.
• Under Tools, Board, select Arduino Leonardo since the pro micro model is not an official Arduino model (but a third-party design). You may also have to select the Port.
• Flash the code (Arrow-shaped button in the top left corner of the Arduino editor window).
• You should see the RX LED (next to pin 7) blink slowly (1 second on / 1 second off).
• It is not obvious now, but note that this LED is inverted: on when the pin is LOW and off when it is HIGH.

Decoding serial messages sent by JeVois

Two pages of the main JeVois doc are critical here:

• Standardized serial messages formatting for info on standardized serial messages, and
• Helper functions to convert coordinates from camera resolution to standardized for info on standardized coordinates.

From the doc of one of the object recognition modules, for example, TensorFlowEasy, we read that:

    When detections are found with confidence scores above 'thresh', a message containing up to 'top' category:score
    pairs will be sent per video frame. Exact message format depends on the current serstyle setting and is described
    in Standardized serial messages formatting. For example, when 'serstyle' is 'Detail', this module sends:

    DO category:score category:score ... category:score

    where 'category' is a category name (from 'namefile') and 'score' is the confidence score from 0.0 to 100.0 that
    this category was recognized. The pairs are in order of decreasing score.

We can further read about the messages in Standardized serial messages formatting

So now we just need to modify the Arduino program to receive and decode these strings, and to turn on the LED when a particular object category is detected.

Writing the code - whole message match

Our approach for the Arduino code that matches the whole message from JeVois is going to be:

• Read a whole line of text from JeVois.
• Match it against a desired message in one step.
• Note that, according to the Standardized serial messages formatting, when we are in Terse serial message mode, only category name will be sent (no ':' and score). Also, only one category name, the top-scoring one, is reported in Terse mode. This is the default when JeVois starts up and we will use it here.

• To get started, let's fire up JeVois Inventor, select machine vision module TensorFlowEasy, and turn on serial output messages to both USB and 4-pin in the Console tab. We see object category names as we point JeVois to various things:

  
• Let's now write our Arduino code to match those TO category messages:

// JeVois + Arduino blink for X - easy version
 
// Pin for LED, blinks as we receive serial commands:
#define LEDPIN 17
 
// Serial port to use: on chips with USB (e.g., 32u4), that usually is Serial1.
// On chips without USB, use Serial:
#define SERIAL Serial1
 
// Buffer for received serial port bytes:
#define INLEN 256
char instr[INLEN + 1];
 
// Our desired object: should be one of the 1000 ImageNet category names
#define CATEGORY "computer_keyboard"
 
void setup()
{
  SERIAL.begin(115200);
  SERIAL.setTimeout(500);
  
  pinMode(LEDPIN, OUTPUT);
  digitalWrite(LEDPIN, HIGH);
}
 
void loop()
{
  // Read a line of data from JeVois:
  byte len = SERIAL.readBytesUntil('\n', instr, INLEN);
 
  // Make sure the string is null-terminated:
  instr[len--] = '\0';
  
  // Cleanup any trailing whitespace:
  while (len >= 0 && instr[len] == ' ' || instr[len] == '\r' || instr[len] == '\n') instr[len--] = '\0';
  
  // If empty (including timeout), stop here:
  if (len < 0)
  {
    digitalWrite(LEDPIN, HIGH); // turn LED off (it has inverted logic)
    return;
  }
  
  // If the message is a match for our desired category, turn led on, otherwise off:
  if (strcmp(instr, "TO " CATEGORY) == 0)
    digitalWrite(LEDPIN, LOW); // turn LED on (it has inverted logic)
  else
    digitalWrite(LEDPIN, HIGH); // turn LED off
}

A few comments on the code:

• line 4: We use pin 17 as it is connected to a LED on our Arduino board
• line 8: We use Serial1 as it is the port connected to pins 0 and 1 on this board
• line 15: Put any ImageNet category name here, with spaces replaced by underscores
• lines 19-20: We setup the serial port. We use a timeout of 500ms. Note that JeVois by default does not send anything when it does not detect anything. This timeout will allow us to stop waiting for serial data after half a second, so that we can turn off the LED if it was on. Another approach would be to use frame markers as described in Standardized serial messages formatting
• lines 22-23: We set our LED pin as output. As noted above, it is inverted, so setting it HIGH actually turns it off.
• lines 28-29: We read a whole line of text from serial (could be empty if we time out).
• line 32: Make sure the line is null-terminated so it is a proper C string.
• line 35: Remove any trailing white space, including CR (\r) and LF (\n) characters so they do not interfere with our later matching.
• lines 38-42: If we end up with empty data, just turn off the LED and stop here.
• line 45: Compare the received message to our desired message. Remember that strcmp() returns 0 when the two strings are equal.
• lines 46-48: If it was a match, turn LED on, otherwise turn it off.

Compile and upload the code to your Arduino and here you go!

Above: Arduino LED near pin 7 is on when looking at a computer keyboard

Above: Arduino LED near pin 7 is off when looking at anything else

Writing the code - parsing the message into tokens

While the whole message match is appropriate for our simple example, it is not scalable to more complex scenarios where messages may contain multiple values that should be decoded and interpreted individually. For example, the more complex messages when serstyle is Detail take the form

DO category:score category:score ... category:score

and the various categories and scores will vary from message to message. These messages reflect the probabilistic nature of the object recognition deep network, which may output multiple guesses about what it is seeing, each with a confidence score. For example,

DO dog:53.6 cat:42.3

reflects that the deep network is hesitant and thinks it might be looking at a dog (with confidence 53.6), or possibly a cat (with confidence 42.3).

Note that TensorFlowEasy already applies a threshold to confidence scores (see parameter thresh of the module), so that only categories that score above that threshold are reported.

Let's see how we could parse these messages, so that our Arduino now blinks if any of the categories in the list is a match, i.e., the LED should still turn on if computer_keyboard was the second or third guess.

Our approach for the Arduino code that parses the message from JeVois is going to be:

• Set parameter serstyle to Detail so we get the detailed messages as above and as explained in Standardized serial messages formatting, and turn on serial output messages to both USB and 4-pin in the Console tab. We see object category names as we point JeVois to various things:

  
• Note that by default, the floating-point precision of the standardized messages is zero digit after the decimal point, i.e., we get integer scores. If you change that using the parameter serprec described in Standardized serial messages formatting, you can get more precise floating-point values (e.g., try setpar serprec 3 in the Console of JeVois Inventor). For the code below, we will assume floating point values which could be integers as well.
• Our code hence will first read a whole line of text from JeVois.
• Then, decode it into tokens, where successive tokens are separated by white space.
• Parse the tokens using a state machine. In this case, the state machine will be very simple: an integer variable will keep track of how far we are with parsing the tokens. We call it the state. The machine is our algorithm that does something in each state, then possibly transitions to a different state. Here:
  • We start with state=0 when a new line of text has been received and we start parsing it. We then look for the first token and check that it is DO (which is the symbol for object in Detail mode). If so, we move to the next step, by changing to state=1.
  • When state is 1 we want to decode the first category:score chunk. If it is a match with the desired category, we switch to state 2.
  • And that's pretty much it. We now have a state machine message parser!

// JeVois + Arduino blink for X
 
// Pin for LED, will turn on as we detect the desired object:
#define LEDPIN 17
 
// Serial port to use: on chips with USB (e.g., 32u4), that usually is Serial1.
// On chips without USB, use Serial:
#define SERIAL Serial1
 
// Buffer for received serial port bytes:
#define INLEN 256
char instr[INLEN + 1];
 
// Our desired object: should be one of the 1000 ImageNet category names
#define CATEGORY "computer_keyboard"
 
void setup()
{
  SERIAL.begin(115200);
  SERIAL.setTimeout(500);
  
  pinMode(LEDPIN, OUTPUT);
  digitalWrite(LEDPIN, HIGH);
}
 
void loop()
{
  byte len = SERIAL.readBytesUntil('\n', instr, INLEN);
  instr[len] = 0;
 
  char * tok = strtok(instr, " \r\n");
  int state = 0, i; float score;
  
  while (tok)
  {
    // State machine:
    // 0: start parsing; if we get DO, move to state 1, otherwise state 1000
    // 1: got DO, decode category name; if it is the one we want, move to state 2, otherwise stay in state 1
    // 2: got DO followed by the category we are interested in - it's a hit!
    // 1000: did not get DO, we stay in this state until we run out of tokens
    switch (state)
    {
      // First token should be: DO
      case 0:
        if (strcmp(tok, "DO") == 0) state = 1; else state = 1000;
        // We are done with this token. Break from the switch() statement
        break;
 
      // Second token should be: category:score
      case 1:
        // Find the ':' between category and score:
        i = strlen(tok) - 1;
        while (i >= 0 && tok[i] != ':') --i;
        
        // If i is >= 0, we found a ':'; terminate the tok string at that ':':
        if (i >= 0)
        {
          tok[i] = '\0';
 
          // Note: we don't care about score here, but it could be obtained as:
          score = atof(&tok[i+1]);
         }
        
        // Is the category name what we want?
        if (strcmp(tok, CATEGORY) == 0) state = 2;
        
        // We are done with this token. Break from the switch() statement
        break;
      
      // In any state other than 0 or 1: do nothing
      default:
        break;
    }
    
    // Move to the next token:
    tok = strtok(0, " \r\n");
  }
 
  // If any of the category names in the message was the one we want, then we are in state 2 now.
  // Otherwise we are in some other state (most likely 1).
  if (state == 2)
    digitalWrite(LEDPIN, LOW); // turn LED on (it has inverted logic)
  else
    digitalWrite(LEDPIN, HIGH); // turn LED off
}

A few comments on the code:

• lines 1-29: Same setup and preliminaries as in our previous code.
• line 31: We start our tokenizer at the start of the line of text received from JeVois. strtok() is a standard C function, see Arduino docs or other standard C/C++ docs like http://www.cplusplus.com/reference/cstring/strtok/ At this stage, tok points to the first chunk of characters until the first white space.
• line 32: We start in state 0, which we decide means 'start of line'.
• line 34: We will loop until we run out of space-separated tokens.
• lines 44-47: If we are in state 0, we expect that the first token should be DO; if so, advance to state 1; otherwise switch to state 1000 which we decide means some other kind of message was received which we will not handle. We are done with this first token and break out of the switch() statement. We will then advance to the next token on line 76. Remember that strcmp() returns 0 when both strings are equal.
• lines 50-65: If we are in state 1, we have successfully parsed the first token as DO, and we are looking at the second token now, which should be of the form category:score. We look for the ':' separator starting from the end of the token and moving backwards to the start. If we found it, we replace it by a \0 string termination character, to isolate the category name. If that is equal to the category we want, advance to state 2. Otherwise, stay in state 1 so that we will try again to decode the next token in the list, after line 76 advances us to the next token.
• lines 81-84: We have parsed all the tokens. If we are in state 2, we have detected our desired object, turn on the led (by setting it LOW on this inverted led). Otherwise, turn it off (by setting HIGH).

This code should behave like the previous one, but remember that you need serstyle to be Detail for the messages to be decoded by the Arduino now. The LED should still turn on even if the desired category was second guess, third guess, etc.

Going further

Check out these other tutorials. They use similar state machine decoding:

• JeVois + Arduino: Decoding object detection boxes
• Tutorial on how to write Arduino code that interacts with JeVois
• Making a motorized pan-tilt head for JeVois and tracking objects
• Build a simple visually-guided toy robot car for under $100 with JeVois