Lab: Vision in Camera Space

From Tekkotsu Wiki

Jump to: navigation, search


Learning Objective: This lab will teach you how to use Tekkotsu's vision system to recognize simple colored shapes such as blobs, lines, and ellipses.


Robots don't see the world the way we do. Achieving truly human-like vision remains a distant goal, but a variety of computer vision techniques have been developed that work well in special cases. One example is the UPC bar codes now commonly used on groceries and retail goods; these can be read by checkout scanners with very high accuracy. AprilTags, which are discussed elsewhere in this wiki, work on a similar principle. Another technique is to use unique colors to mark objects of interest so the computer can easily detect them against a cluttered background. Following this approach, this lab will show you how to get your robot to recognize colored blobs, lines, and ellipses.

Materials Required

To complete this lab you will need some colored shapes for the robot to look at. The recommended materials are some color masking tape (available at your local art supply store) and colored construction paper. But you can also use colored bits of plastic or other household objects.

If you don't have a robot available, you can run the lab using a workstation or laptop if Tekkotsu is installed. Simply plug a webcam into the workstation (or use the one built in to the laptop) and run Tekkotsu like this:

./tekkotsu-CALLIOPE -c webcam.plist

This tells Tekkotsu that it should use the webcam for visual input and not worry about any of the other robot components such as servos or bump switches.

If you're using a workstation and you don't have a webcam available, you can run Tekkotsu in simulator mode using a pre-stored image, such as the sample-image.png file that is included in your project/logged directory. Simply run Tekkotsu this way:

./tekkotsu-CALLIOPE -c sim.plist

Another option is to run Tekkotsu using Mirage to supply a virtual world for the camera to look at. See the articles in this wiki on Mirage to learn how to do that.

Color Image Segmentation

The figures below show an image taken from a robot's camera, and the corresponding color segmented image. Color segmentation begins by assigning a color class to each pixel. The color classes are defined in a color threshold file. By default, Tekkotsu uses a threshold file called that defines four colors: red, green, blue, and black. Tekkotsu provides tools for making your own threshold file and defining any color classes you like, but for now we will stick with the default file.


  1. Run Tekkotsu on your robot, and start up a ControllerGUI.
  2. Click on the "Raw" button in the ControllerGUI to display the raw camera image.
  3. Click on the "Seg" button in the ControllerGUI to display the color segmented image. Position the Raw and Seg windows so they are next to each other.
  4. Lay out some colored objects so the camera can see them. Compare the raw view with the color segmented view. Are there any parts of the objects that don't show up in the segmented view? Why not?
  5. Darken the room by shutting off some of the lights. Does the color segmented image change?
  6. Turn the room lights back on and make the scene even brighter by shining a floodlight or desk lamp on the objects you're examining. How does their appearance change in the raw view, and in the segmented view?

Color image segmentation can get tricked by shiny objects because of specular reflection. When the ambient light bounces directly off the surface of a shiny object, that part of the object looks white. What you're seeing is the color of the reflected light rather than the color of the underlying object. To humans, a shiny red apple looks like a red blob with a "gleam" to it because we subconsciously fill in the red behind the gleam. But what the color segmented image shows, and what a naive robot sees, is a red blob with hole in it.

Objects such as colored masking tape or colored construction paper that don't look particularly shiny to us may still exhibit significant specular reflection. The threshold file has been designed to minimize the effects of specular reflection, but these effects cannot be eliminated entirely. If you have problems getting your robot to see properly, use the Seg viewer to check the color segmented image, and if necessary, alter the lighting to minimize gleam.

Using the MapBuilder

The MapBuilder is the component of the Tekkotsu "crew" responsible for vision. You tell the MapBuilder what to look for by filling in the fields of a MapBuilderRequest object and submitting it to the MapBuilder. The easiest way to do this is to use a MapBuilderNode in your state machine.

The example code below tells the MapBuilder to look for blue lines, red blobs, and green ellipses.

#include "Behaviors/StateMachine.h"

$nodeclass MapBuilderTest1 : VisualRoutinesStateNode {

  $nodeclass LookForObjects : MapBuilderNode : doStart {
    mapreq.addObjectColor(lineDataType, "blue");
    mapreq.addObjectColor(blobDataType, "red");
    mapreq.addObjectColor(ellipseDataType, "green");

    LookForObjects =C=> SpeechNode("Done")



Let's look at the key features of the code above:

  • The parent class of the MapBuilderTest1 behavior must be VisualRoutinesStateNode. This is true for any behavior that makes use of the Tekkotsu crew.
  • LookForObjects is a MapBuilderNode. Inside its doStart method we reference the member variable mapreq, which holds the MapBuilderRequest object.
  • If we want to we can directly modify the fields of mapreq, but it's more convenient to call its addObjectColor method to set up the parameters for our request. In this example we're looking for three types of things: blue lines, red blobs, and green ellipses.
  • The constants lineDataType, blobDataType, and ellipseDataType are defined in the file Tekkotsu/DualCoding/ShapeTypes.h.
  • The color names "red", "green", and "blue" are defined in the file project/ms/config/rbgk.col, which Tekkotsu uses by default. If you define your own color segmentation scheme you can use any color names you like.


  1. Compile the above behavior on your robot.
  2. Run Tekkotsu and use the Seg viewer to monitor the what the camera is seeing.
  3. Lay out some shapes for the robot to look at. You can use colored masking tape or colored construction paper to make lines and circles, or random objects such as a soda can for blobs. Try wrapping the soda can in colored tape to give it a more uniform appearance.
  4. Run the MapBuilderTest1 behavior.
  5. Look on the Tekkotsu console (in the terminal window) for the results. How many lines, blobs, and ellipses did the MapBuilder see? Do the results match your expectations?

Using the SketchGUI

The SketchGUI is a tool for examining the state of Tekkotsu's vision system. Tekkotsu provides three distinct shape spaces, called camera, local, and world space, each of which has its own SketchGUI. In this lab we are focusing on just the camera space.

While your MapBuilderTest1 behavior is running, click on the C button in the ControllerGUI to bring up the camera space SketchGUI. You will see a collection of icons denoting the sketches and shapes that were created by the MapBuilder when it processed your request.

  • Click on "rawY" to see the raw intensity (grayscale) image from the camera.
  • Click on "camFrame" to see the color segmented camera image. Notice that the rawY display goes away when you click on camFrame, and vice versa. By default, the SketchGUI only shows one sketch at a time.
  • Hold down the Control key and click on a sketch. Using this technique you can select multiple sketches, so you can look at rawY and camFrame simultaneously. The SketchGUI will blend them together.

Assuming the MapBuilder found some shapes in the image, you will see icons for lines, blobs, and ellipses. To the right of each icon is the name of the shape, followed by a unique id number and some parameters that define the shape.

Properties of Shapes

Each type of shape has a set of properties that define that shape:

  • A line's properties include the coordinates of the start and end points, the length of the line, and the orientation of the line.
  • An ellipse's properties include the coordinates of the center of the ellipse, the orientation of the major axis, and the lengths of the semimajor and semiminor axes.
  • A blob's properties include the blob's orientation (assumed to be groundplane, but other values are possible), the coordinates of its centroid, and its area.

The SketchGUI display lists some of a shape's properties to help you identify it, but it does not list all of them. The coordinates of end points and centroids are given as (x,y,z) values, but for shapes in camera space, the z coordinate is always zero.

To do:

  • Look at LineData and EllipseData classes.
  • Rotate an ellipse and convert its orientation to degrees.

More SketchGUI Features

1. Move the mouse into the SketchGUI window and look in the window's lower left corner: you will see a pair of numbers. These are the mouse coordinates in pixels. In the top left you'll see the coordinates of the top left corner, normally (0,0), and in the bottom right corner you'll see the coordinates of that corner, normally (640,480).

  • From these values you can figure out the camera space coordinate system. The origin is in the upper left. The x coordinate increases to the right, so x ranges over columns. The y coordinate increases in the downward direction, so y ranges over rows. Thus a point (x,y) in the camera image is bounded by (0,0) and (width,height).
  • Tekkotsu also uses two other coordinate systems, called local space and camera space, that follow different conventions. Each has its own SketchGUI. We will explore local space in a subsequent vision lab; for now we'll stick to just camera space.

2. Click on the Crosshairs checkbox in the SketchGUI window and you'll see crosshairs appear. Using the mouse, verify that the center of the crosshairs is at (320,240).

3. Run the MapBuilderTest1 demo above, refresh the SketchGUI display, and move the mouse to the center of an ellipse or the endpoint of a line. Verify that the coordinates shown in the lower left corner of the SketchGUI window match the coordinates of that point given in the SketchGUI's shape list.

4. Click on the Invert All Shapes button in the SketchGUI and notice that the shapes change colors. Now click on the camFrame sketch in the SketchGUI and you'll see that the shapes stand out nicely against the camera image.