Skip to content Skip to navigation

Connexions

You are here: Home » Content » The Camera as a Sensor

Navigation

Recently Viewed

This feature requires Javascript to be enabled.
Download
x

Download module as:

  • PDF
  • EPUB (what's this?)

    What is an EPUB file?

    EPUB is an electronic book format that can be read on a variety of mobile devices.

    Downloading to a reading device

    For detailed instructions on how to download this content's EPUB to your specific device, click the "(what's this?)" link.

  • More downloads ...
Reuse / Edit
x

Module:

Add to a lens
x

Add module to:

Add to Favorites
x

Add module to:

 

The Camera as a Sensor

Module by: Davide Rocchesso. E-mail the author

Summary: Simple yet significant computer vision tasks are made accessible in the Processing language and environment. Pixel-by-pixel processing is possible through the GSVideo library. The JavacvPro library provides access to a large set of functions of OpenCV.

In Computer Science, computer vision studies the computational aspects of vision and develops camera-based systems as input devices. Simple techniques of pixel-by-pixel manipulation can be used to implement simple vision mechanisms. The examples that follow are taken from Learning Processing by Daniel Shiffman [link] and adapted to use the GSVideo library. A nice introductory article on Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers has been written by Golan Levin [link].

Color Tracking

In the code Code 4, the target color is searched pixel-by-pixel in the image buffer video, of size video.width * video.height. For example, the pixel of coordinates (2,3) (third row, fourth column) is accessed in position 3 + 2*video.width of the unrolled array containing all the pixels. The target color is selected via mouse clicking.


// Daniel Shiffman
// http://www.learningprocessing.com
// Example 16-11: Simple color tracking

// Modified by Davide Rocchesso - 2012

import codeanticode.gsvideo.*;

// Variable for capture device
GSCapture video;

// A variable for the color we are searching for.
color trackColor; 

void setup() {
  size(320,240);
  video = new GSCapture(this,width,height,15);
  video.start();
  // Start off tracking for red
  trackColor = color(255,0,0);
  smooth();
}

void draw() {
  // Capture and display the video
  if (video.available()) {
    video.read();
  }
  video.loadPixels();
  image(video,0,0);

  // Before we begin searching, the "world record" for closest color is set to a high number that is easy for the first pixel to beat.
  float worldRecord = 500; 

  // XY coordinate of closest color
  int closestX = 0;
  int closestY = 0;

  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width;
      // What is current color
      color currentColor = video.pixels[loc];
      float r1 = red(currentColor);
      float g1 = green(currentColor);
      float b1 = blue(currentColor);
      float r2 = red(trackColor);
      float g2 = green(trackColor);
      float b2 = blue(trackColor);

      // Using euclidean distance to compare colors
      float d = dist(r1,g1,b1,r2,g2,b2); // We are using the dist( ) function to compare the current color with the color we are tracking.

      // If current color is more similar to tracked color than
      // closest color, save current location and current difference
      if (d < worldRecord) {
        worldRecord = d;
        closestX = x;
        closestY = y;
      }
    }
  }

  // We only consider the color found if its color distance is less than 10. 
  // This threshold of 10 is arbitrary and you can adjust this number depending on how accurate you require the tracking to be.
  if (worldRecord < 10) { 
    // Draw a circle at the tracked pixel
    fill(trackColor);
    strokeWeight(4.0);
    stroke(0);
    ellipse(closestX,closestY,16,16);
  }
}

void mousePressed() {
  // Save color where the mouse is clicked in trackColor variable
  int loc = mouseX + mouseY*video.width;
  trackColor = video.pixels[loc];
}
    

Background Removal

By explicitely storing a background in a buffer, it is relatively easy to remove it from the images grabbed with a video camera, provided that the illumination condition remain constant and that the camera is not automatically adjusting its exposure. In the code Code 5, the background is stored upon a mouse click. At each frame, the color of each pixel is compared with the corresponding pixel in the background image, and if such difference is large enough it is considered to stay in the foreground.

// Daniel Shiffman
// http://www.learningprocessing.com
// Example 16-12: Simple background removal

// Modified by Davide Rocchesso - 2012

// Click the mouse to memorize a current background image
import codeanticode.gsvideo.*;

// Variable for capture device
GSCapture video;

// Saved background
PImage backgroundImage;

// How different must a pixel be to be a foreground pixel
float threshold = 20;

void setup() {
  size(320,240);
  video = new GSCapture(this, width, height, 30);
  video.start();
  // Create an empty image the same size as the video
  backgroundImage = createImage(video.width,video.height,RGB);
}

void draw() {
  // Capture video
  if (video.available()) {
    video.read();
  }
  
  // We are looking at the video's pixels, the memorized backgroundImage's pixels, as well as accessing the display pixels. 
  // So we must loadPixels() for all!
  loadPixels();
  video.loadPixels(); 
  backgroundImage.loadPixels();
  
  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      int loc = x + y*video.width; // Step 1, what is the 1D pixel location
      color fgColor = video.pixels[loc]; // Step 2, what is the foreground color
      
      // Step 3, what is the background color
      color bgColor = backgroundImage.pixels[loc];
      
      // Step 4, compare the foreground and background color
      float r1 = red(fgColor);
      float g1 = green(fgColor);
      float b1 = blue(fgColor);
      float r2 = red(bgColor);
      float g2 = green(bgColor);
      float b2 = blue(bgColor);
      float diff = dist(r1,g1,b1,r2,g2,b2);
      
      // Step 5, Is the foreground color different from the background color
      if (diff > threshold) {
        // If so, display the foreground color
        pixels[loc] = fgColor;
      } else {
        // If not, display green
        pixels[loc] = color(0,255,0); // We could choose to replace the background pixels with something other than the color green!
      }
    }
  }
  updatePixels();
}

void mousePressed() {
  // Copying the current frame of video into the backgroundImage object
  // Note copy takes 5 arguments:
  // The source image
  // x,y,width, and height of region to be copied from the source
  // x,y,width, and height of copy destination
  backgroundImage.copy(video,0,0,video.width,video.height,0,0,video.width,video.height);
  backgroundImage.updatePixels();
}

Motion Detection

To highlight motion, it is sufficient to compare, pixel-by-pixel, the current frame with the previous frame. In the code Section 3, if the color difference is larger than a threshold than the pixel is colored black.


// Daniel Shiffman
// http://www.learningprocessing.com
// Example 16-13: Simple motion detection

// Modified by Davide Rocchesso - 2012

import codeanticode.gsvideo.*;

// Variable for capture device
GSCapture video;
// Previous Frame
PImage prevFrame;
// How different must a pixel be to be a "motion" pixel
float threshold = 50;

void setup() {
  size(320,240);
  video = new GSCapture(this, width, height, 30);
  video.start();
  // Create an empty image the same size as the video
  prevFrame = createImage(video.width,video.height,RGB);
}

void draw() {
  
  // Capture video
  if (video.available()) {
    // Save previous frame for motion detection!!
    prevFrame.copy(video,0,0,video.width,video.height,0,0,video.width,video.height); // Before we read the new frame, we always save the previous frame for comparison!
    prevFrame.updatePixels();
    video.read();
  }
  
  loadPixels();
  video.loadPixels();
  prevFrame.loadPixels();
  
  // Begin loop to walk through every pixel
  for (int x = 0; x < video.width; x ++ ) {
    for (int y = 0; y < video.height; y ++ ) {
      
      int loc = x + y*video.width;            // Step 1, what is the 1D pixel location
      color current = video.pixels[loc];      // Step 2, what is the current color
      color previous = prevFrame.pixels[loc]; // Step 3, what is the previous color
      
      // Step 4, compare colors (previous vs. current)
      float r1 = red(current); float g1 = green(current); float b1 = blue(current);
      float r2 = red(previous); float g2 = green(previous); float b2 = blue(previous);
      float diff = dist(r1,g1,b1,r2,g2,b2);
      
      // Step 5, How different are the colors?
      // If the color at that pixel has changed, then there is motion at that pixel.
      if (diff > threshold) { 
        // If motion, display black
        pixels[loc] = color(0);
      } else {
        // If not, display white
        pixels[loc] = color(255);
      }
    }
  }
  updatePixels();
}

OpenCV

In 1999 Intel Research decided to develop a software library to encourage the development of CPU-intensive applications. Image processing and computer vision tasks certainly make intensive use of the CPU.

Objectives of OpenCV

  • to advance research by providing a set of optimized base functions;
  • to encourage the production of readable and reusable software;
  • to encourage the production of commercial software that uses an open and free infrastructure (through proper licensing).
The library is written in C language, for maximal efficiency and portability, but there are wrappers for C++ (openFrameworks), C#, Pyton, Java (JavaCV), and Processing (OpenCV and JavacvPro). It is available under several operating systems, including Linux, Windows, Mac OS, Android, and iOS. After several beta releases the first 1.0 version was released in 2006, and the 2.0 was released in october 2009, with a new C++ interface. At the moment of writing OpenCV 2.4 has been released and is available through the developer web site.

Within the Processing language and environment, a small set of OpenCV functions has been available for years through the OpenCV "hypermedia" library. The available functions were real-time capture, video file import, basic image treatment, face and body identification, and blob detection. A quick introduction to the use of this library can be found in Andy Best's tutorial or in Programming Interactivity by Joshua Noble [link]. This library, no longer updated, is now superseeded by the JavacvPro library, which has a much larger set of functions. The JavacvPro library makes its operations on a buffer, and it uses two auxiliary buffers where images can be stored and recalled with remember() and restore() operations. This allows, for example, to compute the difference between two neighboring frames.

Figure 1
Manipulation of buffers in OpencvPro
OpencvPro buffers

Example 1: Background subtraction


import codeanticode.gsvideo.*; 
import monclubelec.javacvPro.*;

OpenCV opencv;  //declare an OpenCV object
GSCapture cam;  //declare a camera object


void setup(){
   size(320,240);
   opencv = new OpenCV(this); //objectcv instance
   opencv.allocate(320,240);
   cam = new  GSCapture(this,320,240); //prepare for reading a video stream
   cam.start();
   }
   
void draw(){
  if (cam.available()){
   cam.read(); //frame grabbing
   opencv.copy(cam.get());
   opencv.absDiff(); //difference of frames: result in Memory2
   image(opencv.getMemory2(),0,0); //display the difference image 
  }
}

void keyPressed() {
   opencv.remember(); //store the frame in the memory buffer 
   println("remember");
}
   
   
// stop processing
public void stop(){
  cam.delete();
  super.stop();  
}
By addition of opencv.remember() as the last line of the internal block of the draw() function, the difference is computed between two contiguous frames, thus obtaining a visualization of motion. If, instead of the image(opencv.getMemory2(),0,0), we put the instructions

   opencv.restore2();
   opencv.gray();
   opencv.threshold(0.5);
   image(opencv.getBuffer(),0,0); 
the difference is reduced to gray levels and then thresholded at 0.5 level.

Example 2: Blob Tracking

Within a frame, a blob is a contiguous area represented in a data structure. A blob has a contour that represent a contrasting edge with neighboring regions. Tracking an object means to follow in time its corresponding blob that is moving across the screen. Previous-frame blobs are compared with the current-frame blobs to identify those blobs that are likely to keep representing the same object. The following code has been derived from Programming Interactivity by Joshua Noble [link], adapted to use the JavacvPro library.



import codeanticode.gsvideo.*;
import monclubelec.javacvPro.*;

int IMG_WIDTH = 320;
int IMG_HEIGHT = 240;
int BLURAMOUNT = 7;
float THRESHOLD = 0.8;
OpenCV opencv;
GSCapture cam;

void setup(){
 size(IMG_WIDTH*2, IMG_HEIGHT*2);
 cam = new GSCapture(this, IMG_WIDTH, IMG_HEIGHT);
 opencv = new OpenCV(this);
 opencv.allocate(IMG_WIDTH, IMG_HEIGHT);
 cam.start();
}

void draw(){
  if (cam.available() == true) {
   cam.read();
   opencv.copy(cam.get()); 
   opencv.remember();
   opencv.blur(BLURAMOUNT); 
   image(opencv.getBuffer(),IMG_WIDTH,0);
   opencv.restore();
   opencv.invert();
   image(opencv.getBuffer(),0,IMG_HEIGHT);
   opencv.restore();
   opencv.gray();
   opencv.threshold(THRESHOLD);
   image(opencv.getBuffer(),0,0);
  }
  
  // Args: minAreaIn, maxAreaIn, maxBlobIn, findHolesIn, maxVerticesIn, debug
  Blob[] blobs = opencv.blobs(opencv.area()/256, opencv.area()/2, 20, false, 100, false);  

  for (int i=0; i < blobs.length; i++){
    noFill();
    rect(blobs[i].rectangle.x, blobs[i].rectangle.y, blobs[i].rectangle.width, blobs[i].rectangle.height);
    stroke(0,0,255);
    line(blobs[i].centroid.x-5, blobs[i].centroid.y, blobs[i].centroid.x+5, blobs[i].centroid.y);
    line(blobs[i].centroid.x, blobs[i].centroid.y-5, blobs[i].centroid.x, blobs[i].centroid.y+5);
    fill(255,0,255,64); stroke(0,255,0);
    if (blobs[i].points.length > 0){
      beginShape();
        for (int j=0; j < blobs[i].points.length; j++){
           vertex(blobs[i].points[j].x, blobs[i].points[j].y);
        }
      endShape(CLOSE);
    }
  }
}

Example 3: Face Detection

With OpenCV, it is possible to load the configuration of a classifier that has been trained to recognize a specific shape, such as a human face or body.


import codeanticode.gsvideo.*; 
import monclubelec.javacvPro.*;
import java.awt.*; // for classes Point, Rectangle, ...

OpenCV opencv;  //declare an OpenCV object
GSCapture cam;  //declare a camera object


void setup(){
   size(320,240);
   opencv = new OpenCV(this); //objectcv instance
   opencv.allocate(320,240);
   cam = new  GSCapture(this,320,240); //prepare for reading a video stream
   cam.start();
   opencv.cascade("/usr/local/share/OpenCV/haarcascades/","haarcascade_frontalface_alt.xml"); // load the classifier configuration
   }
   
  
void draw(){
  if (cam.available()){
   cam.read(); //frame grabbing
   opencv.copy(cam.get());
   image(opencv.getBuffer(),0,0);
   Rectangle[] faces = opencv.detect(); //look for faces
   noFill();
   stroke(255,0,0);
   for (int i=0; i < faces.length; i++)
     rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);
  }
}

Content actions

Download module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks

Reuse / Edit:

Reuse or edit module (?)

Check out and edit

If you have permission to edit this content, using the "Reuse / Edit" action will allow you to check the content out into your Personal Workspace or a shared Workgroup and then make your edits.

Derive a copy

If you don't have permission to edit the content, you can still use "Reuse / Edit" to adapt the content by creating a derived copy of it and then editing and publishing the copy.