Extremely Fast and Simple WebGL Motion Detector to Rotate 3D Graphic


Google+ Facebook Twitter LinkedIn Dzone Reddit Digg Blogger Hacker News Addthis

By Markus SprunckRevision: 1.5; Status: final; Last Content Change: Apr 27, 2014;

In this WebGL experiment the browser captures the video signal and compares subsequent frames to detect motion of the user. With this signal the program finds the viewpoint and rotates the 3D cube. 

The algorithm is very simple, robust and extremely fast compared with classic face detection approaches. 

T
he code is based on Three.js (r56) and JavaScript. It is tested with Chrome (v34) and Firefox (v28). You may download the sources of this article in the last chapter of this article.

Expected Result

Open the WebGL experiment with a WebGL compatible Browser like Chrome you should see something like this:

Figure 1: Chrome asks for allowance to use the camera

Allow this site to access your camera, a small window appears at the top left corner. Now move your head in the direction you like to see. You may also use your hand to rotate the cube.

Figure 2: The cube rotates dependent from your movements

You can change the parameter in the GUI.

Motion Detection Algorithm

The underlying algorithm for the used motion detection is extremely simple.  

  1. First two frames of the video signal are compared. If the viewer moves his/her head the frames show small differences.  WebGL provides the possibility to compare these frames with HW acceleration, so this is extremely fast.  

  2. Then the average position of all the changed pixels (blue pixels) will be computed (small red cross). This moving average shows the average position of all movements in the screen – usually the head of the user. To improve the results the lower part of the screen (green pixels) will not be analyses and the number of pixels for the moving average is changeable. 

The algorithm is independent from the moving object, so you may also use a hand or other objects.  

Source Code

To run the code from a local folder you have to start Chrome with the following options:

chrome.exe --enable-media-stream --allow-file-access-from-files

// File 01: webgl_motion_detector.html - open this in the browser
<!doctype html>
<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0" />
  <style>
         a { color:#009de9; font-family: Helvetica; font-style: normal; font-size:10pt; text-align:left; }
         h1 { color:#FFF; font-family: Helvetica; font-size:14pt; text-align:left;padding-top:0px;padding-left:0px;padding-bottom:10px;margin:0;}
         .text { color:#FFF; font-family: Helvetica; font-size:10pt; text-align:left; padding-left:0px; padding-right:10px;}
         .main { background:#252525; padding:0px; }
        </style>
    <title>WebGL Motion Detector</title>
   </head>

  <body class="main">
   
    <script type="text/javascript" src="moving-averager.js"></script>
<script type="text/javascript" src="request-animation-frame.js"></script>
<script type="text/javascript" src="dat.gui.min.js"></script>
  <script type="text/javascript" src="simple-motion-detector.js"></script>
<script type="text/javascript" src="detector.js"></script>
<script type="text/javascript" src="glfx-neu.js"></script>
<script type="text/javascript" src="three.min.js"></script>
       
    <div id="header" >
       <h1>Extremely Fast and Simple WebGL Motion Detector to Rotate 3D Graphic</h1>

       <table>
       <tr>
       <div class="text">
     <a href="https://plus.google.com/u/0/117292523089281814301?rel=author">by Markus Sprunck</a><p/>
     In this experiment the browser captures the video signal and compares subsequent frames
     to detect motion of the user. With this signal the program finds the viewpoint and rotates
     the 3D cube. The algorithm is very simple, robust and extremely fast compared with classic
     face detection approaches. Many thanks to <i>Pierre-Loic Doulcet</i> for his inspiring chrome
     experiment <i>Pixelate Yourself</i>. Just allow this site to access your camera and move your head.
     The implementation is based on Three.js (r56). You may read more and get the source code
                    <a href="http://www.sw-engineering-candies.com/blog-1/extremely-fast-and-simple-webgl-motion-detector-to-rotate-3d-graphic">                     here</a>.
     </a>
     </div>
       </tr>
       </table>
    </div>
      
    <div id="drawingArea" style="position:absolute; top:155px;"></div>
  
<script type="text/javascript" src="webgl_motion_detector.js"></script>
           
  </body>
</html>

// File 02: webgl_motion_detector.js- draws the cube and calls the SimpleMotionDetector function
var stats, camera, scene, renderer;

if ( Detector.webgl ) {
init();
animate();
} else {
document.body.appendChild( Detector.getWebGLErrorMessage() );
}

function init() {

// add container
scene = new THREE.Scene();
var container = document.getElementById('drawingArea');

// add light
var light = new THREE.PointLight(0xffffff);
light.position.set( 0, 250, 0 );
scene.add(light);

// add cube with six textures
var materials = [];
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'xpos.png' ) }));
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'xneg.png' ) }));
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'ypos.png' ) }));
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'yneg.png' ) }));
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'zpos.png' ) }));
materials.push(new THREE.MeshBasicMaterial( { map: THREE.ImageUtils.loadTexture( 'zneg.png' ) }));
var geometry = new THREE.CubeGeometry( 3, 3, 3);
var cube = new THREE.Mesh( geometry, new THREE.MeshFaceMaterial(materials) );
scene.add( cube );

// add camera
camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 1, 2000 );
scene.add(camera);
camera.position.set(0 , 0, 10);
camera.lookAt(scene.position);

// add renderer
renderer = new THREE.WebGLRenderer();
renderer.setClearColorHex(0x0a0a0a, 1);

// Support window resize
    var resizeCallback = function () {
        var offsetHeight = 150;
        var devicePixelRatio = window.devicePixelRatio || 1;
        var width = window.innerWidth * devicePixelRatio - 25;
        var height = (window.innerHeight - offsetHeight -10)* devicePixelRatio;
        renderer.setSize(width, height);
        renderer.domElement.style.width = width + 'px';
        renderer.domElement.style.height = height + 'px';
        camera.updateProjectionMatrix();
    }
    window.addEventListener('resize', resizeCallback, false);
    resizeCallback();
container.appendChild( renderer.domElement );

// add motion detector
var motionDetector = new SimpleMotionDetector( camera );
motionDetector.domElement.style.position = 'absolute';
  motionDetector.domElement.style.left = '10px';
  motionDetector.domElement.style.top = '10px';
     motionDetector.init();
container.appendChild( motionDetector.domElement );

// dialog to change parameters
var gui = new dat.GUI({ autoPlace: false });
gui.add( motionDetector, 'offsetAlpha', -45.0, 45.0, 5 ).name( 'offset α' );
gui.add( motionDetector, 'offsetGamma', -45.0, 45.0, 5 ).name( 'offset γ' );
gui.add( motionDetector, 'amplificationAlpha', 1.0, 5.0, 0.5 ).name( 'amplification α' );
gui.add( motionDetector, 'amplificationGamma', 1.0, 5.0, 0.5 ).name( 'amplification γ' );
gui.add( motionDetector, 'detectionBorder', 0.25, 1.0, 0.05 ).name( 'detection border' );
gui.add( motionDetector, 'pixelThreshold', 100, 250, 10 ).name( 'pixel threshold' );
gui.add( motionDetector.averageX, 'maxLength', 200, 2000, 100 ).name( 'averager X' );
gui.add( motionDetector.averageY, 'maxLength', 200, 2000, 100 ).name( 'averager Y' );
gui.domElement.style.position = 'absolute';
gui.domElement.style.left = '10px';
gui.domElement.style.top = '210px';
container.appendChild(gui.domElement);
}

function animate() {
requestAnimationFrame( animate );
renderer.render( scene, camera );
}

// File 03:
 simple-motion-detector.js - captures the video input and rotates the camera position
/**
 * 
 * Captures the video signal and compares single frames to 
 * detect motion. The center of this motion is used to find 
 * the viewpoint of the user. With this viewpoint the camera 
 * position will be rotated. 
 *
 */
function SimpleMotionDetector( object ) {

// number of pixels for analysis
var PIXELS_HORIZONTAL = 50;
var PIXELS_VERTICAL = 50;

// size of info window
var WIDTH = 245;
var HEIGHT = 200;

// expected to be THREE.camera object
this.object = object;

// amplification factor for rotation (one is almost natural)
this.amplificationAlpha = 2.5;
this.amplificationGamma = 2.5;

// in degrees
this.offsetAlpha = 34.0;
this.offsetGamma = 10.0;

// just the upper part of the video should be detected
this.detectionBorder = 0.75;

// threshold of detected pixels
this.pixelThreshold = 120;

// average of all x positions of detected motion
this.averageX = new MovingAverager( 500 );
this.averageX.setValue( WIDTH / 2 );

// average of all y positions of detected motion
this.averageY = new MovingAverager( 500 );
this.averageY.setValue( HEIGHT / 2 );

var videoCanvas = document.createElement( 'canvas' );
videoCanvas.width = PIXELS_HORIZONTAL;
videoCanvas.height = PIXELS_VERTICAL;

var canvas = document.createElement( 'canvas' );
canvas.width = WIDTH;
canvas.height = HEIGHT;
canvas.style.position = 'absolute';
canvas.style.left = '0px';
canvas.style.top = '0px';

var videoContext = videoCanvas.getContext( '2d' );
var APP = {};
var simpleMotionDetector;
var texture = null;
var ctx = canvas.getContext( '2d' );
var video;
document.body.appendChild( canvas );

SimpleMotionDetector.prototype.init = function() {

simpleMotionDetector = this;

navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia
|| navigator.mozGetUserMedia || navigator.msGetUserMedia;

video = document.createElement( 'video' );
if ( navigator.getUserMedia ) {
navigator.getUserMedia( {audio: false, video: true},
function( stream ) {
video.src = window.URL.createObjectURL( stream );
APP.videoWidth = PIXELS_HORIZONTAL;
APP.videoHeight = PIXELS_VERTICAL;
APP.frontCanvas = document.createElement( 'canvas' );
APP.frontCanvas.width = APP.videoWidth;
APP.frontCanvas.height = APP.videoHeight * 2;
APP.ctx = APP.frontCanvas.getContext( '2d' );
APP.comp = [];
simpleMotionDetector.run( );
},
function( e ) {
alert( 'getUserMedia did not succeed.\n\ncode=' + e.code );
}
);
} else {
alert( 'Your browser does not seem to support UserMedia' )
}

requestAnimFrame = ( function( ) {
return window.requestAnimationFrame || window.webkitRequestAnimationFrame ||
window.mozRequestAnimationFrame || window.oRequestAnimationFrame || window.msRequestAnimationFrame ||
function( /* function FrameRequestCallback */ callback, /* DOMElement Element */ element ) {
window.setTimeout( callback, 1000/60 );
};
} )( );

}

SimpleMotionDetector.prototype.analyisMotionPicture = function() {
  videoContext.drawImage( canvas, 0, 0 );
  var data = videoContext.getImageData( 0, 0, PIXELS_HORIZONTAL, PIXELS_VERTICAL ).data;

  ctx.fillStyle = '#FFFFFF';
  ctx.fillRect( 0,0, WIDTH, HEIGHT );
  ctx.globalAlpha = 0.2;

  var cubeWidth = WIDTH / PIXELS_HORIZONTAL - 1 | 0;
var cubeHeight = HEIGHT / PIXELS_VERTICAL - 1 | 0;

var yTopPosition = Number.MAX_VALUE;
for ( var y = 0; y < PIXELS_VERTICAL-1; y++ ) {
for ( var x = 0; x < PIXELS_HORIZONTAL; x++ ) {
if ( data[x * 4 + y * PIXELS_HORIZONTAL * 4] > this.pixelThreshold ) {
var xPos = x * WIDTH / PIXELS_HORIZONTAL;
var yPos = y * HEIGHT / PIXELS_VERTICAL;
if ( y < PIXELS_VERTICAL * this.detectionBorder ) {
if ( yTopPosition >= Math.min( yTopPosition, yPos ) ) {
yTopPosition = yPos;
this.averageX.setValue( xPos );
this.averageY.setValue( yPos );
ctx.fillStyle = '#000000';
ctx.fillRect( xPos, yPos, cubeWidth , cubeHeight );
}
ctx.fillStyle = '#0000FF';
ctx.fillRect( xPos, yPos, cubeWidth , cubeHeight );
} else {
ctx.fillStyle = '#00FF00';
ctx.fillRect( xPos, yPos, cubeWidth , cubeHeight );
}
}
}
}

// print red cross
ctx.fillStyle = '#FF0000';
ctx.fillRect( simpleMotionDetector.averageX.getValue( )                                 - cubeWidth*0.5, simpleMotionDetector.averageY.getValue( ), cubeWidth*1.5 , cubeHeight*0.5 );
ctx.fillRect( simpleMotionDetector.averageX.getValue( ), simpleMotionDetector.averageY.getValue( )                                 - cubeHeight*0.5, cubeWidth*0.5 , cubeHeight*1.5 );
}

SimpleMotionDetector.prototype.updateCameraPosition = function() {
     var distanceFromMiddleX = this.averageX.getValue( ) * PIXELS_HORIZONTAL / WIDTH - PIXELS_HORIZONTAL / 2;
var alpha = this.amplificationAlpha * Math.asin( distanceFromMiddleX / PIXELS_HORIZONTAL )                                 + Math.PI / 180 *this.offsetAlpha ;

var distanceFromMiddleY = ( PIXELS_VERTICAL / 2 - this.averageY.getValue( ) / HEIGHT * PIXELS_VERTICAL );
var gamma = this.amplificationGamma * Math.asin( distanceFromMiddleY / PIXELS_VERTICAL )                                 + Math.PI / 180 *this.offsetGamma ;

var x = camera.position.x;
var z = camera.position.z;
var y = camera.position.y;
var radius = Math.sqrt( x*x + y*y + z*z );

camera.position.x = radius * Math.sin( alpha )* Math.cos( gamma );
camera.position.z = radius * Math.cos( alpha )* Math.cos( gamma );
camera.position.y = radius * Math.sin( gamma );

camera.lookAt( scene.position );
}


SimpleMotionDetector.prototype.analyseVideo = function() {
requestAnimFrame( SimpleMotionDetector.prototype.analyseVideo );
videoContext.drawImage( video,0,0, PIXELS_HORIZONTAL, PIXELS_VERTICAL );
APP.ctx.drawImage( videoCanvas, 0, 0 );
texture.loadContentsOf( APP.frontCanvas );
canvas.draw( texture );
canvas.mirror( );
canvas.move( );
canvas.update( );
   APP.ctx.drawImage( videoCanvas, 0, PIXELS_VERTICAL );
simpleMotionDetector.analyisMotionPicture( );
simpleMotionDetector.updateCameraPosition( );
}

SimpleMotionDetector.prototype.run = function() {
   canvas = fx.canvas( );
   texture = canvas.texture( APP.frontCanvas );
   video.play( );
   this.analyseVideo( );
}

SimpleMotionDetector.prototype.domElement = canvas;
  }

// File 04: moving-averager.js - this is a not performance optimized class 

/**
 *
 * Calculates the moving average of a number of data
 * points.  
 *
 */

function MovingAverager(length) {
this.maxLength = length;
this.nums = [];
}

MovingAverager.prototype.setValue = function(num) {
this.nums.push(num);
if (this.nums.length > this.maxLength) {
this.nums.splice(0, this.nums.length - this.maxLength);
}
}

MovingAverager.prototype.getValue = function() {
var sum = 0.0;
for (var i in this.nums) {
sum += this.nums[i];
}
return (sum / this.nums.length);
};


Please, do not hesitate to contact me if you have any ideas for improvement and/or you find a bug in the sample code. 
 

Change History

 Revision  Date  Author  Description
 1.0  Mar 7, 2013  Markus Sprunck  first version
 1.1  Mar 8, 2013  Markus Sprunck  changes to support 'THREE.WebGLRenderer 56'
 1.2  Mar 23, 2013  Markus Sprunck  simple gui added
 1.3  Apr 7, 2013  Markus Sprunck  source code now on GitHub
 1.4  Jun 7, 2013   Markus Sprunck  some words about the algorithm
 1.5  Apr 27, 2014  Markus Sprunck  Fix problem in patched glfx.js

Google+ Comments

You may press the +1 button to share and/or comment