Extremely Fast and Simple WebGL Motion Detector to Rotate 3D Graphic

BY MARKUS SPRUNCK

In this WebGL experiment the browser captures the video signal and compares subsequent frames to detect motion of the user. With this signal the program finds the viewpoint and rotates the 3D cube.  You may start the web application with a WebGL enabled browser here http://webgl-motion-detector.appspot.com/.

The algorithm is very simple, robust and extremely fast compared with classic face detection approaches. 
The code is based on Three.js (r56) and JavaScript. It is tested with Chrome (v34) and Firefox (v28). You may download the sources of this article in the last chapter of this article.

Expected Result

Open with a WebGL compatible Browser like Chrome you should see something like this:

Figure 1: Chrome asks for allowance to use the camera

Allow this site to access your camera, a small window appears at the top left corner. Now move your head in the direction you like to see. You may also use your hand to rotate the cube.

Figure 2: The cube rotates dependent from your movements

You can change the parameter in the GUI.

Motion Detection Algorithm

The underlying algorithm for the used motion detection is extremely simple.  

  1. First two frames of the video signal are compared. If the viewer moves his/her head the frames show small differences.  WebGL provides the possibility to compare these frames with HW acceleration, so this is extremely fast.  

  2. Then the average position of all the changed pixels (blue pixels) will be computed (small red cross). This moving average shows the average position of all movements in the screen – usually the head of the user. To improve the results the lower part of the screen (green pixels) will not be analyses and the number of pixels for the moving average is changeable. 

The algorithm is independent from the moving object, so you may also use a hand or other objects.  

Source Code

To run the code from a local folder you have to start Chrome with the following options:

chrome.exe --enable-media-stream --allow-file-access-from-files

Open file webgl_motion_detector.html in the web browser.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<!DOCTYPE html>
<html>
<head>
    <meta content="text/html; charset=utf-8" http-equiv="content-type">
    <meta content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0"
    name="viewport">
    <style>
        a { color:#009de9; font-family: Helvetica; font-style: normal; font-size:10pt; text-align:left; }  
        h1 { color:#FFF; font-family: Helvetica; font-size:14pt; text-align:left;padding-top:0px;padding-left:0px;padding-bottom:10px;margin:0;}
        .text { color:#FFF; font-family: Helvetica; font-size:10pt; text-align:left; padding-left:0px; padding-right:10px;}    
        .main { background:#252525; padding:0px; }
    </style>
    <title>WebGL Motion Detector</title>
</head>

<body class="main">
    <script src="moving-averager.js" type="text/javascript"></script> <script src="request-animation-frame.js" type="text/javascript"></script>
    <script src="dat.gui.min.js" type="text/javascript"></script> <script src="simple-motion-detector.js" type="text/javascript"></script> 
    <script src="detector.js" type="text/javascript"></script> <script src="glfx-neu.js" type="text/javascript"></script> 
    <script src="three.min.js" type="text/javascript"></script>
    <div id="header">
        <h1>Extremely Fast and Simple WebGL Motion Detector to Rotate 3D Graphic</h1>
        <div class="text">
            <a href="https://plus.google.com/u/0/117292523089281814301?rel=author">by Markus Sprunck</a>
            <p>In this experiment the browser captures the video signal and compares subsequent
            frames to detect motion of the user. With this signal the program finds the viewpoint
            and rotates the 3D cube. The algorithm is very simple, robust and extremely fast
            compared with classic face detection approaches. Many thanks to <i>Pierre-Loic
            Doulcet</i> for his inspiring chrome experiment <i>Pixelate Yourself</i>. Just allow
            this site to access your camera and move your head. The implementation is based on
            Three.js (r56). You may read more and get the source code <a href=
            "http://www.sw-engineering-candies.com/blog-1/extremely-fast-and-simple-webgl-motion-detector-to-rotate-3d-graphic">
            here</a>.</p>
        </div>
    </div>
    <div id="drawingArea" style="position:absolute; top:155px;"></div><script src="webgl_motion_detector.js" type="text/javascript"></script>
</body>
</html>

// File 02: webgl_motion_detector.js- draws the cube and calls the SimpleMotionDetector function

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
var stats, camera, scene, renderer;
if (Detector.webgl) {
    init();
    animate();
} else {
    document.body.appendChild(Detector.getWebGLErrorMessage());
}

function init() {
    // add container
    scene = new THREE.Scene();
    var container = document.getElementById('drawingArea');
    // add light
    var light = new THREE.PointLight(0xffffff);
    light.position.set(0, 250, 0);
    scene.add(light);
    // add cube with six textures
    var materials = [];
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('xpos.png')
    }));
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('xneg.png')
    }));
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('ypos.png')
    }));
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('yneg.png')
    }));
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('zpos.png')
    }));
    materials.push(new THREE.MeshBasicMaterial({
        map: THREE.ImageUtils.loadTexture('zneg.png')
    }));
    var geometry = new THREE.CubeGeometry(3, 3, 3);
    var cube = new THREE.Mesh(geometry, new THREE.MeshFaceMaterial(materials));
    scene.add(cube);
    // add camera
    camera = new THREE.PerspectiveCamera(50, window.innerWidth / window.innerHeight, 1, 2000);
    scene.add(camera);
    camera.position.set(0, 0, 10);
    camera.lookAt(scene.position);
    // add renderer
    renderer = new THREE.WebGLRenderer();
    renderer.setClearColorHex(0x0a0a0a, 1);
    // Support window resize
    var resizeCallback = function() {
        var offsetHeight = 150;
        var devicePixelRatio = window.devicePixelRatio || 1;
        var width = window.innerWidth * devicePixelRatio - 25;
        var height = (window.innerHeight - offsetHeight - 10) * devicePixelRatio;
        renderer.setSize(width, height);
        renderer.domElement.style.width = width + 'px';
        renderer.domElement.style.height = height + 'px';
        camera.updateProjectionMatrix();
    }
    window.addEventListener('resize', resizeCallback, false);
    resizeCallback();
    container.appendChild(renderer.domElement);
    // add motion detector
    var motionDetector = new SimpleMotionDetector(camera);
    motionDetector.domElement.style.position = 'absolute';
    motionDetector.domElement.style.left = '10px';
    motionDetector.domElement.style.top = '10px';
    motionDetector.init();
    container.appendChild(motionDetector.domElement);
    // dialog to change parameters
    var gui = new dat.GUI({
        autoPlace: false
    });
    gui.add(motionDetector, 'offsetAlpha', -45.0, 45.0, 5).name('offset α');
    gui.add(motionDetector, 'offsetGamma', -45.0, 45.0, 5).name('offset γ');
    gui.add(motionDetector, 'amplificationAlpha', 1.0, 5.0, 0.5).name('amplification α');
    gui.add(motionDetector, 'amplificationGamma', 1.0, 5.0, 0.5).name('amplification γ');
    gui.add(motionDetector, 'detectionBorder', 0.25, 1.0, 0.05).name('detection border');
    gui.add(motionDetector, 'pixelThreshold', 100, 250, 10).name('pixel threshold');
    gui.add(motionDetector.averageX, 'maxLength', 200, 2000, 100).name('averager X');
    gui.add(motionDetector.averageY, 'maxLength', 200, 2000, 100).name('averager Y');
    gui.domElement.style.position = 'absolute';
    gui.domElement.style.left = '10px';
    gui.domElement.style.top = '210px';
    container.appendChild(gui.domElement);
}

function animate() {
    requestAnimationFrame(animate);
    renderer.render(scene, camera);
}

// File 03:
 simple-motion-detector.js - captures the video input and rotates the camera position

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
/**
 *
 * Captures the video signal and compares single frames to
 * detect motion. The center of this motion is used to find
 * the viewpoint of the user. With this viewpoint the camera
 * position will be rotated.
 *
 */
function SimpleMotionDetector(object) {
    // number of pixels for analysis
    var PIXELS_HORIZONTAL = 50;
    var PIXELS_VERTICAL = 50;
    // size of info window
    var WIDTH = 245;
    var HEIGHT = 200;
    // expected to be THREE.camera object
    this.object = object;
    // amplification factor for rotation (one is almost natural)
    this.amplificationAlpha = 2.5;
    this.amplificationGamma = 2.5;
    // in degrees
    this.offsetAlpha = 34.0;
    this.offsetGamma = 10.0;
    // just the upper part of the video should be detected
    this.detectionBorder = 0.75;
    // threshold of detected pixels
    this.pixelThreshold = 120;
    // average of all x positions of detected motion
    this.averageX = new MovingAverager(500);
    this.averageX.setValue(WIDTH / 2);
    // average of all y positions of detected motion
    this.averageY = new MovingAverager(500);
    this.averageY.setValue(HEIGHT / 2);
    var videoCanvas = document.createElement('canvas');
    videoCanvas.width = PIXELS_HORIZONTAL;
    videoCanvas.height = PIXELS_VERTICAL;
    var canvas = document.createElement('canvas');
    canvas.width = WIDTH;
    canvas.height = HEIGHT;
    canvas.style.position = 'absolute';
    canvas.style.left = '0px';
    canvas.style.top = '0px';
    var videoContext = videoCanvas.getContext('2d');
    var APP = {};
    var simpleMotionDetector;
    var texture = null;
    var ctx = canvas.getContext('2d');
    var video;
    document.body.appendChild(canvas);
    SimpleMotionDetector.prototype.init = function() {
        simpleMotionDetector = this;
        navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia ||
            navigator.mozGetUserMedia || navigator.msGetUserMedia;
        video = document.createElement('video');
        if (navigator.getUserMedia) {
            navigator.getUserMedia({
                audio: false,
                video: true
            }, function(stream) {
                video.src = window.URL.createObjectURL(stream);
                APP.videoWidth = PIXELS_HORIZONTAL;
                APP.videoHeight = PIXELS_VERTICAL;
                APP.frontCanvas = document.createElement('canvas');
                APP.frontCanvas.width = APP.videoWidth;
                APP.frontCanvas.height = APP.videoHeight * 2;
                APP.ctx = APP.frontCanvas.getContext('2d');
                APP.comp = [];
                simpleMotionDetector.run();
            }, function(e) {
                alert('getUserMedia did not succeed.\n\ncode=' + e.code);
            });
        } else {
            alert('Your browser does not seem to support UserMedia')
        }
        requestAnimFrame = (function() {
            return window.requestAnimationFrame || window.webkitRequestAnimationFrame ||
                window.mozRequestAnimationFrame || window.oRequestAnimationFrame ||
                window.msRequestAnimationFrame || function( /* function FrameRequestCallback */
                    callback, /* DOMElement Element */ element) {
                    window.setTimeout(callback, 1000 / 60);
                };
        })();
    }
    SimpleMotionDetector.prototype.analyisMotionPicture = function() {
        videoContext.drawImage(canvas, 0, 0);
        var data = videoContext.getImageData(0, 0, PIXELS_HORIZONTAL, PIXELS_VERTICAL).data;
        ctx.fillStyle = '#FFFFFF';
        ctx.fillRect(0, 0, WIDTH, HEIGHT);
        ctx.globalAlpha = 0.2;
        var cubeWidth = WIDTH / PIXELS_HORIZONTAL - 1 | 0;
        var cubeHeight = HEIGHT / PIXELS_VERTICAL - 1 | 0;
        var yTopPosition = Number.MAX_VALUE;
        for (var y = 0; y < PIXELS_VERTICAL - 1; y++) {
            for (var x = 0; x < PIXELS_HORIZONTAL; x++) {
                if (data[x * 4 + y * PIXELS_HORIZONTAL * 4] > this.pixelThreshold) {
                    var xPos = x * WIDTH / PIXELS_HORIZONTAL;
                    var yPos = y * HEIGHT / PIXELS_VERTICAL;
                    if (y < PIXELS_VERTICAL * this.detectionBorder) {
                        if (yTopPosition >= Math.min(yTopPosition, yPos)) {
                            yTopPosition = yPos;
                            this.averageX.setValue(xPos);
                            this.averageY.setValue(yPos);
                            ctx.fillStyle = '#000000';
                            ctx.fillRect(xPos, yPos, cubeWidth, cubeHeight);
                        }
                        ctx.fillStyle = '#0000FF';
                        ctx.fillRect(xPos, yPos, cubeWidth, cubeHeight);
                    } else {
                        ctx.fillStyle = '#00FF00';
                        ctx.fillRect(xPos, yPos, cubeWidth, cubeHeight);
                    }
                }
            }
        }
        // print red cross
        ctx.fillStyle = '#FF0000';
        ctx.fillRect(simpleMotionDetector.averageX.getValue() - cubeWidth * 0.5,
            simpleMotionDetector.averageY.getValue(), cubeWidth * 1.5, cubeHeight * 0.5
        );
        ctx.fillRect(simpleMotionDetector.averageX.getValue(), simpleMotionDetector.averageY
            .getValue() - cubeHeight * 0.5, cubeWidth * 0.5, cubeHeight * 1.5);
    }
    SimpleMotionDetector.prototype.updateCameraPosition = function() {
        var distanceFromMiddleX = this.averageX.getValue() * PIXELS_HORIZONTAL / WIDTH -
            PIXELS_HORIZONTAL / 2;
        var alpha = this.amplificationAlpha * Math.asin(distanceFromMiddleX /
            PIXELS_HORIZONTAL) + Math.PI / 180 * this.offsetAlpha;
        var distanceFromMiddleY = (PIXELS_VERTICAL / 2 - this.averageY.getValue() / HEIGHT *
            PIXELS_VERTICAL);
        var gamma = this.amplificationGamma * Math.asin(distanceFromMiddleY /
            PIXELS_VERTICAL) + Math.PI / 180 * this.offsetGamma;
        var x = camera.position.x;
        var z = camera.position.z;
        var y = camera.position.y;
        var radius = Math.sqrt(x * x + y * y + z * z);
        camera.position.x = radius * Math.sin(alpha) * Math.cos(gamma);
        camera.position.z = radius * Math.cos(alpha) * Math.cos(gamma);
        camera.position.y = radius * Math.sin(gamma);
        camera.lookAt(scene.position);
    }
    SimpleMotionDetector.prototype.analyseVideo = function() {
        requestAnimFrame(SimpleMotionDetector.prototype.analyseVideo);
        videoContext.drawImage(video, 0, 0, PIXELS_HORIZONTAL, PIXELS_VERTICAL);
        APP.ctx.drawImage(videoCanvas, 0, 0);
        texture.loadContentsOf(APP.frontCanvas);
        canvas.draw(texture);
        canvas.mirror();
        canvas.move();
        canvas.update();
        APP.ctx.drawImage(videoCanvas, 0, PIXELS_VERTICAL);
        simpleMotionDetector.analyisMotionPicture();
        simpleMotionDetector.updateCameraPosition();
    }
    SimpleMotionDetector.prototype.run = function() {
        canvas = fx.canvas();
        texture = canvas.texture(APP.frontCanvas);
        video.play();
        this.analyseVideo();
    }
    SimpleMotionDetector.prototype.domElement = canvas;
}

// File 04: moving-averager.js - this is a not performance optimized class

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/**
 *
 * Calculates the moving average of a number of data
 * points.
 *
 */
function MovingAverager(length) {
    this.maxLength = length;
    this.nums = [];
}
MovingAverager.prototype.setValue = function(num) {
    this.nums.push(num);
    if (this.nums.length > this.maxLength) {
        this.nums.splice(0, this.nums.length - this.maxLength);
    }
}
MovingAverager.prototype.getValue = function() {
    var sum = 0.0;
    for (var i in this.nums) {
        sum += this.nums[i];
    }
    return (sum / this.nums.length);
};

Change History

 Revision  Date  Author  Description
 1.0  Mar 7, 2013  Markus Sprunck  first version
 1.1  Mar 8, 2013  Markus Sprunck  changes to support 'THREE.WebGLRenderer 56'
 1.2  Mar 23, 2013  Markus Sprunck  simple gui added
 1.3  Apr 7, 2013  Markus Sprunck  source code now on GitHub
 1.4  Jun 7, 2013   Markus Sprunck  some words about the algorithm
 1.5  Apr 27, 2014  Markus Sprunck  fix problem in patched glfx.js

Sponsored Link