Headtracking

To get a little bit variety to shaders I started another project on which I am going to work all the time I have absolutly no motivation for shaders.
I worked on it for about 3 days now from which the first one was just getting a bit into the stuff, the second was writing the dll and the third day was playing around with headtracking and trying to realize motion detection (my biggest problem with this is the background in my room and my bad webcam).

The idea is simple and I guess it is the same the wiimote uses. It is just getting the images from the webcam, looking for bright areas and getting the positions of their centers and sorting those positions within an array depending on the area size. These positions (up to 20) can then be accessed with Lite-C.

I am currently in the need of some LEDs to do further stuff with headtracking. My goal is to move the player completely with the head. And if I am able to do that, archieving the 3D effect which can be seen here: http://www.youtube.com/watch?v=Jd3-eiid-Uw
I don´t think that this stuff could really be used within a game, but at least it is a lot of fun playing around with it.

My try for motion detection for games like the ones for the eye toy or camgoo (www.camgoo.de) is creating a picture of the background without the player and comparing it with the current picture with the player, making differnces white and everything else black. It does basicly work but needs still a lot of work.

Here are two earlier screenshots showing the pointdetection:



And here is a short clip showing some first trying around with headtracking.
I just cleaved a flashlight to a cap which is used for the detection.
Sorry for the mess :P :
http://www.youtube.com/watch?v=qj_kS2hIw5U

I hope you like it
Slin

Looks very cool indeed. I think that if it can be made more reliable (Perhaps you could use a coloured light, thus allowing it to be more easily differentiated from background lights) it could definitely be used in a first person shooter.

...dodging bullets matrix style!...

Anyhow great work!

cartoon_baboon

I think its amazing, i would want this for all my games the experience is brilliant!

looks nice, it would be cool as the camera control, excpet u can;t see the screen when looking the other way

I don't understand how you want to use the head to move forward... can you elaborate? I only get the rotating idea.. not moving...

You can tilt or step forward to the screen. I assume you don't have to move forwards by using the head tracking motion though.

I plan to do it through moving the head into the direction the player has to go. The rotating could probably be used in a real game but I don´t think that the movement can really be done like that.


That is a great idea, at least for a minigame.
It works already very reliable as long as there is no bright light directly behind you (at least when substracting the background without the playing person).

Thanks for your comments

Nice concepts Slin
I know the shader world can be a lonely one thus needing some distraction can be necessary ^^
Looking forward to see more of this

Cheers

Frazzle

Most webcams can be easily modified to use infra-red, like the wiimotes.

There is a small glass filter inside of most webcams, this can be replaced by a filter like on the end of a TV remote control.

Not only would the modification allow the camera to detect infra-red light, but it would also prevent visible light from entering, thus replicating the sensor on the front of the Wiimote.

I tried the mod to an old digital camera, but got sloppy and it broke after working for a short while. Never take experimental projects to school with you.

what i'd do is make it track you heads movement not rotation so the screen rotates, but you wont have to look away, but when you move your head you perspective changes

this would be (in my mind) used if you were to make avertual reality game were the player were a sort of helmet or somthing of that nature which contained a screen so the players eyes never leaves the screen and a wirless keyboard/controler would be recomended if the player desides to do a 360 and look behind himself.

and now that i added my two cents just have to say i am really impressed you showed me something that i did not even know gamestudio was capible of congrats

Hi everybody,

I've first heard of this technic when Jonny Lee's demonstration of the inverted Wii-Remote was shown on several websites. I was truly blown away by the very natural effect this simple modification can produce. You can actually look at the screen as if you where looking out of a window. Or if you were to pass a screen like this you wouldn't be able to say if it where a screen or if there where dept to it. This combined with a pair of 3D Glasses... how cool would that be!
The guys in this video have actually gone a step further by tracking the players face and massuring the distance between the eyes. It seems to work perfectly and is much more userfriendly.
Head Tracking with ONLY a Webcam (Youtube)

I don't really see how this could be used for moving the player in a game but rather to give the (physicle) player some freedom to move around in front of the screen and get a response by the camera (change of angle). I think it could add to the overall feel of the game.

Well, my webcam is broken at the moment but as soon as I get it working agein and my (very) old 3D Relevator Glasses I will try this out and report back.


So long...
Greetz from Zürich
Hitsch

I already saw that "Face API", but I won´t be able to create something like that on my own, and they haven´t released it to the public yet.

I had this idea too and will definatly give it a try But with red/cyan glasses and not with shutterglasses.


I would integrate a compass into that hardware instead of using a webcam :P

great work slin~~~ but there is something i didn't understand:
1 how do u get the image from web-cam ?does it through the dll u made ?

2 i have seen the video of wii head-tracking and face API,and i have tried to do that in 3dgs,but i can't get the exact result(mine is not that reality,i use camera in 3dgs).after a lot of work i found that they seems not useing the "camera" like things in 3dgs,but useing some projection matrix to get image?

that makes me confused,can u or anyone give me some advice about how can i do that with lite-c?

3i didn't understand how can u recognize the brighter field?useing texture samples of shader?how to get the center point of the field?
thanks very much,and to ur great work

1) Yes, I have written a dll which helps me to get the image of my webcam into the engine.

2) If I got your question right, you will have to manipulate the projection matrix, yes.

3) I am just using some simple flood filling on the cpu as I have the image as an array of rgb values.

I hope that helps you a little.

well ,thank u very much.but i m still confused what should i do with the projection matrix in 3dgs to figur that out? the only way to change the image i know is by vertex shader......or there is some other ways to do that ? plz lead me a liite more or give me some links ?i have no idea what to search...

btw: if ur dll works u could do multi-touch with 3dgs,that will be wonderful.u can also take a look at nui.org(i not sure).there is something about how to get a better result(with some image filter)

Yes I could do multi touch especially if I would surround my camera with infrared LOD would replace its infrared filter with a filter for the visible light and put the whole thing behind my monitor

This may helps you a little:
http://msdn.microsoft.com/en-us/library/bb205353(VS.85).aspx
http://www.cs.nps.navy.mil/people/faculty/capps/iap/class2/viewing/projection.html
You will have to build an off-center projection matrix depending on the viewers position. You may also have to move the view a little for good results, but I don't really know. It isn't very easy, at least not for me and I think that you won't find too much help.
I haven't tried that yet and yes that matrix is used in a vertexshader but I would recommend you to just manipulate the engine matrices in a material or render event.

thank u slin!i think that may be the point i missed.i will try to figur it out.
but it seems difficult for me,because even u r not sure on working it out.however i will try.
thank u ~~~~

I have tried headtracking via Wiimote.

The proejction I changed using a shader, that worked fine.
Still I didn't get the real 3d feeling. I think Johnny also had some special code for the "targets" in his demo.
When I have more time again maybe I'll dig up this project again.

Looks good. Nice work Slin!

Great work Slin!
I think you plug-in could be use to make motion capture. In theory possible to write programm in Lite-c to work with motion capture? For example: use entity with bones, then sychronize moving bones in 3d word and in web-cam. It is possible?

It should of course be possible, but I guess you´ll need a good webcam for that and will have to put quite a bit of work into it until it works well
But yes, sure, that should be possible.

Hmmm... I think that I saw a project using motion capture and 3dgs in the past.
Damn I can't remember who made it!

That was just some simple motion detection and it was using my plugin
It was done by cerberi_croman.

There's now a Demo of the faceAPI tracking tool. It works great with my laptop web cam if the light is right.

They have also released a non commercial API now!

faceAPI

I'm not much of a programmer anything apart from some lite-C, so I have no idea how hard it would be to get this working with 3DGS. From what I could understand about the specifications somebody, like Slin :), could wright a wrapper for this and everybody could use it in their projects.

This tool could be used to determine the distance, angle and rotation of the player to the screen and open up a lot of interaction possibilities!

Thanks a lot for mentioning it.
The just downloaded the non commercial api and will check it out in the next days