Playing with pixels in depth with Kinect, part 2

07.02.12 George Profenza

As if the previous post wasn't geeky enough, here's a quick look at project that also ties in a bit of computer vision and neural networks.

One one of the courses, related to Programming for Architecture and Design, among other things, we had a lecture and tutorial on Neural Networks. There are multiple types of Neural Networks, mainly classified as supervised and unsupervised, based on how this networks learn.

Kohonen networks (which is what this post focuses on) are unsupervised networks, also known as a self organizing maps (SOM). As opposed to supervised networks, where neurons are trained what the output should be like (should weigh towards), this type of network is based on competitive learning - the outputs/neurons organize themselves towards the closest inputs. This idea of competitive learning is based on how it is thought the hippocampus(the part of our brain responsible for navigation) works. In a sense, the outputs display a particle-spring like behaviour towards the inputs, which make this type of network useful for surface fitting/dimensionality reduction/etc.

Initially a dataset of 3d points was given, but it thought it would be more fun for some reason to fit a surface on my face (or any face for that matter). This is what the video illustrated:

  • Computer vision(OpenCV's HAAR cascade feature) is used to detect faces and isolate an area in the Kinect depth map
  • depth pixels belong to the face are converted to 3D coordinates
  • once a point cloud was selected, the points can be fed as the inputs of the neural net and the outputs are vertices of the surface. The number of ouputs is variable, so a low-poly mesh can also be calculated

The mesh can also be saved to AutoCAD (.dxf) format, which is what I've used to render a creepy theatre like mask based on Max's face. Currently the default surface is a rectangular grid, which is a good start, but not ideal for fitting on a face. If you can imagine a face unwrapped into 2D space, it would not look like a perfect rectangular, but that's something to explore at a later time.

In the meantime, if you would like to have a play with the code, the source is included. If this leads to something interesting let us know. The code is written using Processing and uses OpenKinect and OpenCV. If you think this is something you would like explained further, leave a comment bellow and we'll post more details on the wiki.

Max Mask Max face point cloud 1 Max face point cloud 2




Playing with pixels in depth with Kinect, part 1

07.02.12 George Profenza

I've managed to get a bit of breathing time so I thought about posting a few nerdy bits and pieces. Currently I'm doing an MA in Adaptive Architecture and Computation at UCL which is pretty cool, but keeps me pretty busy lately. Been learning up a lot of new skills there, among others, using the Kinect Sensor.

In this post I'll demo a few things I've learned.

I'll start with a quick technical demo of what I was able to achieve using Kinect and Processing. It displays the following:

  • user isolation
  • stereo calibration (matching rgb pixels with depth data)
  • hand tracking (in 2D and 3D)
  • skeleton tracking (without the 'cactus' calibration pose)

Although there is an official Microsoft driver for the Kinect, it's for Windows only (no surprize there), so I've used the opensource drivers. There are plenty of wrapper libraries for various languages, but so far I've used wrapper libraries for Processing (Daniel Shiffman's OpenKinect Processing lib and SimpleOpenNI), OpenFrameworks (ofxKinect) and MaxMSP (jit.freenect.grab). Each library has it's pros and cons, but I won't go much into detail in this post.

Here's a list of the data you can get from a Kinect:

  • Depth/IR/RGB pixels
  • Accelerometer (accesible with some of the libraries)
  • Audio data (currently supported by the official KinectSDK at the moment)

Plenty that can be done with the above mentioned. Currently I'm keen to learn more about manipulation the raw data rather than relying on OpenNI to see what sort of interactions can be achieved.

I tend to be gravitate around unusual(think Aphex Twin) ideas lately, hence the first image on the side, which displays how skeleton tracking and user isolation can be used to duplicate parts of the body. When displaying the bounding box, the gray forearm is the copied version.

One unusual idea might be turning people into trees. It seems the Greeks beet me to it (a few thousand years back), as the myth of Heliades also portrays this idea. The second image on the side shows a tracked figure morphing into a tree by recursively copying forearms. You can see the full video here. It's split into 3 parts: context, prototyping and final piece. I'm using SimpleOpenNI and skeleton tracking, but the unstable release of the drivers which allows for a more responsive output, as the calibration pose is not required.

See you in part 2 !

realtime forearm cloning with Kinect Heliades: man morphing into tree




Portrait photos: Joe Syntax

19.01.12 Jace Turner

January has been a very busy month for me and portraits. Over the next few weeks I will be posting my recent photo work. First up: Joe Syntax. I did these shots right next to our office at about 3:00am on the back of a looooooong night at Fabric. Mr Syntax really is an incredible producer and was a great guy to work with. He's on Med School so if you like your slightly more edgy DnB but not super aggressive he's your man. More to come from Med School artists, watch this space!

Joe 1 Joe 2 Joe 3 Joe 4




Fingering is always better at Christmas Time...

04.01.12 Alex Stanbury

After a good 14 months in development, we finally released our first iOS game, Elfinger. It actually started life as a simple experiment when I wanted to see how well a Flash app would run on an iOS device, you can see my findings here. I wanted to test the capabilities of the touch screen so made a simple game where you had to touch specific colours while avoiding others...

It actually turned out to be quite a nice way to play a game, I still haven't seen any other games on iOS devices where you have to touch multiple objects on the screen at once, although admittedly I haven't played every game out there. I was inspired to try and create a full game out of this bare bones idea, I made the decision to ditch the Flash to iOS conversion and work natively in Objective-C.

Instead of using simple coloured squares I started to use different animal images and from this the next incarnation of the game was born - Farmer Finger! The idea was that you were a farmer and you had to collect animals that had escaped from your farm into various landscapes such as hay fields and icy ponds. I collaborated with Greg who kindly provided some character illustrations for me to use, going for a basic but fun cartoon-like feel. The game slowly began to take shape, I was learning the intricacies of Objective-C whilst I went along. With no deadline to meet it was a refreshing way to work as I could take as long as I wanted...Or so I thought...It turned out I took a bit too long as eventually the name 'Farmer Finger' - which I'd registered with Apple's app submission portal - had expired, meaning we could not use that name anymore.

That was one of the many lessons learned throughout the whole process (most of them meant I hated Apple a little bit more each time, but that's another blog post). It wasn't all my fault though, I was waiting for a certain someone to provide me with graphics, which unfortunately never materialised. I guess that's the down side of not having a deadline; you are less likely to get things done as paid work takes priority. Farmer Finger was sadly no more and was placed on the back burner...

...One drunken night a few months ago the conversation somehow became focused on Farmer Finger. Tears were shed and hours were spent reminiscing about the good times. Eventually amongst the drunken ramblings a miracle happened - it was decided that this game should be resurrected, hurrah!

I had the idea that as Christmas was coming we should have some kind of Santa Claus collecting presents in snow storyline or something along those lines, my memory is understandably hazy due to the alcohol consumption. By this time we had hired Antonio Banderas look-a-like Nacho, a talented illustrator, whose task it was to provide all the visual assets for this new version of the game that wouldn't die. Luckily, Nacho had the time to actually do all the graphics I needed and a sterling job he did too. He also came up with the ingenious name, Elf + Finger = Elfinger, obviously.

I spent a couple of weeks updating the app with the new look and added some more functionality. There were the customary head banging on desk moments that every developer knows too well but eventually it was ready to be released to the public. 3 App Store submissions later (It got released with a couple of bugs that were promptly fixed) and Elfinger is now being played by at least 50 people! Yippee!

From the start we made the decision to release the game as a freebie, the main objective was to go through the whole process and gain experience so hopefully we can make more games in the future either for clients or for ourselves with some kind of revenue generating strategy in place. I think it was all worth it, do you? Why not play it and see? (if you have an iOS device that is!)

Title Screen Snow Level Farmer Finger Title Screen Farmer Finger Gameplay




Streetview Stereographic

03.01.12 Max Novakovic

I came across an online demo called Streetview Stereographic and was inspired to create a short animation that 'drives' from Farringdon station past our office and to Smithfield Market. Streetview Stereographic uses WebGL and Stereographic projection on Google Street View images to create a novel effect that looks like what you are viewing is a mini planet.

As the project uses WebGL you will need a relatively new browser such as Chrome 15+ or Firefox 4+. Some interesting results:

I've seen some videos before that use Stereographic projection and thought it was a cool effect. Unfortunately, I can't remember where I saw any of them. I see lots of potential with this; the demo is open source so it could be adapted so that you can specify a route and automatically be walked you through the route in the same style as the video.

The video was made quite crudely by taking a screenshot of each frame and then stitching them together with FFmpeg.

Stonehenge Stereographic

artraffi
2012-01-09   21:23

nice pics!! i just made a video of it in berlin--> http://www.youtube.com/watch?v=oGXYEqVFclg


orgicus
2012-01-16   20:26

That's pretty sweet ! Although not the same, reminds me of Keith Peters' Polar Distortion experiments ( http://www.bit-101.com/blog/?s=polar )...Simpsons did if first :P