3d sensing

ITP 2011 Spring Course – with Zachary Lieberman, Kyle McDonald

04. February 2011
Spring is coming, snow is melting – how do we preserve snowmen?
by Diana Huang, Ivana Basic, Eszter Ozsvald, Nikolas Psaroudakis and Yang Liu

First assignment of the class was to scan an object in an innovative way. Our “charming” weather conditions of late served as inspiration for our 3D scanning device.



Our setup was fairly simple. After scavenging the ITP floor, we found an abandoned fish tank, a mesh translucent cover, several lights, and a PS3 webcam. Snow was collected from the piles that had collected on street corners. Since our idea was to capture the cross-section of the snow as it melted to render the form of the snow, we placed the camera (facing up) below the fish tank. After some initial testing, we realized that we would have a few challenges:

1) Lighting. Since snow is translucent, we needed to position the light in a way that the snow would be visible to the camera. We found that shining a medium intensity light to the side of the tank was the best solution. We tried to position the light far enough away that it did not have a direct impact on the melting rate.

2) Melting rate. To speed up the melting process, we applied a heat gun to the snow. This wasn’t the best idea because not only did it increase the melting rate from the bottom, other areas of the snowman (head, etc.) were shrinking as well, taking away from the figure we wanted to scan. Later we “scanned” a pile of ice leftover from our original collection and applied the heat gun directly to the mesh.


Since the mesh was constructed from a light metal, we thought this would increase the melting rate from the bottom. This proved to be more successful and we used this information to run through the scanning algorithm.

PLEASE NOTE: At the end we ended up running our 3D “device” with a small pile of snow but since our original plan was to scan the snowmen that we constructed, we continue to refer to the final object that we scanned as the “snowman”. This is to avoid general confusion while explaining the process.


Our computer code was divided into two separate programs. The first program processes the frames and extracts the data which is then written into an XML file, while the second program reads the data from the XML file and displays it on the screen.

One of the problems with our setup was that while the snow was melting, a drop of water was falling in the container bellow and was messing up a few frames of the recorded video. We initially though that a way to avoid processing the “bad” frames was by defining some kind of time/frame interval in the video stream between accepted / to-be-processed frames.
This was not a very good idea, since the drop-rate was not constant and any interval we would choose would not get rid of all bad frames. Instead, we made our frame grabbing application interactive and the user had to press the “space bar” whenever there was a frame that he wanted to process. An automated but more difficult way to filter out the bad images would be by detecting bad frames by looking for sudden big changes between consecutive frames. Since the snowman was melting very slowly, it did not make any sense to keep too many frames so we only kept one frame between two water drop events.


For the processing of the video frames we used OpenCV as follows:
The first frame was thresholded and a binary mask was kept that would be applied to all consecutive frames. We did this, because we knew that we were only interested in the region that the snowman was originally occupying. In all consecutive frames, the snowman would shrink.

A Gaussian blur was then applied to each frame to soften most of the noise. The binary mask was applied to the frame and the resulting image was thresholded. The connected components / contour detection algorithm was used and contours that had an area within a predefined range were kept. The contour vertexes were saved into an XML file which was parsed in a different program that rendered the data. The distance between each layer could be changed interactively using the keyboard. However, by knowing the height of the initial snowman we could have calculated a separation between layers that would reflect more accurately the size of our scanned object.

more pictures