A Canvas Made of Pixels

27 December 2015 - Filed under art + projects

A year ago over the holiday break, I created a large-scale digital “canvas” that can reproduce works of fine art – paintings, photographs, lithographs, etc. – in a way that is nearly indistinguishable from the real thing.

For the full effect, you really need to see it in person. But so you can get the idea, here’s a photograph of the frame displaying a Van Gogh:



The Problem with Digital Picture Frames


I’ve been interested in digital picture frames since 2003, when I made my first custom frame out of a cannibalized Apple iBook. The promise of digital picture frames, of course, is that they can display new stuff whenever you want – photos, paintings, whatever. But there’s a real problem with digital picture frames: they just aren’t very good.

What do I mean? To start, in commercially available digital picture frames, you’ve generally got a big black plastic bezel surrounding the display, so the picture frame looks more like a small television than a picture frame. The display itself is all wrong, too. You can see individual pixels, the viewing angle is limited, the contrast is low, and the standard aspect ratio (16×9) of modern displays is too wide for photos and fine art. And until recently, no one has made a big digital picture frame, so the largest thing you could display was a 5×7 or 8×10 photograph. Finally, because screens emit rather than reflect light, they usually appear either too light (glowing) or too dark (washed out), so they look like screens.

With this project, I set out to tackle all of these problems, and to create something so good at reproducing artwork that a viewer forgets (or never even realizes) that he or she isn’t just looking at the real thing.


Lots and Lots of Pixels


The first task was to find a large display with very high resolution, accurate color reproduction, a good contrast ratio, and a wide viewing angle.

The resolution requirement for the display was informed by how people generally look at art: you start at a bit of a distance to take in the piece, then may walk up to it and lean in to inspect details, getting as close as maybe a foot away. If you do the math, at that viewing distance, you need a pixel density of about ~285 pixels per inch to keep individual pixels below the resolving capability of the human eye at the fovea, which is about 60 pixels per degree. (I’ll skip the math and detail about angular resolution and the human eye. If you’re interested, you can read more here and here.)

Today, there’s really only one large display that approaches this kind of resolution: the 5K display on the 27” iMac, which has a pixel density of about 220 ppi and an overall resolution of 5120 by 2800 pixels. (If you’re wondering, 4K “Ultra HD” TVs aren’t even close to having enough resolution for this application. A 40-inch 4K TV is only ~110 ppi.)

Unsurprisingly, Apple also did a great job with the color reproduction and viewing angle on the iMac display. So I had my display.


Aspect Ratios and Gilded Wood


The next thing I looked at was the frame itself: the material, the dimensions, the mat, and how to make it so the whole thing could be mounted flush against a wall, just as real paintings or photographs are.

The first thing I wanted to get right was the aspect ratio of the frame, since it would determine what works of art I’d be able to display, as well as the dimensions of the frame itself. As it turns out, an aspect ratio of 16×9 (or 1.77:1) – which is the aspect ratio of the iMac display and basically every other display on the market today – is really bad for displaying fine art, which tends to be much “squarer”. For instance, the average aspect ratio of the works below is ~1.3:1. The Mona Lisa, the “least square” of the bunch, has an aspect ratio of 1.45:1, still significantly squarer than 16:9.


Of course, I wasn’t about to saw off the sides of the 5K display. Instead, I designed the frame and mat to overcrop the display, resulting in an aspect ratio of around 1.4:1 – still a bit wide, but a good tradeoff between using most of the display and accommodating most works of art without having to chop their sides off.

Next I turned to the frame material. I wanted something that looked totally unlike anything you’ve seen around a computer monitor or television – a sort of “anti black plastic”. And that led me to a handmade, gilded wood frame, with a classic profile and a good amount of patina.

Gold Frame

With the dimensions and frame material set, my last task was to figure out how to mount the whole unit so that it would sit exactly flush with the wall. After cutting a hole in the wall, I had to figure out mechanisms for making millimeter-scale adjustments (solution: screws and wood shims) and properly ventilating the computer while sealed away in the frame (solution: discrete ventilation slats on the bottom of the frame).


Photons that Fit In


The most interesting problem to tackle was “the blue glowing screen problem”.

One of the many ways that screens give themselves away as screens is by emitting light that is “out of character” with the surrounding environment. They can be too bright or too dark relative to the things around them, and indoors, displays often seem too blue.

I solved these problems with what I call “luminance matching”. The basic idea is to sample the light falling on the frame several times a second, and then adjust the display and image parameters so that what’s displayed is “correct” given the surrounding environment.

To solve the brightness problem, I embedded a photodiode – basically a brightness sensor – into the side of the frame, directed at the wall the frame is hanging on. The frame takes luminance readings off of the photodiode via a Phidgets USB sensor interface, then automatically adjust the brightness of the display. Getting the mapping from sensor readings to display brightness was a bit tricky, and I ended up using two photodiodes – one for the wall, one for the display – to perform a calibration step, so that the luminance of the actual white wall matched the luminance of the display while displaying a photograph of the white wall.

For the color matching / white balance problem, I began with the somewhat complicated approach of using an attached USB camera, turning off its automatic white balance, and sampling frames every few seconds to estimate the color of the incoming light. This approach turned out to be overkill, since the color temperature of the light in the hallway where the frame is hanging is very predictable over the course of a day. So instead, I ended up using the awesome little tool Flux, which can adjust the white balance of a display according to the time of day.

All of this comes together to enable the frame to respond to changing lighting conditions in the same way that a real photograph or painting would.

You can see it in action here, where I cycle through several different lighting conditions, from dim daylight, to overhead halogens, to simulated direct sunlight. While filming, the exposure settings on the camera were locked, so you can get a sense for changes in lighting across the whole scene, including on an identical (real) photograph hanging opposite the digital frame.

The effect in person is really quite startling: guests in our house almost never register that the photograph or painting is being displayed is anything other than a photograph or painting.


Shy Mode


The final feature I added was what I call “shy mode”. When shy mode is enabled, the picture frame uses a discrete camera and a simple face detection system to determine whether anyone is currently looking at it. If someone is, then the frame won’t change what it’s displaying, since that would be a total giveaway.

A fun (but impractical and frustrating) variant of this feature is to have the image change as soon as the viewer looks away. So you’re looking at a painting, glance away to another room, and look back to find a new painting hanging on the wall.


For Version 2.0


Building version 1.0 exposed all sorts of things that could be better.

Of course, using a whole iMac just for its display is totally impractical. A custom display with a lightweight, Android-runnning system on a chip to drive it would be much better. The ideal display would be something like a 5:7 aspect ratio 55” 8K OLED panel. (OLED would enable a much higher contrast ratio and better viewing angles.) Wireless power or trickle charging of an onboard battery would avoid unsightly power cords or wall surgery. And a richer set of sensors for doing luminance matching and presence sensing would be nice, too.

In the meantime, I’m pretty excited about what companies like Depict and Electric Objects are doing. Their products could be better – the 16:9 aspect ratio is really off in my opinion, OLED will be important, and 4K resolution isn’t enough – but they’re off to a good start. I’m really excited to see what they and others come up with in this space.

Thanks to Mahmut at Sumner Frames in Palo Alto for helping me with the framing, to Andrew Luo for his help writing various pieces of software, and to Kelly for being okay with me cutting a big hole in our wall.

Comments Off  ::  Share or discuss  ::  2015-12-27  ::  Clay

Flying Cross Country in a New Horizons Space Probe

18 July 2015 - Filed under space

I wanted to get a sense for what it would be like to look out the window of an airplane flying at the speed of the New Horizons space probe which, as the probe passed Pluto, was about 36,000 mph. Flying at an altitude of 37,000 feet, here’s what that would look like, with windows from a 747 and an SR-71 Blackbird for comparison:

To put 36,000 mph in perspective, at that speed, you’d be able to fly from San Francisco to New York in about five minutes. Of course, you’d also be turned into a ball of searing hot plasma.

Comments Off  ::  Share or discuss  ::  2015-07-18  ::  Clay

Abraham Lincoln in Pennies (2013 version)

18 February 2014 - Filed under art + projects

Over the holidays, I spent a couple of days creating a portrait of Abraham Lincoln out of pennies. (1,617 of them, to be exact.)

The portrait is actually a recreation of one I did in 2005, and incorporates several learnings from the first go-around. The 2013 version here is based on a better known photo of Lincoln, and his face is more closely cropped to maximize the number of “pixels” used for defining his features. I also used a tighter penny tessellation (one with offset rows), which reduces the blank space between pennies and improves the quality of the image. The offset tessellation made placing the pennies a bit trickier, and I ended up writing a simple command line tool to compute the exact color and placement of each coin.

I just got the portrait back from the framer on Friday, and figured that President’s Day was as good a day as any to post about it. Here it is.

Here’s a close-up of Lincoln’s left eye:


Thank you to Kelly for all of her help sorting and gluing (and for being really patient with me!), to Jesse for his Objective-C coaching, and to Mahmut and Jaime at Richard Sumner Gallery in Palo Alto for doing such a wonderful job with the framing.

1 comment  ::  Share or discuss  ::  2014-02-18  ::  Clay

Underwater Abstracts

12 May 2013 - Filed under photography

Taken with the waterproof Olympus TG-1 in Hawaii.

Comments Off  ::  Share or discuss  ::  2013-05-12  ::  Clay