A Canvas Made of Pixels

A year ago over the holiday break, I created a large-scale digital “canvas” that can reproduce works of fine art – paintings, photographs, lithographs, etc. – in a way that is nearly indistinguishable from the real thing.

For the full effect, you really need to see it in person. But so you can get the idea, here’s a photograph of the frame displaying a Van Gogh:

The Problem with Digital Picture Frames

I’ve been interested in digital picture frames since 2003, when I made my first custom frame out of a cannibalized Apple iBook. The promise of digital picture frames, of course, is that they can display new stuff whenever you want – photos, paintings, whatever. But there’s a real problem with digital picture frames: they just aren’t very good.

What do I mean? To start, in commercially available digital picture frames, you’ve generally got a big black plastic bezel surrounding the display, so the picture frame looks more like a small television than a picture frame. The display itself is all wrong, too. You can see individual pixels, the viewing angle is limited, the contrast is low, and the standard aspect ratio (16×9) of modern displays is too wide for photos and fine art. And until recently, no one has made a big digital picture frame, so the largest thing you could display was a 5×7 or 8×10 photograph. Finally, because screens emit rather than reflect light, they usually appear either too light (glowing) or too dark (washed out), so they look like screens.

With this project, I set out to tackle all of these problems, and to create something so good at reproducing artwork that a viewer forgets (or never even realizes) that he or she isn’t just looking at the real thing.

Lots and Lots of Pixels

The first task was to find a large display with very high resolution, accurate color reproduction, a good contrast ratio, and a wide viewing angle.

The resolution requirement for the display was informed by how people generally look at art: you start at a bit of a distance to take in the piece, then may walk up to it and lean in to inspect details, getting as close as maybe a foot away. If you do the math, at that viewing distance, you need a pixel density of about ~285 pixels per inch to keep individual pixels below the resolving capability of the human eye at the fovea, which is about 60 pixels per degree. (I’ll skip the math and detail about angular resolution and the human eye. If you’re interested, you can read more here and here.)

Today, there’s really only one large display that approaches this kind of resolution: the 5K display on the 27” iMac, which has a pixel density of about 220 ppi and an overall resolution of 5120 by 2800 pixels. (If you’re wondering, 4K “Ultra HD” TVs aren’t even close to having enough resolution for this application. A 40-inch 4K TV is only ~110 ppi.)

Unsurprisingly, Apple also did a great job with the color reproduction and viewing angle on the iMac display. So I had my display.

Aspect Ratios and Gilded Wood

The next thing I looked at was the frame itself: the material, the dimensions, the mat, and how to make it so the whole thing could be mounted flush against a wall, just as real paintings or photographs are.

The first thing I wanted to get right was the aspect ratio of the frame, since it would determine what works of art I’d be able to display, as well as the dimensions of the frame itself. As it turns out, an aspect ratio of 16×9 (or 1.77:1) – which is the aspect ratio of the iMac display and basically every other display on the market today – is really bad for displaying fine art, which tends to be much “squarer”. For instance, the average aspect ratio of the works below is ~1.3:1. The Mona Lisa, the “least square” of the bunch, has an aspect ratio of 1.45:1, still significantly squarer than 16:9.

Of course, I wasn’t about to saw off the sides of the 5K display. Instead, I designed the frame and mat to overcrop the display, resulting in an aspect ratio of around 1.4:1 – still a bit wide, but a good tradeoff between using most of the display and accommodating most works of art without having to chop their sides off.

Next I turned to the frame material. I wanted something that looked totally unlike anything you’ve seen around a computer monitor or television – a sort of “anti black plastic”. And that led me to a handmade, gilded wood frame, with a classic profile and a good amount of patina.

With the dimensions and frame material set, my last task was to figure out how to mount the whole unit so that it would sit exactly flush with the wall. After cutting a hole in the wall, I had to figure out mechanisms for making millimeter-scale adjustments (solution: screws and wood shims) and properly ventilating the computer while sealed away in the frame (solution: discrete ventilation slats on the bottom of the frame).

Photons that Fit In

The most interesting problem to tackle was “the blue glowing screen problem”.

One of the many ways that screens give themselves away as screens is by emitting light that is “out of character” with the surrounding environment. They can be too bright or too dark relative to the things around them, and indoors, displays often seem too blue.

I solved these problems with what I call “luminance matching”. The basic idea is to sample the light falling on the frame several times a second, and then adjust the display and image parameters so that what’s displayed is “correct” given the surrounding environment.

To solve the brightness problem, I embedded a photodiode – basically a brightness sensor – into the side of the frame, directed at the wall the frame is hanging on. The frame takes luminance readings off of the photodiode via a Phidgets USB sensor interface, then automatically adjust the brightness of the display. Getting the mapping from sensor readings to display brightness was a bit tricky, and I ended up using two photodiodes – one for the wall, one for the display – to perform a calibration step, so that the luminance of the actual white wall matched the luminance of the display while displaying a photograph of the white wall.

For the color matching / white balance problem, I began with the somewhat complicated approach of using an attached USB camera, turning off its automatic white balance, and sampling frames every few seconds to estimate the color of the incoming light. This approach turned out to be overkill, since the color temperature of the light in the hallway where the frame is hanging is very predictable over the course of a day. So instead, I ended up using the awesome little tool Flux, which can adjust the white balance of a display according to the time of day.

All of this comes together to enable the frame to respond to changing lighting conditions in the same way that a real photograph or painting would.

You can see it in action here, where I cycle through several different lighting conditions, from dim daylight, to overhead halogens, to simulated direct sunlight. While filming, the exposure settings on the camera were locked, so you can get a sense for changes in lighting across the whole scene, including on an identical (real) photograph hanging opposite the digital frame.

The effect in person is really quite startling: guests in our house almost never register that the photograph or painting is being displayed is anything other than a photograph or painting.

Shy Mode

The final feature I added was what I call “shy mode”. When shy mode is enabled, the picture frame uses a discrete camera and a simple face detection system to determine whether anyone is currently looking at it. If someone is, then the frame won’t change what it’s displaying, since that would be a total giveaway.

A fun (but impractical and frustrating) variant of this feature is to have the image change as soon as the viewer looks away. So you’re looking at a painting, glance away to another room, and look back to find a new painting hanging on the wall.

For Version 2.0

Building version 1.0 exposed all sorts of things that could be better.

Of course, using a whole iMac just for its display is totally impractical. A custom display with a lightweight, Android-runnning system on a chip to drive it would be much better. The ideal display would be something like a 5:7 aspect ratio 55” 8K OLED panel. (OLED would enable a much higher contrast ratio and better viewing angles.) Wireless power or trickle charging of an onboard battery would avoid unsightly power cords or wall surgery. And a richer set of sensors for doing luminance matching and presence sensing would be nice, too.

In the meantime, I’m pretty excited about what companies like Depict and Electric Objects are doing. Their products could be better – the 16:9 aspect ratio is really off in my opinion, OLED will be important, and 4K resolution isn’t enough – but they’re off to a good start. I’m really excited to see what they and others come up with in this space.

Thanks to Mahmut at Sumner Frames in Palo Alto for helping me with the framing, to Andrew Luo for his help writing various pieces of software, and to Kelly for being okay with me cutting a big hole in our wall.