Making a Scene at the National Gallery of Art
Building my own dioramas using the Index of American Design (without upsetting security)
Inspiration
During a recent hack day collaboration with the National Gallery of Art in Washington D.C., our team explored the variety of ways we could use the Gallery’s open data resources and APIs.
One part of the collection that I wanted to focus on was the Index of American Design. Conceived as “an effort to identify and preserve a national, ancestral aesthetic”, the Index of American Design comprises over 18,000 watercolour renderings of American folk and decorative arts objects from the colonial period to 1900.
Idea
The aesthetic of the collection evoked thoughts of the intricate doll’s house dioramas you often find in museums. This led me to think it could be fun to build a game that would let you create a similar scene of your own.
My plan was to develop an interactive space, a virtual doll’s house if you will, where players could create collages using cutouts of all the objects from the collection, positioning them in an empty room.
Displaying the collection in this context would hopefully encourage people to dive more deeply into the collection, discovering things that they might otherwise have missed compared to using a typical search interface.
The prototype game can be found at https://making-a-scene.vercel.app
Game mechanics
The mechanics of the game are pretty straightforward. Choose a background scene on which you want to build/decorate, then browse or search through the collection, selecting the objects that you want to place into the scene, discovering more about them as you go.
Once the objects have been added to the scene, you can resize them, and move them around. One obvious enhancement I’d like to make is to add the ability to skew images, so that you can place an object such as an artwork or wall hanging, then orient it to provide the right perspective.
Technical Implementation
The first requirement was to create cutouts of each object in the collection. One of the main reasons I thought the idea was feasible was because almost all of the watercolour renderings are well defined depictions on a plain background, making them excellent candidates for generating highly accurate cutouts.
Thanks to the range of pre-trained machine learning models that are now available, this kind of task is relatively simple these days.
Using the excellent rembg Python tool and some manual data wrangling I cutout, resized and trimmed all 18,000+ objects in just a few hours.
To build the interface, I used the tools we’re familiar with at Cogapp, Next.JS, which is as great for rapid prototyping on hack days as it is for building production apps. The search backend uses an Algolia index. (Shout out to Luke who smooshed the object, image, and related terms from the NGA open data repository into a single index).
The background image and the images of the objects themselves are all arranged as layers on an HTML canvas.
Background scenes
We need no excuse to try out cutting edge tools at Cogapp, and so generating the background scenes for the canvas was a perfect chance to use the AI image generation tool Midjourney to create flat 2D views of empty rooms. I asked Scout Burghardt (our in-house Midjourney Prompt Engineer) to help.
Initially the AI seemed to be a little obsessed with abandoned, dilapidated colonial-style mansions.
Using her prompt engineering skills, Scout managed to steer the AI away from that and towards something a bit more homely, closer to the aesthetic found in the watercolours themselves. Though it was insistent on having chandeliers. We didn’t have time to argue, so in this case, AI gets what AI wants.
Prior to the hack day, I hadn’t really used HTML canvas all that much, it’s pretty straightforward, but using it in combination with React perhaps made things a little bit more convoluted. However, I had a helping hand from the excellent Konva library which let me to make quick progress by abstracting away much of the boilerplate needed to manipulate the various elements.
I think the game mechanic really encourages you to go down a bit of rabbit hole, there’s such a huge variety of objects in the collection, so have a play with it yourself and see what you think.
Neil is Deputy Technical Director at Cogapp, a digital agency specialising in the cultural sector. Please get in touch if you’d like to hear more about our work.
If you’re interested in joining us for a hack day, there’s more information on our website.
We’re on Twitter or you can contact us via our website.
