David Bowie Is – AR Exhibition, is an AR project by The David Bowie Archive, Sony Music Entertainment (Japan) Inc. and Planeta. In this post, I jotted down some of my development as Lead Developer, under Director Nick Dangerfield, Art Direction Seth Tillett, and Tech Director Dan Brewster.
The irl exhibition has a lot of 2D materials, and for browsing them in a mobile app, there is a subtle balance we want to maintain, a balance between efficiency and AR-ness. We tried several ways, shuffle, fade in/out, but ultimately it feels the most natural to see them laying or leaning on some sort of surface.
And just like how you would expect and do in real life, we made the 2D materials move with your fingertip.
The effect especially works great on small paper cutout. (Note – all videos in the post are direct screen recordings on my old iPhone6s)
And as a tabletop AR, some 2D artifacts need to be stacked in order for all to fit in, thus we need to a way to shuffle them.
After testing, we found out that, bringing the “focus” artifact to the front of the stack after being sent back, feels the most natural and intuitive.
Paper stack shuffle.
Scene Changes Mask
To pack the whole physical show into a tabletop AR mobile app, it is important to find a way to navigate the rich contents.
It includes over 400 high resolution captures of David Bowie’s costumes, sketches, handwritten lyrics, notes, music videos and original works of art are presented in striking arrangements and immersive settings, as well as dozens of never before seen items, including archival videos, drawings, photographs, and notes.
Within the show, we decided to condense each section into a digestible Diorama, and make them navigable through a map.
And within each diorama, we need to make some sub-scene changes as well to accommodate all the contents.
For this, I used a lot of invisible 3D mask.
The mask uses depth buffer to occlude and prevent the contents from being shown. It is a simple but effective graphic trick, and more can be found here.
For example in Hunger City, two masks are places on the sides of the stage, to hide the off-stage contents.
Under the guidance of Director and Art Director and with feedback from the whole development team, I build the AR scenes including: Early Influences, Space Oddity, Cultural Influences, Songwriting, Recording, Characters, Collaborations, Life On Mars, Ziggy Stardust, Hunger City, Stage and Screen, New York, and Black & White Years. Below are the in-app screen recording of some of those scenes.
Slowly starting working on a personal project “Lost and to Find”, it’s a point & click browser based exploration with chapters of Lazy, Same, Jealous, Unconfident, No Likes, and Can’t See. ( i know! it’s so cheesy ;) )
I was contacted by Gabrielle Jenks, Director of Abandon Normal Devices, in May 17′, about creating a VR experience for their biennial festival, which took place in September 17′ in Castleton, a village inside Derbyshire Peak District National Park of UK, with tons of caves! Together with Planeta, we created something crazy, unusual, and definitely push the boundaries of VR.
Inspired by the cave theme, I had an idea to transform normal people into cave dwellers. From an ordinary, clean, constraint person, to a bizarre, messy, loose wilder who acts freely! I imagine it’s the process of traversing the underground together, doing weird things together, that make the bounds between participants and shape whom they become.
So, we settled down the idea to transform participants as cave dwellers, but what weird behavior can we ask the participants to do?
Around that time, I was developing a Chewing Device for Google Cardboard for my Daily Life VR Ch. 2 Eat (see my instructables here!). Chewing Device is an analog mechanism to bring in the “chewing data” of real life, in order to reenact the chewing behavior in virtual world. Mouth apparently can do a lot of things, what if besides eating, it can also dig the ground? The image of people “chewing at the floor” together is so ridiculous that we all agree it is something worth of experiment.
Analog chewing device in action with real food!
Since we are bring in the chewing data, we need to use the most of the mouth / jaw movement. First of all, mouth is the only way and mean to travel down the ground, literally open & close mouth close to the floor, a.k.a. digging the ground.
We encourage participants to go wild and loose, so in experience participants can hear their voices in an echoey way. And the voice volume will generate gibberish, inspired by the 32 categorized symbols being used in Ice Age (source).
With mouth, participants can also eat stuff (meat), and puke stuff (hair ball, rock, etc). And when a participant opens mouth aiming at another participant, he/she will puke heart. And surprisingly that makes opening mouth as a major way to communicate! I’m so thrilled with this non-verbal communication, and really think it helps to shape the behaviors to be raw and primitive, and fun :)
Also, body contact is something we all love to encourage, so whenever participants touch others, random “twig” will be generated and planted on body where the contact point is.
Avatar – I used FinalIK library for creating VR avatar, and the result is quirky but at the same time very effective. The avatar looks like a possessed doll, and knowing in it is a real human being is a very strange and strong feeling. Throughout the journey, the participants will transform (both visually and mentally) from normal human to wilder, hence the shape and texture of the avatar will change every level down as well.
As for environments, it’s the part I probably spent the least time on because of the tight timeline. Luckily with Dan Brewster and Nick Dangerfield’s advice and encouragement, we settled on the heavy hand-drawn textured style.
We Dwell Below roughly has four parts involved: 1) Chewing Device, 2) Multiuser (networking) system in Unity, 3) Costumes, 4) Story.
For this project, I need to redesign the one used with Google Cardboard + web browser completely, for it to work with HTC Vive + Unity3D. I laser-cut the base and used leather for strong strap in the end. Circular pressure sensor (FSR) for the input.
I made the prototype with FSR sensor and Arduino, and Joseph Saavedra helped on perfecting the hardware design and making! It uses Teensy board in the end. The wonderful chin strap is made by Kelsey LaSeur.
Multi-user / Networking VR in Unity
This is really a pain in the butt lol
I’ve spent a lot of time on the Unity Networking system. The documentation improved a lot but it’s still quite limited in 2017. Websocket is so much clear and transparent compared to this!
According to the latest news, Unity is going to re-write all its way to do networking (2019), so I guess all my efforts won’t be able to reuse anymore hahaha (cry cry).
Still, I could share some thoughts on multi-player in VR— it’s really fun to be able to interact with other people both virtually and physically. It is still awkward to interact with strangers, but with people you know, it feels like going through a great trip together.
Suggested by AND Festival to have some elements of the scenography manufactured in material which was wearable like my project brain, we went a bit far :) Based on the cave theme and idea of participants having some sort of tribal ritual together, costume designer Kelsey designed 4 big coats and head-wears that are easy to take on and off. All 100% hand made!
To reduce infrared blocking for HTC Vive sensors as few as possible, we used mostly mesh fabric for the headwear.
For the 2018 Newcastle tour, we modified the helmet to make it stronger with more obvious face outlines. I used steal wire to make difference eye contours.
I am very lucky that Planeta, where I work at, was interested in co-producing the piece, or this would never happen in this short amount of time! We were all working hard until the last minutes, in the stone cottage near the installation site.
The project turned out to be a huge hit – thanks to the arrangement of AND Festival, we had a tent thus looked very secretive and seductive ;) Also the costume and weird behaviors with the Chewing Device, both make the experience not just interesting to try, but also entertaining to look at.
Sleep Together is a virtual reality group experience for people to use before going to sleep. In 60 seconds, participant (a.k.a. sleepyhead) will go through a series of breathing exercise, with other sleepyheads floating inside a big bird nest, breathing together.
I’m very lucky to make one of the 10 #veryveryshort projects, produced by NFB and ARTE, and co-commissioned by IDFA DocLab. The idea is to create something short yet sweet, because “Life is short, very very short.”
Continuing the idea of Daily Life VR – a WebVR series about life’s most basic and essential pursuits, I installed Sleep Together for the sleep aspect of life.
Extracts of Experience
And I’m also fortunate to share it with people in IDFA DocLab 2017 :)
I was invited to join the team in Summer 2016, to create the back-animation for the character interviews, which were captured with DepthKit. It’s a pleasant process to discuss with director Shirin, from her understanding of the people she interviewed and her visions, to my interpretation and visualization.
For me, it’s an interesting yet challenging task to maintain the balance of interpretation. I learned how to intrigue viewers without overshadowing the monologue, and am really happy with the result of magical feeling. I feel I might purposefully highlight the more lovely moment of each characters stories, and I hope I did a not-bad-job in the end to try to interpret their lives :)
AHHH why is it so difficult to export rigged model to Three.js?
This is a reminder post of how did I export with correct joint rotations.
[From Maya] – export selection as FBX, set Up Axis from Y to Z
[Import to Blender] – Scale up!
[Export to Three.js] – Select Mesh and export as Three.js. Geometry: also select “Bones” and “Skinning”. Type: choose “Geometry” instead of BufferGeometry. Skeletal animations: select “Rest” instead of “Pose”
[Import to Three.js] – Load with THREE.JSONLoader. Set “skinning: true” in material. Create with THREE.SkinnedMesh. And finally manipulate with THREE.SkinnedMesh’s skeleton.bones array.
(Update 3/1) It’s my honor to give Peep Rack a new name Intestine AR, granted by Stupid Hackathon.
For STUPID SHIT NO ONE NEEDS & TERRIBLE IDEAS HACKATHON 2017, I made Peep Rack, a head-mounted display for you to see the world through intestine, a.k.a. sphincter. It’s analog AR, the combination of Peep Show box and cloth drying rack. Made of wood nails, fragile wood sticks, transparency films, strings, and acrylic paints.
Because it’s transparency films, user can create whatever contents they want, and it’s sharable! Besides the sphincter, I also made a less practical one, containing different moods, scenarios, and functions.