Aging. Bacteria. Digestion. Gravity. Supply chains. Solar storms. And the cherry on top: covid-19.
Many things that shape our everyday lives are invisible to the naked eye. They are simply too small, too big, too far away or too abstract to comphrehend.
Yet, as humanity grapples with overwhelming global phenomena such as climate change and the lasting impacts of the coronavirus pandemic, finding creative ways to boost public scientific literacy has never been more important. Finding ways to see, touch, and experience inaccessible things can help us find solutions to our most pressing problems and feel truly connected to ourselves, each other, and the planet we call home. But who has time to seek out dense information on these topics?
Immersive entertainment, extended reality (XR), and mobile apps and games, on the other hand, are only increasing in popularity. It seems we have never been more eager to dive into the fantasy worlds built for us by storytellers and developers. Gamification, design, novelty and community reward motivate us to spend hours in front of Netflix, video games, and social media.
What if we could harness the transfixing power of entertainment — specifically emerging interactive technologies — to reveal the unseen scientific world around us?
For my capstone master's project in the Media Innovation program at the Northeastern School of Journalism, I created a proof-of-concept for an Augmented Reality (AR) science magazine. Specifically, this print/AR hybrid experience shows how the immune system works in the context of Covid-19. I aimed to make it fun — something one might choose to engage with after a busy day at work, out of curiosity rather than obligation.
The project uses accurately-rendered 3D models of cells, bones, proteins, viruses, and bacteria, combined with basic user interface elements such as buttons, switches, and toggles, to allow supplementary content to appear in space above the printed page using AR. Using a companion app, it creates an immersive experience that educates audiences about the human immune response and how vaccines can help make it more effective.
What follows is a look behind the scenes at the creation of Project: Antibody. Jump to the end to try it yourself.
Creation Story
The first step was to figure out what could be created with the resources and time I had. With the help of Prof. Bolor Amgalan and the Experience Design department at Northeastern University, I completed some basic tutorials on setting up Unity and Vuforia. Unity is a game development engine which is relatively easy to learn (and is free for students), and Vuforia is a software development kit for augmented reality applications that works with Unity.
At the time I created this project, the Unity team was in the process of creating their own augmented reality project template and pipeline. If you're thinking of starting a similar project in 2022 and beyond, I'd recommend checking out Unity's documentation and other options to make sure Vuforia is still the best option.
Proof of Concept
I began by following video tutorials and reading up on the documentation to learn about how image targets work in Vuforia with Unity. Image targets use a camera to recognize features of specific images like a mural, blueprint, or advertisement and render 3D content on top of it.
After a quick debugging session, I got a simple image target working using my computer's webcam.
I also made a few mock-ups for different formats for the paper component. I considered a tri-fold design and a DIY comic-book style origami project before deciding that one novel interaction mechanism was probably enough!
I also played around with several narrative ideas. Some of these, like the one below, were more game-based and probably made more sense as a standalone mobile app.
The winning storyboard
Which story elements would work best in print vs. being rendered through the app's UI or camera content? Reading text in 3D is cumbersome, especially on a small screen. That meant the bulk of the story would have to be printed in the magazine. The 3D image would serve as a window of sorts — X-raying the illustrations printed in the magazine and displaying augmented content. Ultimately, I decided to structure the story around questions. The working storyboard outlined the title, 2D image idea, and 3D models to be spotlighted on each spread.
Voice and Tone
For this story to be accessible to non-scientists and vaccine skeptics alike, it needed to have a fun and fantastical tone, leaving audiences in awe of the complexity and beauty of the processes going on inside their own bodies. When it came time to discuss vaccines and mRNA vaccines, I wanted to introduce the topic with a more neutral and respectful tone than most current vaccine-focused reporting. It's my belief that what people put into their bodies is their own choice; they should simply be presented with accurate, concise information that helps them make that decision for themselves. The interactive nature of the story, where audiences help bring the elements to life through interaction mechanisms such as buttons and toggles, helped reinforce this storytelling style. The game-like style gives readers a more active role in their content consumption, advancing the information they receive at their own pace by making active choices.
2D Layouts
In Adobe InDesign, I created basic layouts for each page of the magazine. I saved working versions of these images to use as Vuforia image targets as the project progressed simultaneously in Unity.
I tried a few different versions of the layouts to minimize or maximize the hand-off between phone and magazine. To help readers become comfortable with navigating a novel storytelling mechanism, I established a simple visual rhythm: a page of plain text facing an interactive image. This is in accordance with a theory called cognitive load, which states that a person can only hold so much novel information in their working memory at one time.
3D Models
The 3D models were sourced from Sketchfab, a platform where users can upload and download 3D content with various licenses. Using the Sketchfab plugin for Unity, I found 3D models that were licensed for free use under the Creative Commons license. Some 3D models required cleaning or editing; I accomplished very basic edits using Blender. One asset in particular was worth purchasing: an anatomically accurate 3D model of SARS-COV-2, the virus that causes Covid-19.
Beta Testing
Some generous friends and family agreed to serve as testers for the initial versions. Feedback and advice included making the text more legible as well as adding more interactivity to guide the user through the story. There was also some difficulty standardizing the distance and angle of the phone relative to the printed page from one spread to the next, with participants needing to significantly alter their position (sitting to standing, e.g.) to view the digital content.
Animations and interaction
Using Unity's built-in animation tools, I created some basic move, scale, and in/out movements for the models, such as the virus replicating after merging with a cell. Finally, I added some simple buttons and toggles so that the user can trigger the animations at their own pace as they make their way through the story. At the end, the reader is directed to further vaccine information from the CDC.
Future directions
With more time and a story budget, this project could benefit from:
- Original 3D art using a uniform style guide
- UI polish using a uniform style guide
- Standardizing asset size relative to scene/2D spread
- Rigorous fact checking by specialists in the medical field and updates to the content regarding booster shots and other types of covid-19 vaccines
- Official licensing for distribution in the Apple App Store and Android Play Store
Try it Yourself
With a little bit of elbow grease and a free evening, you can test drive the magazine "AR"-ticle experience yourself. The experience requires a beta mobile app for Android or iOS to be installed on your device.
MOBILE APP
Android Enable developer mode and USB debugging on your Android device and download the .apk here.
iOS The beta release is available for testing on iOS for a limited number of users. To request to be added to the test group and receive a download link via TestFlight, email me with the subject line "Project Antibody Beta."
PRINTABLE MAGAZINE PDF
The magazine's 2D design can be downloaded here. It's highly recommended to print it out for the full experience, but you can also point the mobile app at your tablet or computer screen and the image recognition should work.
