On May 3, festivities honoring the Northeastern University Class of 2021 kicked off with a lively stroll from West Village to Matthews Arena. Adorned in their commencement regalia, graduate and undergraduate students enjoyed a variety of specially designed activities. Along the route, they witnessed murals come to life and stopped for an augmented reality (AR) photo op with President Joseph E. Aoun.

While moving paintings and a man dressed like a wizard (Aoun) sound like something out of Harry Potter, the experience was created, in large part, by two CAMD undergraduates, Ashley Knehans and Erin Mittmann.

The Project Kickoff

In February, after hearing of Erin’s previous AR work, Advancement and Alumni Relations approached her to help realize their vision for graduation. Ashley was later brought in because of her background in animation.  The pair met with Public Art Manager Thomas Vannatter to discuss his concept for two of Northeastern’s most well-recognized murals – “Joy” by Silvia López Chavez and “Ars et Scientia” by El Mac. With a storyboard in hand, Ashley and Erin got to work recording, animating, programming and testing all the elements needed to build the Processional Parade AR Experience. Even under the best circumstances, designing an AR sequence is a lot of work, doing so during a pandemic on a tight deadline, only raises the stakes.

Luckily, even though AR is still in its early days of development, proliferation and adoption, it is not new to CAMD. The same AR publishing platform, Hoverlay, used for the Processional Parade has been taught in Associate Teaching Professor David Tamés’s classes for years. The most well-recognized example of AR in action remains Pokémon Go,” which launched in 2016. In the game, players were able to see computer-generated Pokémon characters in real-life settings through their smartphone cameras.  Ashely and Erin employed similar techniques as the once wildly popular app to create their customized AR displays for the Processional Parades.

Getting a Move On

Ashley, who received a Media Arts B.F.A. with a concentration in Animation over the weekend, used Photoshop to prepare images of the two largescale works. She corrected colors and removed objects obscuring the murals. In order to animate, Ashely had to digitally “cut out” all the moving pieces. After spending hours creating individual layers of each bubble and radiating sunbeam, she was finally ready to animate. Most of the animations were created in another Adobe program After Effects; however, Ashley generated the lightning bolt for the El Mac mural in Houdini, a 3D animation software. One of the hiccups involved with putting this project together was making sure everything looked good both on her laptop and in AR. There were a lot of trial runs to see what worked color-wise and that elements moved smoothly and cohesively. Once the animations were exported, it was Erin’s portion of the relay.

Augmented Install

For many use cases, individuals can get started with Hoverlay without substantial training. After installing the program on your phone and setting up a private channel, you can use any YouTube link to create basic AR content. However, accurately sizing animations for two massive murals is a much more complicated task. Add in a glut of other variables – like pedestrian traffic, shuttle buses … pigeons – and the difficulty ratchets up. Having spent a semester working on Co-Op at Hoverlay, Erin, a junior pursuing an interdisciplinary degree in Computer Science and Media Arts, was ready to take on the challenge.

Erin had to anticipate and remediate many variables when constructing the Processional Parade AR experience. The mural “Joy,” painted on Ruggles Station is approximately 50-feet-tall and about double in width. But how big is the piece on a smartphone screen? What does it look like when you take five steps back and a tree gets in your shot? She even had to troubleshoot the weather when plugging Ashley’s animations into Hoverlay.

The software uses objects in the real world as landmarks for its digital projections. The main subject of “Joy,” a woman blowing bubbles, could act as a reference point. When the camera aligns with the figure, it prompts Hoverlay to display an animation. But, part of the mural is painted on windows, which changes color in the rain versus the sun. If the landmark looks different then the AR program will have to “guess” how to display the animation. Diligent testing and foresight helped Erin and the Hoverlay development team avoid forecast-related glitches.

Despite managing the plethora of moving parts associated with programming the two mural animations, Erin felt there was an opportunity to enhance the graduation walk even further. It was her idea to use the magic of green screen to create a computer-generated President Aoun for the event. Normally, he would shake hands with each student as they crossed the stage during Commencement. Safety and health restrictions make that impossible this year. Augmented Reality offered a hashtag-worthy solution. Students used the Hoverlay app outside of Snell Library Quad and afterHOURS in the Curry Student Center for a truly memorable shot with Northeastern’s president.

The Experience

While giving acting notes to Northeastern’s top VIP was certainly a notable experience, it’s not the biggest takeaway Erin has from her work over the last few weeks. Helping share CAMD’s knowledge of AR with the university as a whole has felt pretty great too. For Ashley, preparing her own graduation celebrations was an honor and an opportunity to show her friends her work in person. She said, “I was extremely excited and proud to be working on such a high-profile event. It was really invigorating to know that my work would be seen by the other students and I wanted to make sure everything looked perfect and worked well. [The day of the processional] it was really fun to watch other students hold up their phones and see the murals come to life! It was also exciting to secretly be among the crowd. Hopefully, the tradition of animating the murals will carry on!”

Helping to create the AR experience for the Processional Parade has been a whirlwind for the duo. It has challenged them to draw open all the skills they have gathered during their time at Northeastern. In her final year at CAMD, Erin is looking forward to future AR and virtual reality projects. Ashley plans to apply for visual effects jobs as a CG generalist or VFX artist and to hopefully be working on some upcoming films, television and commercials.

Interested in learning and exploring augmented reality, 360 video and virtual reality for yourself? Check out CAMD’s Immersive Media minor.

View the video to learn more about Ashley and Erin’s experience.