New York Film Academy offers VR Workshops: Creating Human Magic

About a year ago, the New York Film Academy extended its offering to include courses for virtual reality. When I was in New York myself, I finally had the chance to discover more about it. This is an interview with Caitlin Burns and Jonathan Whittaker of the New York Film Academy dealing with storytelling in VR – as well as the many other things they can teach us about VR.

The New York Film Academy

The NYFA is a private film academy that boasts outposts all over the world. The academy offers a plethora of programs ranging from short workshops to complete master’s programs. This variety extends through virtually any genre and discipline: direction, camera, documentary filmmaking, journalism, 3D modeling, acting, game design, screenwriting… Virtual reality joined the long list in the spring of 2017.

The academy distinguishes itself by promoting especially practical teaching methods while always keeping an eye out on one unifying factor: the story. As you can imagine, my interest was promptly piqued.

I dropped by during the academy’s open house, where I got to know Jonathan Whittaker, chair of the New York Film Academy’s short term programs. What he told me about the VR workshops was so fascinating that I felt compelled to ask him for an interview with himself and Caitlin Burns, the lead instructor for 360-degree filmmaking and VR gaming.

A huge thank you to both for taking the time to answer my barrage of questions.

Why Virtual Reality?

VR Geschichten: To begin with, how did both of you come to find virtual reality? Or should I say, did virtual reality find you?

Jonathan: Well, I think the latter is more true; VR finding me. The president of the of the New York Film Academy was very interested in starting the program. So he came to me and asked me: Would you help develop a virtual reality program? And I had been interested in it but I really hadn’t gone beyond the point of interest. And so, obviously, I said yes. One of my very first classes actually was with Caitlin and the company with whom she was collaborating at the time, Datavized. So I went down to the Made in New York Media Center in Dumbo, and I was really turned on to the possibilities of what VR can be. They kind of piqued my interest in a sense. I thought, “My gosh, here we go.” As a filmmaker I’ve spent my entire life trying to put everything into the frame, trying to control the audience’s attention. And now, that has vanished. To me that was like the big “aha” moment —  it has been exploration ever since.

Caitlin: For me, my background comes in helping really big story worlds and big narrative businesses figure out how to tell stories in different ways on different platforms over time. Doing that I’ve had the opportunity to work on a wide spectrum of different platforms: games, film, advertising. Most recently, I have been working with a commercial space tourism start up. All of these companies are trying to connect to people in new ways, using the tools that are the best fit for it. So over time, obviously, people started talking to me about virtual reality. It was in the sphere and I spent about twelve months at a virtual reality startup playing with data visualization. That kind of gave me the buck.

VR Geschichten: What do you love most about virtual reality? What captivates you?

Jonathan: When developing the program, I just relied on the generosity and the expertise of people like Caitlin. As I went around to different production houses, different hardware and software creators, everyone seemed to really embrace this open source mentality. There were no trade secrets. Everyone was, “Hey, I know how to do this. This is how you do that. You guys should try this, too.” That was so refreshing — especially coming from filmmaking. There, when someone knows how to do something that others don’t, the last thing they do is share it, right? That endeared me to the VR market, to the medium.

Caitlin: Personally, I love the idea that we can take someone inside of a story world. I’m fascinated by what this means — in terms of the creative outcomes great storytellers might produce. And second, in terms of what we’re seeing from the use of VR in psychiatry. The research from its use in neurology shows incredibly exciting possibilities for what it means to tell someone a great story.

But I’m even more excited to see what happens when it stops being “film in VR” or “game in VR” — and starts being virtual reality in virtual reality. That can only come when we really have enough people who understand the principles of what it means to work with an audience inside of that world; and who have enough knowledge of how to work with people of different expertise on teams to be able to push everyone into a new sphere of understanding.

VR Geschichten: What do you mean by “make virtual reality in virtual reality”?

Caitlin: This is where I think everyone is looking ahead. What is it when there’s enough of it that is substantially fascinating and unique, so that we can call it an experience type unto itself rather than a VR film or VR game? We’ll still use those names to talk about passivity or interactivity or active player engagement. But I’m really excited to see what people do with it when it’s understood and engaged with for ten, twenty years.

Storytelling in VR: On Interactivity, World Building, and the Big Why

VR Geschichten: Lately, I’ve been thinking a lot about interactivity and non-linear storytelling or multi-narratives. However, these are not necessarily the same things. Interactivity doesn’t have to lead to non-linear stories, which is something some people get confused. Do you discuss complex questions like this with your students?

Caitlin: We do. On the first day we start talking about the principles of passive and active interactivity. While there is a big spectrum, film tends to be more passive. However, even though it’s sort of acting at you, you’re still having emotional responses. The audience is engaging, but the number of times there’s an active exchange is much smaller. At this point, it is so well understood and so subtle you don’t even notice yourself interacting. So we go back even further and start with principles of ritual performance and how we develop the schema for an experience from start to finish. And then, we start talking about what goes inside the specific piece of creative work.

We have a whole understanding for film that is intuitive because its audiences have been doing it for decades. For VR, we as creators have to understand every interaction and every point around our work step by step; to understand how we can frame a scenario for our audience that’s comfortable, exciting, engaging. But also, to help them understand the path into the hardware and software of the work, whether that’s in live exhibition or it’s in our home versions where many more people have the potential to see your work.

VR Geschichten: What is the first and most important rule for storytelling in VR that you teach?

Caitlin: Well, when we’re defining the story for the sphere, for a 360 framework, the first thing that is critical to understand is: What we are doing is telling a story by simulating a much larger sensory experience. When we’re telling a story in film we’re simulating the act of observation, we’re simulating human sight. When we’re going into a virtual reality framework, whether it’s a game framework from an engine or it’s a 360 degree film, we have to remember that we’re simulating the much wider group of sensory points that humans have. So the first thing we want to do is step back and think almost theatrically: How am I engaging with this other person? How do we create an experience that’s more similar to how a person would experience the real world?

Next, it comes down to: what is the purpose of your story? What are you trying to achieve at its broadest scale? Whether that’s to convey a memory, to inspire someone to do something or to show someone something that they haven’t seen before. But why that thing — and why should it be told in this way? When you have that goal, that understanding of what you’re trying to do, it really helps to reflect your own creative process. Next, you can talk about the techniques of scenography, of staging, of points of interest in the sphere. But if you understand your goal with the audience member first, it helps you define: is this element of a scene or this moment helping? Or is it cool but unnecessary?

Work by a former NYFA student: This fascinating and award-winning 360 degree music video “Praying from Afar” by Melody Liu is about love and anger.

VR Geschichten: This has been a big discussion in transmedia. It has been posited that for VR, world building overshadows storytelling. I am unsure yet if I agree. What is your take on this?

Caitlin: World building is critically important to understand the how of what you’re going to do. But for each individual piece, each production component of an art or commercial project: if you don’t have that purpose, the why you’re doing this thing, you can have three hundred pages of backstory that’s never going to hit the audience in an emotional or real way. It’s fantastic that we can create experiences and scenes with rich compelling detail where every part of it has meaning to you as the artist. But if that meaning is not conveyed successfully to the audience, is that appropriate?

There is a lot of work that’s being done that is very scenic or experiential. But is it hitting people with that empathy machine promise? And I think in part that’s because it’s not defining a purpose for itself, before it’s going into that work. It’s wonderful to explore creatively with an amazing artistic team, but that isn’t necessarily satisfying to the audience on the other side. We are humans, we’re great in understanding stories, we are great in putting patterns together. But that intent for the creator to connect with the audience — we feel it when it’s there. You know it’s happening when an artist has taken the time to really consider why they’re playing with you in this way. Why they are setting up this experience to take you somewhere.

VR Geschichten: Speaking of each story’s purpose — this doesn’t mean you have to give your audience a role. For example, stories by Penrose Studios put a lot of thought into how the audience may interact — even though the stories themselves are not interactive in a direct sense.

Caitlin: This is one of the sessions I’m teaching today. What is the perspective of the audience member? And that can be as an observer, you can cast your audience as the camera. You can also insert them as a character and challenge their perspective by creating forced passivity. We get to think about all of these elements of casting — in part in the way we would think about casting a character or an actor, but also in the same way we might think about presenting a game experience. We play with first person and third person, observers, participants’ characterization. But it all comes back to directorial choices. In each scene that you’re piecing together, you should have a sense of who the audience is in that space. Because you’ve invited them in and you need to be a good host. Not just technically but in terms of the experience of the story you want to convey.

You should know if they are a fly on the wall, if they’re going to be spoken to as a guided character like in Clouds over Sidra, if you’re going to characterize them and then put them into a deeply unsettling perspective like in Rose Troche’s Perspective series. Because the audience can feel it when you’ve made that choice — even if they don’t know what the choice was.

The Future of Storytelling in XR

VR Geschichten: This year’s Tribeca Film Festival blew me away with Jack. More than being a film, it was a one-to-one theatrical installation. Is that one of the purposes you are talking of? And what, in general, do you think of merging different artistic disciplines?

Caitlin: I actually started university as a theatrical production designer. I studied environmental design for theater. So for me, I love working with theatrical artists. But also: theater is where you are closest to your real audience. It’s a natural fit between immersive content that is digital and simulation to work with these new tools, because the methodology is how you work with audience members. They are right there. Pieces like Draw Me Close that experiment with merging physical storytelling and virtual storytelling are amazing and deeply moving. But you also see a lot of them coming from that lyric experience of: what is the purpose of taking this audience member through this story in this way?

Actors spend so much time going through intention and the work that is never seen in their performance to build it up to be rich. That is critical to the sort of world building that creates an emotional connection — but also requires a little bit more of that thinking from everyone in the team.

VR Geschichten: In your opinion, what would be the perfect XR experience in, say, 5 years from now? What would be the perfect story?

Caitlin: We will talk about virtual reality, but also augmented reality, mixed reality, XR crossed reality — all these involve mixing the real world with new technologies in different ways. I’ve always wanted to be able to have interactive frameworks that would allow a mix of all of those. I think for me, the great joy lies in watching new stories and new experience types emerge. Because to me this is not just practice and production. It’s human magic. Just because we know how it works doesn’t mean it’s not magical.

Ultimately, I’m looking forward to a time when it’s not about the excitement of the technology but rather what it holds inside. Because the real threshold for its success will be when you’re talking about “this story,” and not “this story in virtual reality.” I know that’s a little vague. But the story I’m looking forward to seeing is what someone achieves with it – a story where you forget that it’s virtual reality.

Jonathan: For me I don’t know if it’s five years off or less… But the reason why I loved film and wanted to work in film was that it transports you to a place where you’ve never been and you can picture yourself being there, feeling like a voyeur. For VR, it’s true in an even deeper sense than for film: full occlusion; being in this story and then being able to make choices that would then dictate the outcome of the story. Filmmaking and choose your own adventure — those are the two things I grew up with and I loved. And if I can merge both of those… What are we learning as human beings about ourselves, about stories, about choices we make and the outcomes as results of it? I hope we’ll be able to use the great power of it all.

I am also very fascinated by the idea of perception. Perception is reality. And so I’m still waiting for a VR version of Rashomon. Where we’re seeing through people’s eyes and just seeing how an event changes solely based on who you are and where you’re sitting. Your own little view. And I think once we can unlock that, hopefully, that will make us kinder to each other and more understanding as human beings.

Another great student work: Jasmine Omidfar’s amusing 360 degree short film “The Bad Date” plays with different perspectives.

The VR Programs of the New York Film Academy

VR Geschichten: What VR programs do you offer?

Caitlin: Our two course offerings are: first, narrative virtual reality. That is focused on cinematography and camerawork. And second, our game based virtual reality program that is building in Unity and other game engines for VR. Both are eight-week programs and employ project-based methodologies. In both, we provide an overview of the workflows and lead the people through specific projects step-by-step. One of the great things about this program is that it is so condensed. So what we ask of the students is very intense. What we’ve seen is that it builds skills really quickly.

VR Geschichten: Let’s talk about the narrative program. What projects do the students work on specifically?

Caitlin: It’s four projects in total. One is based on camera work proficiency. One is spatialized sound, storyboarding and pre-production skills. The third is interior location sets and practice. The final one is an independent piece that the students direct and that employs all of the skills and the workflows they’ve acquired to date. The real goal for these is to give an overview of the process and what goes into the teams to build these pieces.

VR Geschichten: Can you tell me more about the gaming program’s curriculum?

Caitlin: So far, we’ve been able to run fewer of the game programs than the cinematic VR programs. With the gaming program, there is a much larger learning curve for someone who’s coming in and who’s never touched a game engine before. We actually focused on two formal projects and then a very intensive game engine oriented recap.

In both programs we really focused on implementation of creative work, narrative structuring, and — probably most critical for virtual reality — human centered design. I mean, with virtual reality it is  there quite literally a human at the center of the experience. So we do frame up everything in an immersive design curriculum that’s focusing on sensory design but also cares for the individual. Nobody wants to be known for their work that gave someone a headache or made them feel nauseous.

Prerequisites And Networking

VR Geschichten: That seems pretty intense. What skillsets do (future) students need?

Caitlin: We recommend some knowledge of game engines before you enter into the game course. But we’re not testing for it. If you start in our VR game design course, you will be able to create something by the end. If someone has professional experience with Unity or Unreal, their outcome at the end of eight weeks will be different to someone without pre-existing knowledge. But you’ll both be able to learn something.

We don’t require specific prerequisites for the narrative course. That means that the students that have been drawn to this program are both geographically and professionally quite diverse. It’s been fantastic in a lot of ways. Learning to work with teams with different compositions is part of what we want to build in terms of team intelligence, in terms of learning how to run a set, how you work in practice in a new creative discipline. I find the best teams that I’ve worked with professionally tend to come with a mixture of different expertise and backgrounds.

VR Geschichten: You are also introducing guest lectures to the classes. Is this where students can get a foot in the door of the business world?

Jonathan: Yes, that’s it. The programs are designed for networking and making professional contacts – and, of course, to see what else is going on out there in the world. For example we had Samantha Quick come in. At the time, she was running the 360-degree department of the New York Times. Chris Bobotis, who was the lead engineer at Mettle, came in and gave everyone a demo. We bring in individuals who are either hardware designers, software designers, creators, producers or venture capital people. So we tried to just expand our networks — see more of what’s going on out there, things we don’t carry in-house. We’ve also been able to do studio and program visits at production companies here in New York City.

VR Geschichten: One last question: what have you learned from teaching VR?

Caitlin: One of the things that I love as an instructor is that it really helps me challenge my preconceived notions of what is and isn’t going to work. Watching the students experiment, seeing them get out there and supporting and challenging them in that creative process — and seeing impressive implementations of things that I was been skeptical about at first. One of the reasons I believe in this course so much is that we need to have a boot camp for immersive design. We need  people to be able to rapidly scale up to understand the challenges of virtual reality. Whether that’s coming from 360-degree filmmaking origins or a game-engine-based experiential starting point, a lot of the challenges in conveying your message to the audience are universal. And so the more opportunities we get to play with it, the faster we learn from doing.

 

Further information on the New York Film Academy’s VR programs

Those wishing to learn more about the programs can take a look here: Narrative VR and Game Design VR.

Both courses are eight-week full-time programs with bi-monthly entry dates. They are currently only available in New York. The programs cost around 6700 USD (approx. 5700 EUR) including equipment. Classes are kept small, so far they’ve been ranging between 3 and 8 persons. Virtual reality is already being integrated into many other programs of the NYFA (e.g. Filmmaking).

Jonathan Whittaker confided to me that the New York Film Academy could imagine the establishment of further programs revolving exclusively around VR: short crash-courses for example, an extra-occupational course spanning a couple of months, or even a yearlong conservatory program.

 

About Jonathan Whittaker

Jonathan Whittaker looks back at over 15 years of experience in the film industry, having produced, directed, edited and lensed short films, music videos, live concerts, commercials, TV shows, feature docs and 3D. He is resident professor, chair of short-term programs and co-chair of virtual reality at the New York Film Academy, and founding member of ManInHat, a New York based production company.

About Caitlin Burns

Caitlin Burns has developed story worlds and multiplatform strategies for franchises ranging from global blockbuster films, television shows, AAA console games to virtual and augmented reality. Past projects include Pirates of the Caribbean, Tron: Legacy, Avatar and Halo. She currently is Vice Chair of the Producers Guild of America’s New Media Council and lead instructor for VR at the New York Film Academy.