Bounced
Bounced
Exploring the World of Immersive and Interactive Sound with Finlay Braithwaite, Kevin Konarzewski and the Class of RTA 864 2021
Strap your headphones on because we’re exploring a new dimension of immersive and interactive sound! This episode features guest hosts Finlay Braithwaite and Kevin Konarzewski as they describe what the brand new RTA 864 (Advanced Audio Immersive and Interactive Sound) course at X University has to offer along with sharing some incredible assignments created by students. We’ll listen to surround remixes, ambisonic, immersive and interactive projects. The coolest part? ALL of these projects were created at home during the COVID-19 pandemic. No studios or fancy equipment, just good old fashioned creativity and some help from tools like Reaper, ProTools and Unity game engine.
*This episode is best listened to with headphones. For the immersive and interactive assignments we strongly encourage you to watch the videos linked in our show notes to enjoy the full experience of these assignments.
Produced by the RTA School of Media at X University
To read the complete show notes click here
Visit our website to view the transcription or access hyperlinks in this description: https://bounced.buzzsprout.com/
Timestamps
0:00 What is immersive and interactive sound?
4:00 About the RTA 864 course, where it came from and what you can learn
Surround Assignments
13:20 Never Learn by Julianna Romanyk
18:33 Jungle Song by Ofer Weis
23:24 Sex Bracelet by Noah Caccamo
27:59 LOST by Emily Bak
Ambisonic Assignments
34:20 What is ambisonics?
38:03 Club Bathroom by Alexa Patino
42:10 Christmas Recital by Madi Hanaka
45:31 Pet Shop by Jacob Krywetzky
48:56 WW2 Battle of Britain by Matthew Page
52:07 RAIN by Nic Muia
55:29 Bad Dream by Ethan Horbay
Immersive Assignments
58:00 Ambisonic and Immersive experiences
1:01:05 Patrick Remy-Danziger Feature
1:05:20 Jim Carrey House of Horrors by Monica Brighton
1:07:25 Rocket Launch by Kyle Damji
1:08:15 Howl’s Moving Castle by Eden Lim
Interactive Assignments
1:10:02 Interactive and Spatial audio in Unity game engine
1:12:32 The Disappearing Door by Noah Caccamo
1:17:56 Finding Kitty by Ethan Horbay
1:21:44 Interactive Goodie by Ben Gingerich
1:21:44 Immersive Carnival by Bayli Iorio-Wilson
1:25:27 Future of RTA864
Credits
Podcast Producer and Host: Sam McNulty
Project Supervisor and Guest Host: Lori Beckstead
Artwork: Yvette Sin
Sound Design (Trailer, Opening and Sweepers): Anita Porgozelska
Opening and Closing Music: PHAS3.EXE by Drake Sanderson
Show Notes and Transcription: Yixin (Cherry) Yang
Find us on
Instagram @bouncedpodcast
Twitter @bouncedpodcast
Website: https://www.podpage.com/bounced/
Buzzsprout: https://bounced.buzzsprout.com/
Samantha McNulty 0:00
So you've chosen this episode to listen to. Great choice. But before we jump into things, you're probably asking yourself what the heck is immersive and interactive sound? Well, to put it simply, immersive sound is sound that puts us in a 360 degree space. There sound happening all around us in every direction and at varying distances. To give you an example, strap your headphones on and take a listen to this immersive piece taking place at the beach.
Interactive sounds on the other hand is a sound experience that responds to our interactions with it. For example, in a video game, when you're walking around, you can hear the grass crunching underneath your footsteps as you walk on it. You can hear a glass click when you pick it up or put it down. Or if you're equipping a weapon, you'll hear a corresponding sound effect. When you put immersive and interactive sound together, you're in a 360 degree space that you can interact with in the sound and your perception of the sonic world will respond to your movements. Intrigued? Well, you'll be hearing a lot more examples and learn all about this type of audio in this episode. So I'm gonna shut up now and I'll see you on the other side of this intro.
Anita Pogorzelska (Opening) 1:52
You're listening to bounce an RTA school media student showcase podcast.
Samantha McNulty 2:04
Welcome Welcome. Welcome back to bounced a podcast where we feature the work of amazingly talented RTA students. I'm Sam McNulty, a fourth year Media Production student and today we have a very special episode in store for you. We'll be talking about RTA 864, our new advanced audio immersive and interactive sound course here at x University. It ran for the first time this past winter semester and it went swimmingly. Despite the challenges posed by covid 19 students were able to experiment and create fully immersive and interactive audio experiences all from the comfort of their own homes. That's right, there were no studios this semester. So everything you'll hear in this episode was created in students bedrooms, bathrooms, kitchens, who knows maybe it's all the above whatever it was, it's going to blow your mind. I won't be doing this alone, here to guide us through these impressive projects are the two big brains behind this course: Media Production administrative coordinator, Finley Braithwaite,
Finlay Braithwaite 3:03
Hey!
Samantha McNulty 3:04
And production technician and contract lecturer Kevin Konarzewski
Kevin Konarzewski 3:08
Hello.
Samantha McNulty 3:09
By the end of this episode, I guarantee that you will be convinced to put RTA 864 into your course intentions if you haven't already. So without further ado, Kevin and Finley, how's it going?
Finlay Braithwaite 3:21
Things are good things are things are looking apt. Kevin and I just got our our second jabs independently, of course, but you know, I have a lot of hope as the administrative coordinator that we will be back on campus to a certain extent this coming fall. So we'll have people back in studios, which, you know, as you mentioned in your introduction, has been sorely missed this year. Kev, how you doing?
Kevin Konarzewski 3:46
I'm doing great. I'm feeling really optimistic about about what the future is, is bringing certainly compared to the the really rough year we've had. So I'm doing great. I'm really looking forward to to running this course again, particularly.
Samantha McNulty 4:00
Amazing, amazing. I think I get my second dose tomorrow actually, I don't remember what time so we'll have to look that up. But to start off, can you guys tell me a little bit about where this cast came from? And the need that it fulfills in RTA?
Finlay Braithwaite 4:13
Yeah, absolutely. We just have a class called RTA 981 which was the advanced audio class, it was a double credit, six hour course. It carried a lot of weight. We tried to kind of fit everything in there. So we had an element of, you know, complex live broadcasting audio, you know, a very large radio show. And we had a, you know, a live event with a concert and a 5.1 surround mix. And then we had a larger scale that extended the music production practices of our intermediate audio course. Anyway, suffice to say, there was a lot in there and we decided as a school to break this up into smaller, more focused, streamlined components. And so one of those was an advanced music production course, which really kind of took the mantle of continuing students are in terms of music production practices. As well, there's a live production course, which will scaffold production audio, but also, you know, live events, press conferences, etc. And then there was this course, which put a focus on two practices of immersive audio. Took the elements of 5.1 surround mixing that we used to do in advanced audio, and added a layer, an element of interactivity, kind of a required contemporary element to have our students be able to interface with the, you know, burgeoning applications of immersive and interactive audio in the game space, for example. So it really came from there, you know, a larger class broken down into these smaller, more focused engagements.
Samantha McNulty 5:59
That's fantastic. Yeah, that sounds like it would have been a lot for just one course, I can't even imagine what it would be like to take it. But I'm glad that it's now its own individual course and we have a dedicated space for that. So can you get into what students are going to specifically get out of this core skill-wise?
Finlay Braithwaite 6:17
Yeah, absolutely. So we start in a kind of a more traditional channel, configuration based space. So think of your cinematic or home theater or 5.1 7.1 surround configurations. From there, though we get into, you know, truly immersive, spherical soundscapes, we get into the world of ambisonics, where students can realize their material in a complete 360 sphere of sound. From there, though, we add the ability to imagine and realize audio experiences with six degrees of freedom. So, you know, experiences that surround you, you're immersed within them, but you can also traverse them, you can walk around in that space. And then finally, we add a layer of interactivity. So we have the students engaged with these immersive audio spaces that they've created and start to give the audience agency over different parameters, elements of the experience, they're creating events, and they're having the sound the soundtrack respond to their movements within the game space. So it's really quite exciting, going from this kind of traditional base of, of cinematic work in 5.1, all the way to contemporary practices within game engines.
Kevin Konarzewski 7:41
Yeah, I mean, I would, I will heavily echo everything that that Finley just said, particularly just the, for me, students really getting just this, this exciting new experience, from audio. Something that's probably very different from traditional audio media that they've produced up until this point. And for myself, just being involved with that with the class that was something that I really, really enjoyed, was just that this was new for me, you know, as somebody who's been an audio professional for years and years ambisonics and a lot of these newer spatial sound formats is it's brand new to me, it's something that I've kind of always been aware of on the periphery, in recent years. But this was my first opportunity as well to sort of dive into that. So yeah, I think I got as much out of this course as this as the students did. Maybe that's maybe that's selfish, but it's the truth.
Samantha McNulty 8:33
That's fantastic. I can imagine that with COVID-19, like I mentioned in my intro there, you know, the remote learning style that came out of the pandemic, you must have faced some challenges. Can you tell me how you and the students kind of adjusted to remote learning for this course?
Finlay Braithwaite 8:50
Yeah, absolutely. I mean, the course goes in all sorts of different directions, you know, as mentioning spheres of sound, surround configurations, you know, even discuss surround configurations or Atmos configurations that have 20 to 30 speakers, in this context that we had to go to the back to the base realization, right and, you know, admit that at the end of the day, we only have two ears. And this was kind of the solution, the way out in this challenging time is that, you know, we could actually translate these experiences to a certain degree, it wasn't perfect, over traditional stereo infrastructures. So we could realize, you know, three dimensional sound, and everything in between, in a stereo context, using binaural sound, using traditional stereo deliveries. And it was actually really nice Zoom is kind of a right in step with us, you know, kind of augmented and refined their audio capabilities. So just in time for our class, we were actually able to, you know, rely just on Zoom but still be able to teach these elements of three dimensional immersive and interactive sound.
Kevin Konarzewski 10:07
Yeah, I think that the transition went surprisingly well. And really just as Finley mentioned, some of the changes in the technology from Zoom and the platform and being able to transmit binaural sound, you know, I don't think this course would have been possible, certainly to run remotely, even one or two years ago, I just don't think we would have been able to do it. So the timing of the rest of the industry and technology and you know, all the rapid advancements in the quality of web conferencing applications and things like that really, really coincided nicely with the launch of this course.
Samantha McNulty 10:45
Must have been that 5g you're getting from your vaccine today.
Kevin Konarzewski 10:48
That could be a big part of.
Samantha McNulty 10:52
So I'm just gonna get right into it. You already mentioned how the course runs, and you kind of given us a pre face of what sort of assignments are in this course. But I want to go from beginning to end. Can you guys take me through your first assignment to your last?
Finlay Braithwaite 11:07
Yeah, so So we started with traditional, fixed a channel configuration surround formats. So again, think of your cinematic, or home theater, 5.1 7.1 experiences. And so to do that, we had students actually remix or reinterpret previous work. So there's a lot of other courses that students can take up before they arrive in 864. And so students have done a lot of really, really cool work in their audio post production class. Sam, you are no exception to that. And people have done a lot of really interesting narrative work in the intermediate audio course, music productions, suffice to say they have something in their portfolio that would be appropriate to essentially reimagined remix in 5.1. So we had students kind of immerse themselves, pardon the pun, in 5.1, by taking their existing work and, and translating it into that space. So we can start with an example. A great example is Juliana robotics work, she took an existing piece of music, it's a wonderful piece, and then had a lot of fun. You know, translating that into 5.1 surround sound. Now as a note for you listening at home, this would be the way that we would have listened and critiqued in class, this is a 5.1 mix. So you know, five channels left, right, center, left surround, right surround and a subwoofer. But this is translating to us through by normal, so it's spatialized it's essentially made into a 3d soundscape which is then translated through the perspective of two ears. Not perfect, they're not your two ears, the translations a little bit off, but it gave us purchase to translate these immersive experiences. So when you're listening to this, you should be able to you know discern sound across the front three speakers and then the surrounds on both sides. And we actually had some nice tweaks with the subwoofer where it's non directional, it's coming from everywhere, but it has really kind of nice low end to it as well. With that, let's get to the work. It's wonderful. Here's Julianna Romanyk's 5.1 surround assignment
Kevin Konarzewski 13:27
Very quickly, before we play it just want to add on to Finley's description of a binomial for the listeners, make sure you're listening in headphones because this is one of the things about binaural encoded sound, it just does not translate through loudspeakers. You do have to have you know the fixed headphones with the speaker drivers directly on each ear for the final translation to work the way it's intended. So, yes, please plug in your headphones if you haven't already, or connect them with Bluetooth or whatever, whatever you do with your headphones these days. So yes, without further ado, here is Julianna's piece.
Unknown Speaker 14:07
*Julianna Romanyk piece: Never Learn*
Samantha McNulty 17:01
That was so good. I'm sorry.
Finlay Braithwaite 17:06
She's a rock star she's fantastic, and she just recently graduated and so I wish her all the best in her future endeavors I'm sure our paths will cross again in the future. But the song was actually featured on CBC last week. So again, a true true Rockstar, Julianna Romanyk fantastic work here and you can hear the voices the synths you know, all the instrumentation surround you as the listener in this 5.1 translation of our work.
Kevin Konarzewski 17:34
Yeah, I think this is a really effective example of surround mixing for sure. It's, you know, there's enough fun little tricks that play with the spatial sound and the surround channels. Without it being gimmicky, you know, it's still very tastefully done and suits the song in a very nice way while it's sort of adding another dimension. So yeah.
Finlay Braithwaite 17:52
It's playful, it's experimental and it translates really, really well in the surround space but also still is you know, works by binaurally normally over our headsets really, really nice work. And the challenging thing about you know, having me picking these examples is Juliana did fantastic work throughout the semester we're trying to be as representative as we can throughout the suffice to say I was blown away by the quality of the assignments overall. So yeah, Juliana did really really great work throughout the semester, but this one in particular surround work just you know, I thought we should play it first as the first example of a wonderful year of assignments. Next we have Ofer Weis another person that excels all year, and he comes at us with something that he did for the sound synthesis class. This is called Jungle Song. When I first heard it in teaching RTA 972 sound synthesis I was blown away it already has a real kind of immersive spatial feel to it. So this is a real this was a great choice to essentially remix reimagine in in 5.1.
Kevin Konarzewski 19:05
Here it is, enjoy and listen, listen for the toucan.
Unknown Speaker 19:07
*Ofer Weis piece: Jungle Song*
Samantha McNulty 19:08
The thing that sticks out for me is I'm just amazed by how wide the space really is in this composition, it's just super expansive. And I'm really amazed by what he could create in that surround sound and how it translate to just your two pairs of headphones or your two ears, I should say,
Kevin Konarzewski 23:06
Yeah, I just really love the aesthetic of this too, because, you know, it's presented and with this, the surround format, and it's presented very much as sort of an ambient type piece, but there's kind of like just a pretty little pop song kind of kind of hiding out in the in the jungle there. So really, really nice stuff all around.
Finlay Braithwaite 23:24
So the next piece is hilarious. And you know, just as with all these examples, Noah did all sorts of wonderful work. I think Noah, this is Noah Caccamo I am hoping I pronounced his name correctly. But anyway, Noah did fantastic work and he wins the prize for the scariest moments in the class in his interactive work, which I think we'll get to later. But this was you know, he added an element of humor and and use the 5.1 surround space to kind of introduce a diegetic story to this piece of music. So one thing we considered with the the pandemic year COVID years is that even those students have this wonderful work in their portfolios that they can up next into surround. Technically, that might be a challenge because they couldn't get into the studios they couldn't get into the facilities where they originally authored this material to get the material they needed to to remix it. So we provided a kind of a base set of of materials for people that didn't have an option of their own. So so this is a song called Sex Bracelet. Kev, Do you remember whose Sex Bracelet is by?
Kevin Konarzewski 24:40
Yeah, this is the band partner.
Finlay Braithwaite 24:41
This is a band partner so one of our production assistants Max Cotter was involved in this producing this session and got us the stems for this. And Noah picked up on the fact that like hey, there's no bass guitar in the song until kind of the climax near the end, and you'll hear what he did to address that problem in 5.1. Here's Noah Caccamo's interpretation of sex bracelet by partner.
Unknown Speaker 25:37
*Noah Caccamo: Sex Bracelet*
Finlay Braithwaite 26:37
Yeah, so if you didn't guess already know, his title for his interpretation of sex bracelet was sex bracelet. But the bassist is locked out and frantically trying to get on stage in time, and he just makes it or they just make it just in the nick of time to add their base to the track. But yeah, it really kind of fun extra diegetic element that he added into the song through the power of 5.1. So we've had a lot of fun in in 5.1, there's a lot of creative work that's done here. But a big part of 5.1 is, you know, if you want to work in the traditional cinematic or television space is conforming to the rules of those spaces. So if you think of film and television, everything's anchored through the screen, right, all of the diegetic action dialogue happens in the screen. And so largely in a 5.1 context, all of the dialogue and action in that diegetic space would come out of the center speakers, yes, you know, reverb and some extra width comes in to the left and right, and extreme width and reverb and movement comes out of the surround speakers, but really everything is anchored down the middle. So with the previous examples, you heard things happening all around you is really, you know, creative exploration of the 5.1 space. But our next example, Emily Bak, took her work from RCA 971 the audio post Production and Sound Design course and translated it into 5.1. So this much more kind of conforms to those rules of a very kind of front wall heavy center diegetic space in in her interpretation of this mix. So she took a scene from LOST this will be a little bit this will be our first theater of the mind. exercise for you listening at home. But there will be no visual. So just close your eyes and imagine you're in the world of lost. So here is Emily Bak's interpretation of a scene from LOST.
Unknown Speaker 28:52
*Emily Bak: LOST*
Finlay Braithwaite 32:28
That's just amazing. And you know the thing is, is like we said at the scene from LOST but she's not taking these listing soundtrack from LOST, for the previous class RTA 971 she recreated everything you hear in that soundtrack, you know, all the layers of background ambience, all of the detailed sound effects all of the sound design, redid all the dialogue, sourced and edited the music, you know, really, really fantastic detailed work. And then to bring it onto this stage is just such a great exploration. Again, there are rules, right in terms of, you know, mixing for cinematic or television 5.1. But there's still a lot of kind of space for creativity. Within that, especially, you know, the bigger, more dramatic moments where you might hear something fly by you or come from behind you, etc. Really, really, really cool work on display here.
Samantha McNulty 33:27
I can see if I can put the link to this video in the show notes of this episode. So people can watch it afterwards. Yeah, total, the visual context of it.
Finlay Braithwaite 33:36
Anything you'd add to that Kev?
Kevin Konarzewski 33:41
Yeah, I mean, I the thing that really stands out for me with this one, you know, the level of detail that you kind of already mentioned, but just some of the great use of space, it was almost sounds particularly for me, the moment that stands out is the when the radio kicks in, and we hear that fantastic BTO song coming through, it's just treated so perfectly that you you know, even without the visual, you get a very good sense of where that sound is coming from and what that room, you know, looks like in terms of its shape and its size and your distance from the source. So, really, really effectively placed. A lot of sounds are but that's the one moment I think that really sort of sticks out for me.
Finlay Braithwaite 34:20
So moving on from the surround assignment, students then went on to their ambisonic assignment and they were like, well, what is ambisonics. So in surround we created a soundscape that surrounded the listener with left, center, right speaker the left surround, right surround, but they're all on a single plane. Ambisonics allow us to create a full sphere of sound, where things can, not only surround the listener, but also be above and below them. So 360 degrees sphere of sound, really, really really exciting stuff. And so, we had students design a spherical world of sound using third order ambisonics. We essentially had them create an immersive environment to place the listener and so that they could hear things above them below them, etc. So we're in a really exciting place right now, going back to the surround assignment for a second. Unfortunately with the year that was people haven't been to cinemas they are not necessarily connecting with the 5.1 7.1, Dolby Atmos experiences of the world. And increasingly, we see people getting away from the larger home theater experiences, right, the concept of having a television at home theater is becoming outdated with people just gravitating towards whatever screen is available and in front of them. The exciting thing though, is that now, the next generation, the current generation of wireless earbuds, the apple airpod, pros, the newest galaxy buds, all have spatial head tracking built into them. So the really cool thing about that is that these spatial audio experiences are becoming ubiquitous. And so there's an emerging market for these spherical ambisonics experiences that can translate now over a range of devices, I'll admit, you know, a lot of, you know, example, material you'd see in terms of ambisonics is, kind of comes from a hobby space, or an academic space of "Oh, this is what it sounds like, when a truck passes by" in a full sphere of sound. That's changed this year. Now, with these new devices, you know, allowing for essentially ubiquitous immersive spatial audio experiences, we are seeing essentially a rebirth of this kind of spherical content.
Kevin Konarzewski 36:46
Yeah, I think this is a place where, you know, people have only really started to scratch the surface of what this technology and this format is capable of doing. For listeners. So I'm really excited about ambisonics in general, just because of the newness and the wide open possibilities that this format presents both from, you know, cinematic and VR and game related audio, but also things like music, you know, music has really just been sort of locked in to channel stereo for decades now. And so there's a real potential here to to really sort of expand, you know, how people would consume and listen to even just more traditional forms, like music.
Finlay Braithwaite 37:28
Yeah, when we designed the assignment, we were really looking for scene based work. Just like very very simple scenes that put your listener, your audience within that diegetic space. However, we had a lot of students come back and as is their want with with musical examples, they still had that immersive quality, but they leveraged ambisonics to create fully immersive musical experiences and I was totally fine with that. So let's get to some of the assignments. We'll start with some of the scene based assignments. Alexa Patino did this fantastic recreation of a club bathroom Now again, as per the nature of this assignment, the request the ask was just create a very very simple you know experience for your listeners. And I think we've all been there we've all you know been you know, to a club that's really loud where you know, getting some relief using the restroom. And here we are here is Alexa Patino's interpretation of that.
Unknown Speaker 39:05
*Alexa Patino: Club Bathroom*
Finlay Braithwaite 40:48
Yeah, and right away, you hear what a difference this is right? You know, it was things surrounding you in 5.1. But all on that kind of playing at head level. Here, we have things above you below you coming from all vectors in this sphere of sound. And this is just such a strong example. She's done such a great work, great job in terms of setting up this environment. All the little nuance details, the door, the footsteps, the reverberation in the space. It really makes you feel like you're transported into that nightclub restroom.
Samantha McNulty 41:24
Yeah, something nice that I thought about this assignment, I was reading the artist statement for Alexa Patinos assignment here. And she just was talking about the uglier parts of a pre COVID world. A lot of the time throughout this pandemic, we've been kind of reminiscing about the good times, like, you know, I want to go to a club when this pandemic is over, I want to go to parties and, you know, drink my brains out. And sometimes we forget those very ugly and not so pleasant parts of those experiences. So I thought it was a nice take that she instead of just doing a regular kind of club experience, she took us into the bathroom where, you know, there's some unpleasant experiences going on. But it's just a nice reminder of that the world wasn't perfect before COVID, it won't be perfect after COVID.
Finlay Braithwaite 42:10
Yeah, and this this next example kind of got me right in the feels because, you know, one of the things that was cancelled was my son, Ernests' kindergarten Christmas recital, you know, where they all get on stage and their little tiny voices, very sweetly singing in unison to a degree, some sort of holiday song. And so Madeline Hanaka, kind of stepped in with this, this recreation of that kind of, for me filled the gap a little that I felt like I was there. And yeah, with her interpretation of a children's Christmas recital, hit it, Kev.
Kevin Konarzewski 42:46
Yeah, this is a really nice example of sort of that first person point of view perspective. Listen carefully as you're as you're listening to this, and there's a movement that happens in here, but it's largely the movement of you as the listener, you know, entering and moving in the space and reacting to the sounds around you. Here it is.
Unknown Speaker 43:19
*Madeline Hanaka: Christmast Recital*
Kevin Konarzewski 44:24
Hopefully, that did it for you, Finley, I know it's not quite the same as hearing your own offspring on stage and seeing them but
Finlay Braithwaite 44:31
I'm going to do the moment.
No, it's just fantastic work and you hit the nail on the head Kev where it's like, you know, she's creating this sphere of sound but she's also offering creating your perspective within that sphere of sound. She's, you know, changing the perspective having you rotate, having look up and down. And this is a great exploration as a class, you know, for me as an instructor as well in terms of what works and what doesn't work. There's that moment there near the end where our perspective as the audience shifts to shush someone else in the crowd. And for me, you know, in our critique, we felt that it was, you know, so quick that it kind of lost us and made us feel a little bit kind of imbalanced in that. So we resolved maybe in a second pass that the perspective would maybe be more fixed and have things happening around that perspective, as opposed to, you know, guiding and leading the perspective throughout. The next piece is Jacob Krywetsky's pet shop. When I first heard this, my mind was blown. My friend Alex Kareena, a colleague of mine, always used to joke it, I think it was a joke kind of pointed towards me as a sound designer, but he always wanted to hear clock store, you know, just a cacophony of crazy sounds. And this, this thought stuck with me when I was listening to Jacob's interpretation of a pet store, because it's wild, and it is all around you. And this is another example where as he is, you know, guiding your gaze your perspective on this space. However, in this one, I feel it's a little bit more gentle, you're, you know, not looking around really quickly, but you are slowly kind of traveling from from room to room to room, which has kind of a slightly different effect than turning around quickly in an audience. So let's take a listen to Jacob Krywetsky's pet shop, buckle up because this is a wild one.
Unknown Speaker 46:35
*Jacob Krywetsky: Pet Shop*
Samantha McNulty 48:16
I don't know about you, but that was a scary pet shop. There is a lot going on.
Kevin Konarzewski 48:20
There's a lot going on that that is a terrifying experience. From a listener perspective.
Finlay Braithwaite 48:25
You know, I that's kind of how I feel about Pet Shops though. I don't know. It's it's not far off from my in real life experience with Pet Shops. Get in, if you need to get out as soon as possible.
Samantha McNulty 48:40
How many of them have you been where there's a cow inside? That's a pretty big pet shop.
Finlay Braithwaite 48:44
It's true. I have not been to that pet shop. Yeah, just another really, really great example of the potentials of that format. So so finally in the ambisonics category, no, sorry. Finally, in the scene based ambisonics category, we go to Matthew Page who recreated the WWII Battle of Britain. Yes, that's right. He created a whole battle scene for us. And it's just spectacular. If I had one critique here, it's that the bigger moments and things exploding things happening you know, at the climax could be bigger. But overall here the attention to detail and the overall quality of the immersive experience is off the charts. So here is Matthew Page's, World War Two Battle of Britain.
Unknown Speaker 49:47
*Mathew Page: WWII Battle of Britain*
Finlay Braithwaite 51:26
Yeah such a fantastic scene and such attention to detail in terms of the sound design. But what really makes it come alive is that, again that ambisonic mix where he's placing things above us below is coming from all different vectors in the soundscape really, really exciting work. And looking forward to hearing more of his work in the future.
Kevin Konarzewski 51:45
Yeah, really good example to me to have, you know how sound can be used to propel a very distinct narrative, you know, without we don't need the pictures to be able to follow along with this story. It's all being done. sonically and very effectively.
Finlay Braithwaite 52:00
And so some students went in a musical direction we said, Hey, make a scene. And for them that meant Hey, let's make a piece of music, which I'm fine with. And so there's a lot of these wonderful kind of narrative creations, musical expressions, let's say that took advantage of the ambisonic mix space here so let's start with Nic Muia's Rain. He's done fantastic work all semester really talented musician. And he did this interpretation of a piece Rain of his in ambisonic.
Unknown Speaker 53:37
*Nic Muia: Rain*
Finlay Braithwaite 54:56
So another another really exciting development in terms of space audio is yes, there is, you know the spatial audio headsets. But on the computer side Apple Music as an example, is now supporting, immersive Atmos based music projects. So something like Nic's work here Rain already has a home. There's, you know, a new marketplace, a new stage for this type of music to take center stage, which is really exciting. We have another example of another music. Another music jam here this is Ethan Horbay's bad dream.
Unknown Speaker 55:35
*Ethan Horbay: Bad Dream*
Finlay Braithwaite 57:23
Yeah, Ethan's such a talented composer, just the, you know, the overall mood feel, and textural quality of that is fantastic. And to take it to another level with an immersive ambisonic mix just makes this something really, really special. Kevin.
Kevin Konarzewski 57:42
Yeah, I mean, the you use the word that I was gonna use to describe this, which is texture. To me, that's what this piece is really all about is this really nicely densely textured composition. And yeah, as you said that the use of space and the ambisonic sphere, it really does take this to the next level.
Finlay Braithwaite 58:00
So with that assignment with the ambisonic assignments, students were able to now explore, create, author, design soundscapes in a, you know, an immersive sphere of sound. But they're limited to three degrees of freedom. Which I touched on earlier, right, which is the ability to look up and down, left and right, and essentially roll your head with those three degrees of freedom. But what it doesn't allow for is you as the listener to traverse the space, right, you're in this sphere of sound that you cannot, you know, move from your fixed perspective at the center of that sphere. So the next assignment moved, and we had to completely switch platforms, moving from the Reaper digital audio workstation, into a game engine unity to create six degrees of freedom, immersive audio experiences. So the first assignment in that space is the immersive audio assignment. And it's still a linear experience. And it's still largely an immersive experience, as was the ambisonic assignment. But we add the ability for you as the audience, you as the listener to traverse the space and gain new perspectives on the immersive experience by essentially building it in a game engine.
Samantha McNulty 59:28
Do you want to just go into how they recorded these because we can't actually walk around and interact ourselves? They recorded it as themselves, like interacting within it, right?
Finlay Braithwaite 59:38
Yeah, so for this assignment students created a very, very simple game experience a unity scene with constituent audio elements and an audio listener. A perspective that can traverse and move around the space. With that, they deliver the the game itself to me. But for the purposes of today's demonstration, they also delivered a rendered video output. Essentially their playthrough, one possible playthrough of the experience. So they were able to essentially fix their their immersive six degrees of audio sorry six degrees of freedom experience into something resembling the three degrees of freedom ambisonic assignment that sounds complicated, but the students have a lot of options here to deliver the game experience as a game to deliver it as an ambisonic file as an output or to actually render it down to binaural and include a video. So this is another theater of the mind exercise. Well you have to close your eyes and and follow along for this one because they are so visual, visually heavy, Kevin, myself and Sam will chime in and just kind of relate what's happening on the screen for you. So let's start with with the kind of most fun flippin example. We have Patrick Remy-Danzinger. He was fantastic to have in the class throughout the year, he had some coding jobs. And so he was able to kind of pipe in, in unity. And sometimes as if you have an experience of coding sometimes say this, you know, maybe three hours of troubleshooting with just a quick, simple observation.
Kevin, load it up.
So we're following a ball rolling down a hill. And the ball is screaming and just fell off a cliff.
And now we're moving through this kind of nebulous space, it's really quite ugly, and there's just text labels everywhere. That's kind of its appeal, but I love the bits of text. So that's a clock that's ticking. And now we're gonna go into this, we're gonna go check out the air hockey,
Samantha McNulty 1:02:14
Air hockey here.
Finlay Braithwaite 1:02:16
The text label says, and now we're going to follow a text label that takes us to the Uber seeker, sorry, Uber secret hackerspace
Samantha McNulty 1:02:54
And now we moved out of that space and back towards the center. Or just looking around at each of the text spaces.
Finlay Braithwaite 1:03:06
So that was a unity scene in which Patrick created all sorts of game objects and associated sounds with them. And with the resonance, spatial audio engine, there's a lot of different spatial audio engines, but we use resonance in this class. The sounds are really easily spatialized. So they sound like they're coming from above you or below you, or in front of you, or behind you or to the left and to the right. And that's really just kind of automatic. So wherever you are, as the listener, relative to the game objects that's spatially translated to you, it's actually really easy to activate as a result. The other thing we have the students, you know, investigate and explore. And you heard some of this to a certain degree is concepts of reflection, and occlusion, right? So in Patrick's example, he has this super secret hackerspace. And so what he's done is he's put that in a separate room, blocked by walls, and so the sound is actually occluded from the listener perspective, unless the listener actually goes into that space. Then once within the space, you can hear that super seeker, super secret hacker music reflect within that space. So it's a really kind of low stakes, you know, exploration of this game engine, spatial audio space to get the students interested and where and activating these skills in a contemporary context.
Kevin Konarzewski 1:04:43
Yeah, keeping in mind too, that for the majority of the students in the class, you know, this was their first time working in this platform. You know, whether it's a you know, a 3d modeling environment, or a game engine, so very, very different to what people are you used to doing which was primarily working in Pro Tools or some other digital audio workstation So, you know, hats off to everyone for jumping in, you know so with such gusto and producing such great results in a piece of software that really was brand brand new to them all at the start of the course. So
Finlay Braithwaite 1:05:20
Yeah, the next example is out of control. It's wild. You know, this is for me when I received this this is Monica Brighton's immersive assignment. When I received this I'm like this is enough work for the entire this output is large enough in scope for the entire class. So this is Monica Brighton's work. This is her take on Slenderman. You're in what looks like a campground and there's a bunch of notes pinned to trees and you're going around with a flashlight investigating all of these different notes. Whenever you approach a note, there is a floating spatialized orb of audio that is spewing out a Jim Carrey movie clip. Here's the taste.
Samantha McNulty 1:06:10
Oh even looks like Slenderman she's like good job. So we've got a flashlight and we're walking through the woods here. I see some papers.
Finlay Braithwaite 1:06:22
And every time you get to a piece of paper, there's a floating orb. And spatially that orb is spewing out Jim Carrey movie clips.
Okay, so so next we have Kyle Damji's rocket take off. This one's really really really simple. Again, theater of the mind but you're in this chamber. And from the chamber there's a narrow hallway which leads to another chamber. And in that other chamber, there is a rocket taking off. The rocket is a shepherd stone, which if you're not familiar with the shepherd stone, they're both amazing and also just insane. Anyway, you'll hear it. This is Kyle Damji's rocket take off.
Unknown Speaker 1:07:25
*Kyle Damji: Rocket Take-off*
Kevin Konarzewski 1:07:53
Short but sweet. I really enjoyed that one. I'm a sucker for a good shepherd the Good Shepherd stone and just the cliffhanger you're we're waiting for that ship to take off. And we don't get it.
Samantha McNulty 1:08:08
We don't know what happens next.
Kevin Konarzewski 1:08:09
No, but always leave your listener wanting more. That'sa really great, great approach to produce work with.
Finlay Braithwaite 1:08:15
The next assignment was spectacular. This is Eden Lim's, She did a interpretation of from the film Howl's Moving Castle by Miyazaki Hayao. And, in her words from our artist statement in the scene, we see the first floor of Howl's home, which is adorned by a fireplace, a table chairs and a sink we can hear the fire crackling on the right, with a wooden chair creaking in front of it on the front wall is the entryway into the castle, which is accompanied by a magic dial that turns to indicate the changing of Portal locations. On the side of the door is a sink where Sophie would be washing the dishes. And right in front of the sink is a table filled with all kinds of magical tools and instruments, which we can hear charming softly
And so that was Eden Lim's Howl's Moving Castle just a fantastic example of what's possible in space. This kind of short, linear, immersive experience where the user the audience has six degrees of freedom to fully traverse this immersive soundscape. Moving on, you know, if we didn't get in deep enough with this class, you know, in terms of introducing the cohort into entirely new skills of, well, for most of them entirely new skills of working within a game engine with spatial audio controls, we took them one step further where we demanded that they create some simple layer of interactivity into their experiences to create truly interactive audio experiences. And so our demands were fairly low, we wanted them to create events. So if your character or your perspective did something in the game, it would trigger something. In the experience, we also wanted them to make a parametric connection where, you know, maybe the speed of your hero has an impact on the pitch of a sound as kind of a classic example. So students really kind of took this in stride and created all sorts of wonderful expressions using the basics of code.
Kevin Konarzewski 1:11:08
I would just add that, you know, this, this assignment for me personally was one that I really enjoyed being a part of, as well. Because this, much like the students, this was brand new to me, I've never done any sort of scripting in C sharp before, or anything in a game engine like this before. So I had a great time learning this content myself in order to, to support it for the students. I even built a game of my own. This isn't about me, of course. But I did build a little game called the Virtual Train Wreck where users get to interact with a pair of turntables and try and do beat matching by using controls on their keyboard. So anyone who was wanting to take this course you'll get to you get to play a virtual train wreck. And I'll show you how it's made and the code that went behind it with much help from Finley. But yeah, really great jumping off point for the world of code. And, you know, creating these types of user interactions, really, where there are no limits. If you can imagine it, you can make it happen. It's not always that simple. You have to go to her to come up with the code that can do it. But there really are no limits into as far as what your imagination can have you thinking of when it comes to these interactive assignments. So really, really fun moment in the course for me, and of course, seeing what the students produced as a result is really where it all came together.
Finlay Braithwaite 1:12:32
So to jump off into the assignments for this, Noah Caccamo did something called The Disappearing Door. Now we heard from Noah earlier in the podcast with is flippant, you know, the bass player was locked out, ran and played the bass. We'll had a good laugh. Complete 180 here where Noah kind of terrified us he created this really kind of like dark experience filled with existential dread.
Kevin Konarzewski 1:13:04
That's good way to describe it. It's terrifying but fantastic.
Samantha McNulty 1:13:08
I'm concerned for now there's some wrong stuff going on in that mind.
Kevin Konarzewski 1:13:14
So hopefully the audio can do this justice without the visual accompaniment but here's a brief glimpse of what this world sounds like.
I wanted to sort of describe what was happening there but I feel like I couldn't really do it justice. This one was the assignments you have to you have to experience to really get the full gist of it but it gives me goosebumps. In a very good sense.
Finlay Braithwaite 1:16:55
He's done really interesting thing here earlier in this podcast we talked about, you know the elements of reflection and occlusion. But he's also really explored and play around with elements of directionality, but also proximity, where you can only hear things if you and the game object are aligned and just the right way or you're just the right distance away. So it creates these really kind of interesting interactions between you as the listener and the objects in the scene. And you're going from room to room to room you're kind of stuck in this loop of rooms until you do the right sequence of movements and you find this other hallway. But the other hallway has a door at the far end and as you get to it the music starts to melt and the door disappears in the floor leaving you trapped in a sea of darkness. Excellent. Excellent Noah. Fantastic, fantastic work. The next assignment here is Ethan. The polar polar opposite this is Finding Kitty. So Ethan has done a really cool thing here where he created this is like this very kind of large open world where you are a large cat and you are listening and trying to listen as carefully as possible because somewhere in this island wonderworld there is a kitten your kitten that you need to go and find and rescue. Let's take a listen. And you found the kitten at the end, Kevin, what are your thoughts on this?
Kevin Konarzewski 1:21:02
This was really, really well done. I mean, I know this is a you know, primarily an audio based course. And so the emphasis really is on creating soundscapes but I just want to find out how you know visually This is a really a really well done game experience as well there's there's a whole 3d environment of this wonderful forest area with a lake and hills and trails you know, the amount of time that went into just creating this this environment for your character to interact with cannot be cannot be overstated, but really nicely densely layered in terms of the sounds. And you know, and little kitty gets found at the end which is which is the most important.
Samantha McNulty 1:21:42
Happy ending.
Finlay Braithwaite 1:21:44
The next one's really fun. This has Ben Gingerich and this is interactive goodie. It's a man running around, kind of in this virtual space, jumping on different squares. And as you jump into square a new sample plays to add to kind of an emergent song that you are listening to you're creating through your interactions. So let's take a listen to Ben Gingerich's interactive goodie.
Unknown Speaker 1:22:09
*Ben Gingerich: Interactive Goodie*
Finlay Braithwaite 1:23:00
So that was just really fun. He created this thing where you jump around, jump on these multicolored squares, turning layers on and off that create this emergent masterpiece let's call it. But next steps would be to add more samples be able to change the tempo different levels. Who knows this could be a really really cool fun thing you know, maybe like a little mini game or something but a really kind of good and strong and creative start here. So the next example long same lines really fun exploration. We're taken to a carnival remember those? Bayli Iorio-Wilson did this immersive carnival. Let's take a listen.
Unknown Speaker 1:23:42
*Bayli Iorio-Wilson: Immersive Carnival*
Finlay Braithwaite 1:25:09
So that was Bayli Iorio-Wilson's immersive carnival. Again, another really strong example of creating this immersive soundscape, but then giving the listener the audience agency, to essentially create a truly interactive and responsive experience.
Samantha McNulty 1:25:27
Yeah, that was fantastic. Thank you. If I can just bother you for one last question.
Finlay Braithwaite 1:25:31
Sure.
Samantha McNulty 1:25:32
We've already been here for almost two hours, it feels like now that we've had a taste of what this course has to offer, could you tell me a bit about what you see in the future of this course, what it can look like in a non pandemic setting?
Finlay Braithwaite 1:25:45
Yeah, we're really conflicted because so many things worked really well in this remote context, in the sense that we had everyone on headphones, and we're all able to, you know, kind of, in this nice parody, enjoy and appreciate the same experiences. One thing you know, in a traditional, let's say, 5.1, or immersive listening space, is not everyone gets to be at the center of that space, right. So we're wrestling with what you know how we interact, the lessons learned in the COVID, remote headphone context, into the in real life in person engagement. And so we're weighing our options there, maybe to keep some of it remote, or, you know, have a system where everyone can be on headphones, but in the same space moving forward. And in terms of, you know, the the next logical upgrade is to give students the ability. Provide them with some sort of access to head tracked, spatial experiences, again, everyone was working at home. And they were working largely with by normal renderings of their ambisonics of their surround of their interactive experiences. But again, they couldn't move their head around, they couldn't explore that kind of truly spatial experience. So the dream would be to have everyone on not just headphones, but spatial headphones next year, so that we can really explore those spaces in a more meaningful way,
Kevin Konarzewski 1:27:16
I certainly have a dream of being able to potentially have a facility on campus, you know, that would be an ambisonics sphere of speakers where you could be at the center of the sphere, you know, have that the three degrees of freedom to move your head around, but do it in a tangible space. For me, that is something I would absolutely love to see. I don't know if that will come to fruition. So I don't want to make any promises that we're building an ambisonic sphere, on campus, but to me, that would be a real high wishlist item to make this course, that much more exciting.
Finlay Braithwaite 1:27:52
And there's all sorts of other developments. I mean, I've mentioned really kind of stressed, you know, headphones and earbuds as being kind of one future direction of that. But my experience at the AAS conference on spatial audio, this is 2019. In York, in the United Kingdom. They're on display where all sorts of wonderful solutions for head tracking experiences through loudspeakers. So using beamforming, using advanced techniques, in head tracking, but also binaural reproduction. You're able to essentially create fully immersive, you know, spatial audio experiences over loudspeakers. We stressed at the beginning, like, make sure you're listening on headphones. Yes, in this case, that is true. But there will be a future where your speakers will be able to track you around the space and render sound perfectly for each ear, allowing you to create a spatial audio experience. That's future research. That's really exciting things to come in the future. But the future is wide open and opens itself for all sorts of wonderful potential narrative musical interactive experiences.
Samantha McNulty 1:29:05
Yeah, it certainly does. Is there anything else you'd like to say to our listeners or to each other? Any last thoughts?
Finlay Braithwaite 1:29:13
So while I'm still recording, I just want to say thank you, thank you, thank you, from the bottom of my heart to the students for for persevering through this year. It wasn't easy, and it wasn't the same as being in person but that you guys made it really special and something will always remember for all the positives that the students added to the experience. And then importantly, I want to thank from the bottom my heart Kevin Konarzewski who made this this class such a special experience, he really, you know, allowed us to try some things that could have failed that could have, you know, had alternate results, but he really stepped up and made sure that they they were as successful as possible. And I'm just gonna say he really geeked out in terms of like coding and immersive experiences like you really hit your stride here and some of the materials you came up with ping pong madness and Creek of solitude as an example, will live on for years and years and years and years. So from from the bottom of my heart, thank you. Kevin.
Kevin Konarzewski 1:30:11
Well very kind Finley But you know, like I mentioned earlier, but this class doesn't happen. This class doesn't even exist without you. So, you know your vision for for this course. I remember you talking about this course, years ago, and, and suddenly it's here, and it's awesome. And that is, and that's on you, man. I just, I'm just along for the ride.
Finlay Braithwaite 1:30:35
Okay let's get outta here you rascals. Sam, thank you so much. And we'll see you on campus next year.
Samantha McNulty 1:30:41
Yes, yes. Yes, thank you, again, Finley and Kevin for coming on the show. And for all the work you've done to make this class happen. Yeah, all I can say is I'm so excited to take this course. And I wish that I could be at Ryerson a little bit longer to see what this turns into in the future. Thanks again.
That was Kevin and Finley on this very special episode of Bounced. We've covered a lot there. And all I can say after listening is wow, these were all fantastic projects. And it blows me away that these were all created outside a studio environment. Just goes to show you that you can access this technology at home and with some creativity, you too can make something amazing funny. And if you're Noah Caccamo, something deeply unsettling and disturbing. I'm going to have some nightmares about that one. Thank you again, Finley and Kevin for coming on the show. And for all the work you've done to make this episode and class happen. I'm beyond stoked to take this class next winter. If you want to do the same, all you have to do is to have taken five there the intermediate audio production course or take 311 or the interactive storytelling course RTA 320. From there, just pop this baby into your enrollment shopping cart when the time comes this August. I believe the 2021 winter course intention adjustment period is Monday, August 23 to Friday, September 17. That's a lot of time so no excuses, get your butt into this class, learn some cool things and have an awesome semester. That's all for this episode. Thanks again guys. Until next time.
Robot Voice (Closing) 1:32:27
Thank you for your cooperation.