Colton Provias

Syncing and Editing a Virtual Orchestra

Wed, 12 Feb 2025 00:00:00 +0000

This is part 2 of a 2 part series on how a the Googler Virtual Orchestra piece Finlandia was done from an audio perspective.

File Preparation

Once the recordings were completed, we could now begin processing all of the files. The files we received were in a myriad of formats from different phones, cameras, audio recorders, etc. For the video editing team, this meant re-encoding all of the files into a singular format then providing a low-scale proxy and a screenshot to speed up editing. The audio side here was a bit easier.

A for loop was written in the shell to loop through all of the files and to run them through ffmpeg to extract the audio (if needed) and convert it to a WAV format. The tracks were also named to a format of {instrument_part}_{performer}_{take}.wav for organization purposes.

Synchronization

Every take wass then brought into the same Logic Pro X file as the ref track. Thanks to the file naming, the individual takes were quickly sorted next to the original tracks from the ref track. The first goal was to find the best takes and to combine them onto a single track per performer/part pair. Once that was done, the next step was to normalize the region gain across the entire file and to begin the process of synchronization.

Syncrhonizing Finlandia had a few challenges. As an almost 9 minute piece, there was some drift within the recording speed of different recorders. Fortunately having the original MIDI data from the ref track next to the tracks made it easy to play the age-old game of lining up the large transient to what looks to be the correct note start. The process for a rough sync took an hour or two, but the piece quickly took shape.

(I would insert examples of the piece at this stage here, but these early mixes contain some musician voices that may be better kept unreleased.)

With a rough sync done, we could finally start to hear that the ref track worked successfully. Several more hours later and most of the more moderate sync issues were resolved enough to get us to our desired “live” feel.

Mixing and Editing

Now that all of the parts were synced, the attention then changes to mixing and editing. A bus was created for each part, then a bus above that for each instrument family (strings, woodwinds, brass, percussion, …and saxophones?), and a final bus for the orchestra without ref track. The process now became quite methodical.

On a per part basis, I had to go through each musician playing that part and ensure their performance mostly aligned with the ref track. Wrong pitches were cut out, notes held too long were muted, notes that were played too early were either muted if there was already enough coverage in the section or trimmed and shifted to fit. Every track was panned to its approximate position within the orchestra to help build the stereo image.

Let me take a quick diversion here to talk about recording equipment. Some musicians used pro audio equipment to capture their sound. Their recordings had great dynamic range and caught many details of their instrument. In isolation, they sounded great. However, most musician recordings were on cell phones. Cell phones aren’t as detailed and often utilize dynamic range compression, thus making much of the recording the same loudness. You could still tell when a musician was playing forte or piano based upon their timbre. This discrepancy meant that one set of musicians now had to sound more like the other. Unfortunately for those who had the pro audio equipment, they lost the battle and got the compression and gain automation treatment.

Once the musician’s performance was edited, now it could be mixed. The tracks were balanced against one another to create a cohesive section playing a single part. For me this was a magical moment where you could hear the ensemble sound develop. Violins may sound scratchy solo, but they become lush and cohesive once at the same loudness. After all of the tracks in a part were balanced, dynamics were readded by drawing volume autmation into the bus tracks.

Once all of the parts had been mixed, we could step up the tree and mix across sections. Violin 1 inner and violin 1 outer would be balanced against the basses, cellos, and other strings. Any issues in dynamics were addressed once again with a general preference for higher-level mixing.

The process then repeated at the orchestra level to create the final mix.

Once the mix was done, there was some collaboration with the music director to fine tune the mix to their preference before finally exporting it for sync to video.

The entire project took about 3-4 months to execute but produced a project that we were proud to release.

Recordings of the full and per-section mix are available for listening.

Virtual Orchestra: Preparation and Recording

Mon, 10 Feb 2025 00:00:00 +0000

During the pandemic, I ran the Googler Virtual Orchestra project. This consisted of syncing recording and syncing dozens of musicians to create a seamless performance.

Here is how Finlandia was done from an audio perspective.

Prep

Everything begins with obtaining a copy of the score. With the score we can derive the instrumentation for the piece and start getting an idea of what will be needed. As a key tenant of the Googler Virtual Orchestra was musician inclusion, we found ourselves in need of adding additional parts. This meant adding saxophones and other instruments to a traditionally classical score.

Once all of the parts were finalized with our music director, we could move onto ref track production while the process of recruiting musicians began in parallel.

Ref Track Production

When playing in an orchestra, there are really many tempos. Each musician internalizes a tempo and then essentially tries to keep their tempo close to those around them, including that of the conductor. The conductor can try to suggest tempos to the orchestra, but it is ultimately up to the musicians.

In the virtual orchestra environment where everybody is playing solo at home, a musician can no longer seek a mutually agreed upon tempo with their fellow musicians as there are no other musicians involved. Instead a strict tempo needs to be prescribed for all musicians to orient themselves to. This is where the reference track comes in.

The reference track is a version of the piece with a loud click, markers for locating oneself in the music, and reference music for the musicians to perform to. To construct the ref track for Finlandia, every note had to be entered into Logic Pro X by hand, including all articulations and dynamics. The entire process involved close collaboration between myself and the music director to ensure that the track conveys the correct general feelings to the musicians and has more natural tempos.

One challenge with the reference track approach is that there is no visual for musicians. Not only do musicians communicate via sound, but they also communicate with each other via body language. The conductor can inject a lot of emotion into a performance with their movements. However, these movements are not as precise as a click and thus leave room for divergence in the tempo. This can be worsened by playback systems where the video and audio is tens of milliseconds out of sync, which is surprisingly common. Thus the decision, much to the dismay of musicians, was to keep the reference to purely an audio file to ensure that there was only one source of truth.

Once completed, the reference track for Finlandia was provided to musicians as both an MP3 and a WAV for them to practice to and to record to.

All generations of the Finlandia reference track throughout its preparation are available for listening.

Recording

Once the reference track is released to the orchestra, musicians were given about a month to practice to it. We set up virtual sectionals where section leads could communicate guidance and answer questions. Chat groups quickly developed among the sections to answer questions and to provide social opportunities in the WFH period.

After the month period ended, musicians were instructed to record themselves performing the piece while listening to the ref track. They could use any device to record themselves ranging from cell phones to professional audio and video recording equipment. Musicians were able to provide multiple takes and could even split the piece into separate segments, which was necessary given its length.

The recording process was by far the hardest portion for musicians. When performing in an ensemble, it’s easy to accept mistakes as a consequence of a live performance. In a recording environment, however, musicians tend to become more critical of themselves and often found themselves performing many takes, trying to get the perfect take. We reminded musicians that submitting multiple takes was acceptable, but our ultimate goal was to produce a performance that sounded more live rather than studio-made.

In my next post, I’ll discuss how the mix and edit was done with the hundreds of recordings received from musicians.

Syncing A Live Orchestra to Video

Mon, 20 Jan 2025 00:00:00 +0000

I lead the technical production for the Googler Orchestra, an orchestra comprised of Google/Alphabet musicians. This entails coordinating the technical requirements that a show main demand. Sometimes it’s just coordinating simple stage movements for a basic show, and sometimes it stretches to cover lights, cameras, audio, graphics, and other show elements.

For the 2024 annual holiday concert, our music director Nick Mazuk proposed synchronizing the orchestra to an animated film. This raises a simple question: how does one sync a live orchestra to video without detracting from the experience for the audience?

The answer is a specialized video. And a click track. And by breaking the orchestra’s habits.

Preparing the Audience Video

For those who have ever watched a movie in concert, you may have noticed that there are two videos playing in sync. The primary video, which I will refer to here as the audience video, contains the clean plate that is presented to the audience. The other is a video meant for the conductor.

For this show, we first had to acquire the performance rights and the media assets. This was rather simple…except we didn’t get the media assets until just a couple of weeks out to the concert. The clean plate asset we received required a couple of cuts for count-in buffer, alignment to the score, and show/logo graphics at start and end of the playback for a smoother transition. This was easily done in DaVinci Resolve in just a few minutes. We now had our clean plate for projection to the audience.

Preparing the Conductor Video

The conductor of the orchestra serves multiple jobs. The most basic is acting as a visual metronome. More importantly, they are a leader who injects an interpretation of the music into the orchestra and guides the orchestra through the piece. Thus to get an orchestra to sync to an animated movie, the conductor needs additional information that they must use in deciding what information the orchestra needs.

The conductor video we received provided very minimal additional information. It was the equivalent of asking for directions from San Diego to New York and being told “start by driving down this street” with almost no other information. Animated movies tend to make use of Mickey Mousing, which is a technique where the music acts as the sound effects for actions on screen. Mickey Mousing requires near frame-perfect precision. It was clear the conductor video needed an overhaul.

The first step of creating the video was to build a click track. I brought the audience video with synced music into Logic Pro and spent hours mapping every measure (and many beats) to the recording. Key points required additional care. Given that the animated film was on the older side, the amount of fluctuation in timing meant that hundreds of points needed to be created.

With the click track done, the conductor video could now be built using Video Sync 6. The audience plate was pulled in and connected to Logic Pro using MTC/MTU for syncrhonized playback. The conductor/music director provided notes on key hits throughout the piece. With this information, I was able to walk through the piece and add punches for each downbeat. A punch is traditionally a hole made in a frame that presents itself as a quick round white flash on screen. Hit points and rehearsal marks, along with the start and end of the piece, were marked with 4 beat streamers. Streamers are colored bars that move from left to right across the screen, often ending at a punch. Traditionally they were made by scratching diagonally across several feet of film prior to the hit point.

These visual overlays plus the auditory beeps and clicks help to provide the conductor with enough information to keep the piece in sync.

Syncrhonize the Watches Videos!

In venue to sync the audience and conductor video, we used Video Sync’s multi-screen playback feature. The clean plate was presented on two projects to the left and right of the orchestra, while the conductor video was on a laptop in front of the conductor. Getting video to the projectors required sending an HDMI signal over SDI, even for a relatively short distance as it was outside of HDMI’s range.

Breaking Habits

Playing in any ensemble, whether it is an orchestra or a band, is an exercise in team dynamics. The conductor merely provides a strong suggestion of a tempo or a particular style, but much of the actual interpreptation comes from subtle cues among the musicians. Your goal as a muisician is partially to build an ensemble sound, which involves blending into your surrounding peers. If a few violins start playing louder, the rest of the section may follow. If two out of three trombones start playing a note staccato, the other trombone will follow. While the conductor is a strong leader in rehearsal, the ensemble often takes the lead during the show with the conductor being there to course correct as needed.

This dynamic changes once you are performing to video. The source-of-truth on tempo is no longer an average from across the orchestra, but a very distinct point in time rendered digitally. This introduces conflict into a musician’s training as they now have to fight years of learning to sync to each other and instead strictly follow the conductor (or click) during the show. The conductor moves from the peripheral toward the center of their attention.

It’s a difficult habit to break, but the musicians were able to break it after a few rehearsals. Once we were finally in the performance space, the musicians and conductor were able to perform in sync with the video. It was a delight for the audiences, and a relief for those of us who spent the past few weeks getting it prepared.