Inside Netflix’s bet on advanced video encoding

All copyrighted images used with permission of the respective copyright holders.

The Unending Quest for Perfect Pixels: Inside Netflix’s Obsessive Pursuit of Video Encoding Excellence

Anne Aaron, Netflix’s senior encoding technology director, is a woman possessed. Not by demons, mind you, but by an insatiable drive to ensure that every pixel on your screen, no matter what you’re watching, looks its absolute best. Whether it’s a glitzy awards show or a dusty detective drama, a galloping horse in a Western or a sparkly-suited character in an animated children’s show, Aaron’s eyes scan every visual element with the critical eye of an artist, but the analytical mind of an engineer.

Her obsession, you see, isn’t just about content; it’s about encoding, the complex process of making sure those pixels travel from Netflix’s servers to your screen without losing quality along the way. And in the world of streaming, where bandwidth and connection speeds are crucial, encoding is an absolute necessity.

Aaron’s journey at Netflix has been a testament to this relentless pursuit of visual clarity. Starting in 2011, she joined a company focused on video-on-demand (VOD), a domain where careful planning and meticulous execution reign supreme. In those early days, the industry standard was to provide viewers with a pre-determined range of bitrates and resolutions, automatically adjusting based on connection speeds. This "one-size-fits-all" approach, however, didn’t sit well with Aaron’s discerning eye.

"Why," she wondered, "should we dedicate the same amount of bandwidth for a visually complex action movie with explosions and car chases as something like My Little Pony?" The answer was simple – dynamic content requires more data to maintain quality. This realization led to a paradigm shift in 2015, when Netflix embarked on a massive project to re-encode its entire catalog with title-specific encoding settings. This meant tailoring the encoding process to the unique visual elements of each show or movie.

The results were staggering. Bandwidth savings averaged around 20%, a significant boon for viewers, particularly those in areas with limited broadband infrastructure. This strategy, pioneering in its time, has been adopted by industry giants like Amazon AWS, further highlighting its impact on the streaming landscape.

However, even the most groundbreaking technology has its limitations. Netflix’s per-title encoding approach, as effective as it was, couldn’t account for every nuance. A particularly sparkly scene in "Barbie Dreamhouse Adventures" exposed a glaring flaw: "There’s this guy with a very sparkly suit and a sparkly water fountain behind him," Aaron noted. The per-title settings, designed for the show’s overall visual simplicity, couldn’t effectively handle the excess sparkle, leading to a noticeable dip in quality.

To overcome this, Netflix needed to get even more granular, and in 2018, they did just that. The company switched to per-shot encoding, individually fine-tuning each segment of a movie or show. A conversation against a plain white wall would be encoded with lower bitrates compared to a fast-paced car chase.

The advent of 4K and HDR further amplified the need for this per-shot precision. "In The Crown, there’s an episode where it’s very smokey," said Aaron. "There’s a lot of pollution. Those scenes are really hard to encode," requiring significantly more data to achieve the desired visual fidelity.

This meticulous approach to video quality has become a hallmark of Netflix’s encoding strategy. To ensure its effectiveness, Aaron employs a unique system of subjective testing. Her team compiles short clips from various Netflix titles and encodes them with a range of settings. These clips are then presented to test subjects, who rate the visual imperfections on a scale from negligible to highly noticeable.

"It’s all based on ground truth, subjective testing," said Aaron. This valuable data has been used to train a machine learning model capable of analyzing video quality across Netflix’s entire library, ultimately determining the optimal encoding settings for each individual shot.

In a further commitment to pushing the boundaries of streaming technology, Netflix has also played a pivotal role in advancing video codecs. A founding member of the Alliance for Open Media, Netflix has contributed to the development of the AV1 codec, an open, royalty-free format that provides significant efficiency gains over older codecs.

The company has been steadily rolling out AV1, streaming select shows to a range of devices, and has even begun work on its successor. This next-generation codec, though still in its early phases, shows immense promise with potential for a 30% reduction in bit rate while maintaining the same quality as AV1.

Netflix’s drive for excellence has extended beyond just codecs and algorithms. In 2016, the company released "Meridian", a 12-minute 4K HDR short film designed specifically to test and train codecs and algorithms used by the industry. The film, complete with various visually challenging scenes, is available under a Creative Commons license, making it a valuable resource for independent video engineers worldwide.

However, while Netflix has dominated the VOD landscape, the move into livestreaming has presented new challenges. Their initial efforts, particularly a disastrous "Love Is Blind" reunion event, exposed a vulnerability in their otherwise impeccable streaming infrastructure.

"We’re quite early into livestreaming," admitted Aaron. The immediate focus is on stability and resilience, ensuring that the system can handle the scale of Netflix’s user base. For now, they’re relying on industry-standard encoding methods, "and then from there, we’ll optimize."

Cloud gaming, another frontier for Netflix, introduces even more complexities, requiring ultra-low latency. The company is still experimenting with different approaches, pushing the boundaries of what’s technically possible.

With live sports content, including NFL games and WWE matches, now joining Netflix’s offerings, the company is facing intense competition. The future holds even more challenges with potential expansion to 8K and immersive experiences.

But even with these hurdles, Aaron remains hopeful. "In the codec space, people were saying that 20 years ago," she said. "In spite of that, we still find areas for improvement. So, I’m hopeful."

Anne Aaron, with her unwavering obsession and relentless drive, is a testament to the constant evolution of streaming technology. While she might not be able to control every pixel, she’s constantly striving to get as close as possible to that perfect visual experience. In the ever-evolving world of streaming, that pursuit seems to know no bounds.

Source link

Emily Johnson
Emily Johnson
Emily Johnson is a tech enthusiast with over a decade of experience in the industry. She has a knack for identifying the next big thing in startups and has reviewed countless internet products. Emily's deep insights and thorough analysis make her a trusted voice in the tech news arena.