Enhancing the Netflix UI Experience with HDR
by Yuji Mano, Benbuck Nason, Joe Drago
Some of you have probably heard about HDR, or High Dynamic Range, which can produce a higher level of brightness, a wider range of colors, and an increased level of detail and accuracy that empowers visual storytellers to deliver more impactful experiences for people to enjoy. Netflix has been delivering HDR video for several years now, so it made sense for us to start exploring ways to provide the benefits of HDR to the imagery we showcase in the browse experience where viewers initially discover and connect with stories.
We’re excited to roll out experimental HDR images for the very first time to the Netflix app on the latest generation of game consoles. These are images that take advantage of a display’s HDR capabilities (just like HDR video) and not to be confused with HDR photos that your phone or camera might take by combining multiple exposures to generate a high-contrast scene.
We’re starting with a small test on two shows: The Innocents and Marvel’s Iron Fist. This modest “proof of concept” test will allow us to begin learning how to create and deliver HDR images globally to millions of people. Members with HDR capable TVs using Netflix on an HDR enabled PS4 Pro, Xbox One S or X will have a chance to see these experimental images as these titles get promoted on the main billboard of the app. And while it’s a small set of titles, it’s a big first step down the path of an exciting technical innovation. Reaching this initial milestone was a culmination of many months of research and countless discoveries along the way.
The Motivation
‘What’s the motivation behind HDR images?’ If you haven’t already noticed, the industry has been in the process of a technological transition from SDR (Standard Dynamic Range) to HDR over the past several years. The promise of HDR is not just in the superior picture quality it can deliver but in its ability to better preserve the creative intent of content authors.
Netflix has been at the forefront of bringing HDR video to our service, but the reality is that the still images associated with these titles that viewers see prior to playback remain left behind in SDR format.
Furthermore, consumers have been subjected to and have adjusted to a world where SDR is not shown the way it was meant to be shown. In order for HDR TVs to maximize consumer appeal, SDR content is algorithmically adjusted (or boosted) to fill up the range of colors and brightness levels closer to the capabilities of the display. While HDR video may benefit from HDR TVs as it tries to accurately represent colors and preserve the creative intent of the source content, SDR is unfortunately subject to the inverse effect of being altered to ‘appear’ more HDR.
We see HDR becoming the default format in the future. We want to deliver our still images with the same standard of picture quality we have for video by using brighter and more vibrant pixels that HDR can produce. We want to preserve creative intent for all graphics associated with our content just like we do for our video. That’s The Motivation.
The Problem
You might be thinking: ‘Netflix has been streaming HDR for a couple of years now. What’s the big deal? Just throw some HDR images into the UI and you’re done, right?’ It turns out that compositing and displaying HDR graphics along with video is harder than you’d think — there are a variety of technical hurdles yet to clear, and it will take some time for the hardware that can do this compositing to find its way into reasonably priced TVs and streaming accessories. Also, it wasn’t even initially clear to us how to create or store HDR images.
We started off by asking some of the basic high-level questions:
- What exactly is an HDR image?
- What file format do you use to store HDR images?
- What tools can you use to create and manipulate HDR images?
- What devices can support decoding and rendering of HDR images along with video?
While HDR capable TVs and PC monitors are becoming more and more standard, the truth of the matter is that the concept of an HDR image and how it exists in the realm of our HDR capable devices is an almost uncharted territory. Various standards for HDR video have emerged over the years such as Dolby Vision, HDR10, HDR10+, Hybrid Log-Gamma, etc. However, there is no such clear standard defined yet for HDR images.
The Problem is in trying to answer the above questions when there is no clearly paved path for us to follow. It requires exploration and carving out new solutions based on existing technologies and at times making wrong turns in the process. Ultimately, we arrived at our intended destination, but by no means is this meant to be a map for others to follow. We are sharing part of our journey in hopes to find others in the industry with similar stories and to help pave a solid path forward for HDR graphics and images to exist in our ecosystem.
Visualizing HDR Pixels
In order to begin answering what an HDR image is we first need to define what HDR really means in this context. HDR in UI graphics is essentially no different from HDR in video. The underlying value of HDR is in its ability to specify a wider color gamut (more saturated colors) and wider luminance range (more shades of dark and bright) in the pixels delivered. So simply put, an HDR image is an image that contains any of these pixel characteristics.
We would love at this point to be able to show you an actual HDR image on your screen to show you exactly what brighter, more saturated colors look like, but there isn’t any current technology that lets us do that (which is the basis of this whole problem).
One way to imagine this is to picture a single frame of video from our app that contains pixels outside of what its SDR equivalent source can express. To visualize these differences in expressibility, our team first started off by implementing a debug feature within the Netflix app that is able to highlight HDR pixels in real-time on the screen.
The above are screenshots with the HDR highlighter feature enabled during video playback of our HDR-packed Netflix Original — Altered Carbon. We grayscale the entire screen, then highlight each “HDR pixel” with various colors, typically with stronger colors for pixels that we consider “more HDR” (or farther away from what’s possible with SDR).
We highlight pixels based on the following conditions:
- The pixel is out of SDR color space. Colors outside of the existing Rec.709 color space used by SDR are considered to be in a wide color gamut (WCG). Some example WCG standards are P3 and Rec.2020, which can support redder reds and greener greens. Any pixel falling outside of Rec.709 is highlighted in cyan on our debug screen.
- The pixel is out of SDR brightness range. Brightness or luminance is measured by the unit known as a candela (cd/m2) otherwise known as a ‘nit’. While sRGB relies on a gamma curve that targets 80 nits in a dark viewing environment, HDR uses a PQ curve that can specify up to 10,000 nits. The precise definition of ‘SDR brightness range’ is debatable since SDR displays have been capable of brightness levels higher than 80 nits for years. This has resulted in SDR pixel brightness being boosted far above what was originally intended by the source content in order to maximize brightness to better match the capabilities of the display. For this reason, we chose to have the option to adjust what we consider to be SDR brightness (or reference white level). Any pixel above that is highlighted in magenta on our debug screen.
- The pixel is out of SDR color space AND SDR brightness range. These are pixels that satisfy both of the previous requirements. Any pixel outside of Rec.709 and above SDR brightness range is highlighted in yellow on our debug screen.
After visualizing where the HDR pixels actually are in real time within our production Netflix experience, we can now begin to understand where the benefit of HDR really shines (literally speaking).
HDR Image Format Requirements
Next came the problem of how to store these HDR pixels into an image file format. The difficulty in choosing a format is that there is currently no commonly accepted standard in the industry. With the understanding of what an HDR pixel consists of, we were then able to define several requirements for what an HDR image file format needed:
1. Color profiles
Ultimately, image files are nothing more than a bunch of numbers stored in a container. With that basic concept in mind, it’s not clear what makes an HDR pixel different from an SDR pixel. For example, imagine an image format containing 8 bits for each red, green, and blue channel of a pixel (this is very typical currently). To represent maximum white in this image we would store RGB(255, 255, 255). But the question is: ‘How bright of a white is that? For maximum red of RGB(255, 0, 0), how saturated of a red is that?’
Currently, images that don’t have explicit color profiles are usually treated as using the limits of the current display, meaning that these maximum white and red values would just end up being the whitest and reddest that the screen can handle. This has served us reasonably well for a long time and for many use cases, but since every screen has different capabilities in terms of color gamut and brightness, it gives different results on every screen. What if we want to differentiate between a pixel showing a white piece of paper and one showing a bright halogen light on various screens?
In order to know what those numbers are intended to mean in terms of an actual color value and brightness intensity, we need to know the color space (color primaries) and the brightness (absolute luminance) those values represent. Furthermore, there needs to be a transfer function (tone reproduction curve) that helps represent brightness levels in a way that more closely matches human visual capabilities. As mentioned earlier, SDR images mainly specifies sRGB that defines Rec.709 color primaries at 80 nits using a gamma curve. For HDR images there is no common and widespread standard yet, though HDR10 video uses Rec.2020 color primaries at 10,000 nits and a PQ curve.
2. Higher bit-depth
In order to specify a wider gamut of colors (Rec.2020) and higher luminance (0~10,000 nits) we really need more code points to represent that additional range of values nicely. While it’s possible to stuff a wider range of values into existing 8-bit per channel formats, that leaves less code points in between which can result in banding artifacts, typically seen in gradients (see image below for an exaggerated example of this). Just like HDR video requires at least 10-bits (per HDR10 standard), so do our images.
3. Codec support
Unlike HDR formats used for photography, it is critical to be able to compress our images as much as possible without losing too much quality, so that they can be delivered efficiently to millions of members under all kinds of network conditions. Also, we need to be able to decode them very quickly on the device so we can get them displayed on the screen as fast as possible. For some images in our UI, we want to have the option of an alpha channel for transparency. Although we found a variety of file format options that are ‘HDR-capable’, most did not meet these criteria, lacked enough adoption by the community, or their open source libraries were not license-friendly for us.
4. Tooling support
Before we can even think about delivering HDR images to devices we need to first create them somehow. There obviously need to be tools that artists and designers can use to author the HDR images that get injected into that pipeline in the first place. Artists need to be able to import/export these files and understand how to manipulate these assets within the desired color space and brightness levels.
Solving for the HDR Image Format
After exploring all of the requirements above we ultimately came to the realization that many common image file formats that have existed for years can already technically support the characterization of HDR pixels, but there are trade-offs for each one. The key realization was that we could add an ICC Profile to each image to signal the HDR color profile of that image.
ICC Profile — The information needed to define and convert between color profiles (color primaries, tone reproduction curve, absolute luminance) can be encoded in an ICC Profile, which is an ISO-standardized binary file format that can be embedded in many image file formats. It’s also natively supported by industry standard tools like Adobe Photoshop, and by operating systems like Windows and macOS.
We chose to adopt the ICC Profile as our source of truth for defining color profiles and to differentiate between SDR and HDR images while allowing both to coexist even if they are stored in the same file format. As long as the image file format supports embedding of the ICC Profile, it satisfies our current needs. The only tricky thing is that as far as we are aware, no existing tools pay attention to the standard absolute luminance value specified in the ICC Profile. This required us to develop some custom tooling that understands absolute luminance and can generate custom ICC Profile definitions with this luminance value defined for our needs.
Potential Options — We compared a lot of possible options for use as our initial HDR image file format. Here is a table showing some of the pros and cons for a few formats that have broad support and are decodable by our devices:
Source and Delivery Formats — Right away we concluded that 8-bits were simply not enough to maintain a desirable picture quality for most HDR images. We ultimately decided on 16-bit PNG for our source image format and JPEG2000 for our compressed delivery image format.
The source image format is what the artist might export out of Photoshop after authoring an HDR image. It needs to be lossless to preserve the pristine source and at this stage the encoded size is not too much of a concern. 16-bit PNG made a lot of sense here considering its ubiquity in tools and devices.
The delivery image format is what we actually download for decoding and rendering on devices. Unlike the source format, here we need to carefully balance the desire for small file size and fast decompression with the desire for optimal image quality. JPEG2000 was a good choice among the higher-bit depth compressed formats above, primarily due to its acceptable performance and availability of an open source library to support it.
Emerging Formats — Our decision to use 16-bit PNG and JPEG2000 in this very first phase of HDR image experimentation is by no means final. We are expecting that new emerging formats may become more standard in the future and are ready to adjust based on the broader industry adoption of some new HDR image file format.
One such format is HEIF, which can store single frames (among other things) from HEVC encoded HDR video. However, due to its relative infancy there is still lack of support for it in terms of license-friendly open source decoders, device hardware decoder support, and authoring tools that can import/export HEIF.
There is also work being done on a new AV1 image format that looks promising, and we’re hopeful this royalty-free format will continue to make progress and gain the adoption needed for widespread use.
Updates to the Netflix App
Once we had actual HDR images to work with, the next step was to add HDR image awareness to our device software implementation. Our Netflix SDK and rendering engine is responsible for powering the Netflix TV Experience to all CE devices including our HDR game consoles.
Color Profile Support — We first needed to make our rendering engine color profile aware. This means associating color profiles with our image textures and perform color profile conversion on-the-fly (using GPU shaders) while rendering them. This simply means that we can now support compositing and rendering of any color profile whether it be an sRGB (SDR), Display P3 (WCG), Rec.2020 (HDR), or other arbitrary color profiles, so long as the source image file format can communicate the appropriate color profile. Currently we rely on the ICC Profile embedded into the image file format, but if/when other industry standard HDR image formats emerge, we should be able to support those as well assuming they will have some way to define their color profiles (whether this be explicit in the format itself or implicit through some standard).
Higher Bit Depth Surfaces — In order to properly load image files with per-channel bit depths greater than 8, we must support surfaces (textures and render targets) whose bit depth is at least as large. For color channels (red, green, and blue), 10 bits is enough if you store these values with the PQ curve. However, current hardware requires these pixels are stored in 32 bits, leaving only two bits for the alpha channel, which isn’t sufficient for complex UI composition (feathered edges, cross fades, etc). Because of this, we must use even larger bit depth surfaces (64 bits per pixel) than actually required. The additional cost in memory and bandwidth is not cheap, as you can imagine.
Blending Curves — The other problem presented by HDR is the introduction of the PQ curve in addition to the long understood gamma curve that is associated with SDR sRGB images. In order to properly blend images with different curves in a mathematically correct way, you must first de-curve (or linearize) the values. However, existing sampling and blending hardware does not compensate for tone curves which has resulted in ‘incorrect’ gamma blending of sRGB for all these years. To make matters even worse, this ‘incorrectness’ is widely understood and unfortunately now expected within the current state of SDR graphics pipelines. But now with the introduction of HDR and an even steeper curve such as PQ, these blending issues have far greater negative visual effect. In order to work around this, we only enable blending in hardware when compositing surfaces with a simple gamma curve, and convert that final surface into the PQ curve as a separate step using color conversion shaders with blending disabled. This maintains years of blending expectations for existing SDR images while blending HDR correctly, but at the cost of additional memory and GPU time.
The latest Netflix update on game consoles already includes all the changes described above. With the end of the HDR processing pipeline ready to consume HDR images, the final piece of the puzzle was getting our UI to serve the first set of experimental HDR images from our image asset CDNs.
The above is a screenshot of our HDR image billboard currently live in our production UI with the HDR highlighter feature enabled. It’s visible proof that the Marvel’s Iron Fist billboard image packs quite the punch with HDR pixels!
More Challenges Ahead
Our exploration into bringing HDR images into the UI has definitely pushed us to various limits of the current state of technology that exists throughout this ecosystem. But we are still far from being done, with many more challenges anticipated ahead.
HDR Workflows — While we have a solution for introducing HDR into parts of our image asset workflow pipeline, we have much more work to do internally within Netflix to introduce the concept of HDR into the complete end-to-end workflow. You can start to imagine what that workflow might look like when working with a variety of content production studios and design firms that generate a variety of image assets that then need to be imported into our own internal systems for further processing until they reach the user. We have product guidelines, art asset templates and various tools in place to support all of that, none of which currently have any concept of HDR images. Even the authoring of our experimental images required our engineers to explain to our creative designers how we might be able to generate HDR assets using their own tools like Photoshop. The concept and experience working with HDR images from the creative design perspective is all still very new and foreign.
Ecosystem Support — Overcoming this creative learning curve is also hindered by the chicken-or-the-egg problem introduced by current limitations presented by devices. HDR support in terms of PCs and monitors is definitely starting to emerge now at an increased rate. However, while HDR displays are becoming more abundant, OS and application support that allows the artists to actually see HDR pixels on those displays is not there yet. Imagine asking an artist to create a beautiful image of a rainbow on a screen that could only show shades of gray. This is similar to what we are asking for with HDR images: we want them to create images with brighter and more saturated colors than they can possibly see on their screen.
Device Hardware — Furthermore, the limitations in current generation devices go further down into CE hardware architecture. We are currently able to explore HDR images on the game consoles as they provide powerful but flexible hardware designed to push games to their graphical limits. They are built such that we can support larger bit-depth textures and composite our UI graphics and video with relative ease. The same luxury does not exist on other CE devices like TVs and mobile phones that have very specialized hardware built specifically for their graphics and video needs. The step up from 8-bits to 10-bits and above is also a huge one — especially when 8-bits have been the standard for so long. One of our focus areas is to help our partner device ecosystem to enable support for this feature in the coming years.
Final Thoughts
We have been reaching out to other industry leaders to have discussions and to share our knowledge and learnings around this topic. Without industry-wide collaboration and momentum, we risk being stuck with SDR images in an HDR world. We hope we can work together to help push the industry forward so that viewers can enjoy an even richer experience on every device.
Netflix works hard to be at the forefront of innovation, especially when it comes to delivering the best-in-class experience for our global members. While we are excited about reaching this milestone of delivering HDR images to select game consoles, we are eager to push the technology and capabilities further into many more devices and experiences.
We are always open to new ideas on how we can improve. Interested in learning more or contributing? Please reach out!
