- Optical Illusions
- Human eye
- What is color ? And what is light ?
- The CIE 1931 Color Matching Functions
- The CIE Chromaticity Diagrams from 1931 and 1976
- RGB Colorspace and its components
- Industry Standards
- Linear workflow
- Lookup table (LUT)
- OCIO (OpenColorIO)
You cannot talk about cinematography without addressing the color issue. It is essential to know within which primaries (or gamut, that is the same thing) you are working and what is your display target. Chapter 1 and 1.5 are the only technical chapters of the book and yes color management is a pain in the arse, but it will give you strong foundations to build a proper lighting later on.
It is not the easiest way to start but I would rather evacuate all the technical stuff first. Lighting in computer graphics is so connected to technology that a good understanding of these concepts is important. I have tried to make it easy to understand to anybody. We will focus on the artistic side in the next chapters. Don’t be scared !
If you don’t feel like starting by a technical chapter, I don’t blame you. You can skip directly to chapter 2.
Most of my data comes from this amazing article by Thomas Mansencal. As it is a bit technical, I have tried to simplify it for readers who are not familiar with the subject. Let’s start with this great quote from Mark D. Fairchild :
“Why should it be particularly difficult to agree upon consistent terminology in the field of color appearance ? Perhaps the answer lies in the very nature of the subject. Almost everyone knows what color is. After all, they have had firsthand experience of it since shortly after birth. However, very few can precisely describe their color experiences or even precisely define color.“
To define the subject of this chapter has been a tough one : What is color management about ? Are we talking about Primaries ? Or LUTs (Lookup Tables) ? Or Wide Gamut Rendering ? Actually, a bit of all this stuff…
There are three things to take in account when we talk about color :
- The subject
- The eye
- The brain
Here are a few examples that will show you that our brain can be easily tricked and that we should speak about color with humility.
The cornea is the clear, transparent front covering which admits light. Its refractive power bends the light rays in such a way that they pass freely through the pupil. The pupil is the opening in the center of the iris and works like a shutter in a camera. It is an adjustable opening that controls the intensity of light permitted to strike the crystalline lens. It has the ability to enlarge and shrink, depending on how much light is entering the eye.
After passing through the iris, the light rays pass thru the eye’s natural crystalline lens. This clear, flexible structure works like the lens in a camera, shortening and lengthening its width in order to focus light rays properly. The crystalline lens focuses light through the vitreous humor, a dense, transparent gel-like substance, that fills the globe of the eyeball and supports the retina.
The retina functions much like the film in a camera. It receives the image that the cornea focuses through the eye’s internal lens and transforms this image into electrical impulses that are carried by the optic nerve to the brain. Retinas are made of photoreceptors, which come in two kinds : rods and cones. We will come back to them later.
The vitreous also absorbs the UVs to protect the retina from these wavelengths.
The mantis shrimp
The creator of Guerilla Render, Ben, actually explained to me : our eyes actually only see in red, green and blue. They are crap ! Nothing compared to the mantis shrimp. Here are some pretty amazing eyes ! They have 12 photoreceptors, when humans only have 3 ! It can even see UV and Infrared, like the Predator !
Finally the brain is responsible for managing all this data. It is very well explained in this article. When the image hits the retina, it actually is upside down. The brain put its back correctly :
What we really see is our mind’s reconstruction of objects based on input provided by the eyes, not the actual light received by our eyes.
What is color ? And what is light ?
To define color, you must first define light because no color is perceptible without light.
- Light is a type of energy.
- Light energy travels in waves.
Some light travels in short, “choppy” waves. Other light travels in long, lazy waves. Blue light waves are shorter than red light waves.
From Cinematic Color : a study of color science begins with the spectrum. One measures light energy as a function of wavelengths. […] Light towards the middle of this range (yellow-green) is perceived as being most luminous.
All light travels in a straight line unless something gets in the way and does one of these things :
- Reflect it (like a mirror).
- Refract it (bend like a prism).
- Scatter it (like molecules of the gases in the atmosphere).
Light is the source of all colors. It is actually stunning how important light is in our lives. When a lemon appears yellow, it is because its surface reflects the yellow color rather than because it is really yellow. It is something that has confused me a lot but pigments appear colored because they selectively reflect and absorb certain wavelengths of visible light.
Most of these notions come from Wikipedia. And you can find plenty of articles on this topic online. I have never been very good at Physics but here is a more technical description :
Light is an electromagnetic radiation which is visible to the human eye. Visible light is usually defined as having wavelengths in the range of 400-700 nanometres between the infrared (with longer wavelengths) and the ultraviolet (with shorter wavelengths). These waves are made of photons.
The CIE 1931 Color Matching Functions
After reading the amazing Cinematic Color by Jeremy Selan, the guy who invented OCIO, I felt like it was necessary to add a text about the Color Matching Functions. I cannot explain it more clearly than Mr Selan, so I’ll just quote.
The human visual system […] is trichromatic. Thus, color can be fully specified as a function of three variables. Through a series of perceptual experiments, the color community has derived three curves, the CIE1931 color matching functions, which allow for the conversion of spectral energy into a measure of color.
The output is referred to as CIE XYZ tristumulus values, with the individual components being labelled X, Y and Z (the capitalization is important). […] When one converts all possible spectra into x,y,Y space and plots x,y they fall into a horse-shoe shaped region on the chromaticity chart.
Chromaticity diagrams are the subject of our next paragraph.
The CIE Chromaticity Diagrams from 1931 and 1976
We are going now to dig deep in the color subject. It has taken me more than six months to understand the whole process and there are still many blurry areas. But this journey through color, made with my colleague Christophe Verspieren, has been a very interesting one.
CIE scientists met in 1931 to represent the colors as we see them. The CIE XYZ chromaticity diagram was born ! They met 45 years later to improve it : CIE U’V’ was created. Even if the CIE U’V’ from 1976 is more accurate, the one from 1931 is still the most used in the color community. Old habits die hard.
- These chromaticity diagrams represent the color space of our eyes.
- 2 axis are available to give each color an unique coordinate on this diagram.
From Cinematic Color : the region inside the horseshoe represents all possible integrated color spectra; the region outside does not correspond to physically-possible colors.
The CIE XYZ serves as a standard reference against which many other color spaces are defined. Keep these diagrams in mind because we are going to constantly refer to them later on.
RGB Colorspace and its components
An RGB colorspace is composed of all the colors available in our workspace. It is defined by three components :
This is what an RGB colorspace looks like :
On the image above, there are three important things to notice :
- RGB color spaces actually exist in the 3D space (screen left).
- To make it easier to visualize, we generally only use a 2D slice, at a given luminosity (bottom screen right). In this example you can see the Rec. 709 colorspace against the CIE 1931 diagram.
- Notice the black points ? They represent the pixel values of the image (top screen right) within an RGB colorspace (in this case Rec. 709). It is called to plot the gamut.
“The primaries chromaticity coordinates define the gamut (the triangle of colors) that can be encoded by a given RGB colorspace.”
In other words, the primaries are the vertices of the triangle. We are going to complicate things a bit… Pay attention ! Each of these vertices has a pure RGB value :
- Red = 1,0,0
- Green = 0,1,0
- Blue = 0,0,1
But each of these vertices has an unique xy coordinate on the CIE diagram. That is how we are able to compare them. The only way to define a color (independently from its luminance) in an universal way is to give its xy coordinates. In this chart, you can see the coordinates for different color spaces :
Working in a wider color space will not only give us more saturated values but also more range between them. That actually makes a big visual difference !
In this image, you can clearly see the different primaries of each color space :
- Different xy coordinates.
- Same RGB values.
Do we always want to work in the biggest color space possible ? Not necessarily… We will come back to that later.
The whitepoint defines the white color for a given RGB colorspace. Any set of colors lying on the neutral axis passing through the whitepoint, no matter their Luminance, will be neutral to that RGB colorspace.
An RGB color space can have different whitepoints depending its context usage. It is a creative choice :
- If you wish to simulate the light quality of a standard viewing booth, choose D50. Selecting a warm color temperature such as D50 will create a warm-colored white.
- If you wish to simulate the quality of daylight at noon, choose D65. A higher temperature setting such as D65 will create a white that is slightly cooler.
- If you prefer cooler daylight, choose D75.
Transfer functions (OETF and EOTF)
The transfer functions perform the mapping between the linear light components (tristimulus values) and a non-linear R’G’B’ video signal (most of the time for coding optimisation and bandwidth performance).
Okay. Here things get a bit more complicated… We have seen earlier what the tristimulus values are. But what about coding optimization and bandwidth performance ? Here is the answer :
From Substance PBR guide : The Human Visual System (HVS) is more sensitive to relative differences in darker tones rather than brighter tones. Because of this, not using a gamma correction is wasteful as too many bits will be allocated to tonal regions where the HVS cannot distinguish between tones.
Transfer functions or gamma help to encode better the bits, thus increasing performance. There are two transfer functions :
- OETF : the opto-electronic transfer function, converts linear scene light into the video signal, typically within a camera. When you shoot or scan.
- EOTF : electro-optical transfer function, which converts the video signal into the linear light output of the display. When you send a signal to the screen.
Important stuff to notice is that :
- Linear is not a color space.
- Linear is not necessarily wide gamut and certainly not infinite.
- It is a transfer function which is 100% gamut dependent.
That is a very common mistake among even veteran VFX sups in the industry. Secondly, to simplify, let’s say that these transfer functions are related to gamma and tv signal :
A common belief is that a non-linear Optical to Electrical Transfer Function is needed because of the non-linearity of the human visual system. It is true that the human visual system is very complex and lightness perception is non-linear, approximating to a cube-root power law.
We are going back to our first point : human vision is incredibly complex. And that we need these transfer functions for our visual comfort.
One of the biggest fight I ever had was to convince my supervisors that linear was NOT some kind of infinite color space.In this case infinite means access to all colors of the spectrum.
Screen manufacturers and Cinema majors have agreed on some standards. Their characteristics change the way our images are displayed and rendered. Here are the four most important for us :
- Rec. 709 for internet, windows and camera photos. It has the same primaries than sRGB.
- DCI-P3 for cinema projectors.
- Rec. 2020, also called UHD TV, the future of colorimetry.
- AdobeRGB for printing projects.
What are these color spaces for ? For two things :
- Rendering Space : also called working space, for our lighting calculation.
- Display Space : should match our monitors, which obviously need proper calibration.
Rendering and Display
Rendering and Display Spaces do NOT have to be the same. It is really important to understand the distinction between those two. In CG, the Rendering Space will always have to be linear. We will detail linear workflow in a bit.
And for Display Space, when you work on a project, you have to ask yourself : are my images going to be seen on a smartphone, on a TV set or in a theater ? This is where Color Management really comes in handy. In a CG workflow, it is vital to know for every single step which primaries you are working in.
When you buy a monitor, you should check the coverage of these color spaces. For example the monitors we use at a famous Parisian studio only cover 93,7% of the P3 gamut. That’s too low ! We will never match perfectly the projectors from theaters. There is no point on working in a Display Space that does not match exactly the specifics of your monitor or needs of your project.P3 is a common RGB color space for digital movie projection, defined by the Digital Cinema Initiatives (DCI) organization and published by the Society of Motion Picture and Television Engineers (SMPTE).
What is sRGB ?
As this has been explained brilliantly in the article from Thomas Mansencal, sRGB is still a confused notion for many artists. Whats is sRGB ?
- Some say : “It is a color space !“
- Others reply : “It is a transfer function !“
sRGB is actually both ! It is a color space that includes a transfer function which is a slight tweaking of the simple gamma 2.2 curve. This is why you can also use gamma corrections (0.45 / 2.2) to be in linear or not.
From the Substance PBR guide : “It is critically important to disambiguate the sRGB OETF from the sRGB colorspace; the OETF is only one of the three components that make an RGB colorspace.”
This is for me the biggest step between a student work and a professional work. Linear workflow is compulsory in the industry. I do not know any respectable studio that does not use it.
Jeremy Selan : it is often preferable to carry over those aspects of the traditional film workflow which are beneficial even in a fully-CG context. We thus advocate thinking of an animated feature as creating a “virtual movie-set” within the computer, complete with a virtual camera, virtual negative stock, and virtual print stock.
As seen earlier, monitors have a 2.2 gamma correction to display the images correctly. It is an industry standard for visual comfort since we had CRT monitors. This is why we need to put an invert gamma curve (0.454) to compensate for that. This is a very important point. Your renders will never be correct if you do not use a linear workflow. Rendering in sRGB is just wrong !
Linear workflow example
Middle gray in a Maya linear workflow is 0.214. But it can be 0.18 in other softwares… Or even 0.218 ! Actually this value is completely arbitrary. If you look at these curves, you can read the input value : 0.5 gives an output value of 0.218.
A great effort has been put to get a decent color management in Maya. Switching from Display Space to Rendering Space is automatic ! Super handy !
- Display Space : I put 0.5 as I want to see a middle gray on my screen.
- Rendering Space : Since I am working in a linear workflow, middle gray in my render will be 0.214 which makes my calculation correct.
Check three things :
- You burn less in linear workflow.
- You get a better bounce.
- Shadows look more accurate.
From Cinematic Color : Why is scene-linear preferred for lighting ? First, the render itself benefits. Physically plausible light transport […] such as global illumination yield natural results when given scenes with high dynamic ranges. […] Light shaders also benefit from working with scene-referred linear, specifically in the area of light falloff. […] when combined with physically-based shading models, using an r² light falloff behaves naturally.r2 light falloff is just another term for quadratic decay.
Lookup table (LUT)
From Cinematic Color : Lookup tables (LUT) are a technique for optimizing the evaluation of functions that are expensive to compute and inexpensive to cache.
LUTs originally come from live-action cameras. Thanks to a list of values (table), they allow to display high-dynamic range images (HDR) on low-dynamic range monitors (LDR). Our monitors and film projectors are not able to display the entire HDR range. A LUT is the best way to display pixels on a screen and to get a nice filmic look. This is called tone mapping.
There are two kinds of LUT :
- 1D LUT (left image) : it contains only one column of numbers as it affects RGB pixels the same way. They are also called Display LUT, generally using a S-Curve Shape to display better values. I used the vd16.spi1d from the spi-anim (Sony Pictures Imageworks Animation) config as an example. If the gamut of your image is Rec. 709, a 1D LUT will NOT change it.
- 3D LUT (right image) : it contains three columns of numbers as it affects RGB pixels differently. They are used to map one color space to another. It actually modifies the primaries ! I used the Rec. 709 for ACEScg Maya.csp from the ACES 1.0.3 (Academy Color Encoding System) config as an example. A 3D LUT allows you to switch from one gamut to another. That is a very powerful tool !
From Cinematic Color : A lookup table (LUT) is characterized by its dimensionality, that is, the number of indices necessary to index an output value.
We also divide LUTs in two categories :
- Technical LUTs : to get the best display possible of your images.
- Artistic LUTs : to shift hue, luminosity and saturation based on your personal taste.
OCIO is an open source framework for configuring and applying colour transformations. […] It is designed for rolling out complex colour pipeline decisions to lots of people who might not individually understand what’s going on (ie : not colourists).Great OCIO definition that I copied from this post.
To load a LUT into Maya, Nuke, Guerilla Render or Mari, we will use a OCIO configuration. The OCIO config is the easiest way to share a LUT between different programs. And guess what ? They are also available for free. Here is what the OCIO config from Sony Picture Imageworks looks like :
- Different roles like color picking, texture painting or compositing are predefined.
- Different displays are also available between DCI-P3 and sRGB.
- Setup for Film look loads the file vd16.spi1d with a quick description.
There are several file formats for LUTs : spi1d, spi3d, csp, cube… Photoshop only accepts cube files. Otherwise you can use an ICC profile. Adobe softwares are not OCIO friendly unfortunately.
It is crucial that anyone doing look development or rendering uses the same LUT. Artists from the Surfacing, Lighting and Compositing departments need to see the same thing. Texturing in plain sRGB but rendering with a s-curve LUT will give issues. There are plenty of solutions for Mari, Substance or Photoshop to make your work consistent.
The OCIO configs are very handy because you can share them between softwares. You can use the official ones or build your own in Python. LUTs in animated movies have arrived pretty late compared to visual effects (VFX). But they are kind of a game changer in the industry. Naughty Dog has done some really good papers about them.
1D LUT example
What happens if you do not use a LUT ? Every time your shot is overexposed, you will have to manually “fix” it. That is just wrong.
Without a LUT, you will decrease your light’s intensity if it burns out. BUT you will loose some bouncing from Global Illumination (GI), some Sub-Surface Scattering (SSS)… So you will probably have to create plenty of lights to compensate for the loss of energy. And this is how you end up with a complicated rig of 50 lights ! You want to be able to put as much energy as you need in a shot.
Check the example below : this is the SAME render displayed differently ! I repeat : the exr from the render engine does not change, it is just the way we view it.
As a student I was obsessed to get a filmic look and pure blacks without making my renders look dirty. The only solution to that is to use a LUT. It will give you contrast and fix any overexposed areas. Here is what a “filmic” s-curve for tone mapping looks like :
From Cinematic Color : most tone renderings map a traditional scene gray exposure to a central value on the output display. […] one adds a reconstruction slope greater than 1:1 to bump the midtone contrast. Of course, with this increase in contrast the shadows and highlights are severely clipped, so a rolloff in contrast of lower than 1:1 is applied on both the high and low ends to allow for highlight and shadow detail to have smooth transitions. With this high contrast portion in middle, and low contrast portions at the extrema, the final curve resembles an “S” shape as shown below.
The rolloff of the highlights is pretty clear : all the values between 1 and 10 will be displayed in a range from 0.8 to 1.
In the version 2.0 of this book (release in September 2020) I will try to explain the Zone System. For the moment I will have to let you look for yourself.
We are all set for proper Color Management and ready to make some beautiful renders. But how do we keep all these values, especially above 1, in a proper format ? We can thank ILM to have invented the OpenEXR.
OpenEXR allows you to save in 16 bits half float to avoid any banding/clamping issue. You will also be able to write some Arbitrary Output Variable (AOV) for your compositing process.
From Cinematic Color : It is important to remember when dealing with float-16 data, that the bits are allocated very differently from integer encodings. Whereas integer encodings have a uniform allocation of bits over the entire coding space, the floating-point encodings have increased precision in the low end, and decreased precision in the high end.
After a couple of e-mails following the release of my website, I thought it would be worth mentioning that 16 or 32 bits linear files do not give you magically access to infinite color ranges. You still HAVE to take in account the primaries (or gamut) you are working in.
We did render some AOVs in 32 bits on Playmobil : Ncam, Pcam and Pworld for better precision.
Finally we have to take care of our range. This topic is subject to controversy in many studios ! Basically you want to go as low as possible without affecting the rendering. There are two settings to take care of :
- Clamp Pre Filter (also called Indirect Clamp Value) : Clamp the sample values.
- Clamp Post Filter (also called AA Clamp Value) : Clamp the pixel values.
Clamping the sample values will help you to reduce fireflies and noise. The lowest value I ever used was 10 and it helped me greatly. Clamping the pixel values will help your anti-aliasing for a smoother result. If you have a very dark pixel next to an overexposed pixel, you do NOT want the overexposed pixel to blow out every other pixel next to it. This is why we clamp our renders.
It may also depend on your scene and your render engine. Some studios clamp their renders at 50. I have tried a couple of times as low as 30… I guess you need to try and see for yourself. Solid Angle explains pretty well the topic of clamping if you want to read more about the subject. We will see in chapter 9 why clamping pixel values can be an issue.
Color Management is a never-ending topic that I continue to explore almost on a daily basis. I have tried to sum it up in an easily understandable post. A good color management is when you see the same thing along the color pipeline and in different softwares.
If your colors or your contrast shift when you go from Substance to Maya, it may be worth to analyze the issue. A good color pipeline is all about controlling the gamut from the first sketches (probably done by the Art Department) to final delivery. Constancy is the key.
Here are my thoughts on Color Management. It has been a great and long (a year !) journey through color for me. But it doesn’t end here. That would be too easy. If you are still up for it, we can move to chapter 1.5 about the Academy Color Encoding System : ACES.
If you are saturated with this technical talk, I don’t blame you at all ! You can just skip to chapter 2.
- This article about terminology was a game changer for me.
- If you need any information about OCIO, check their website.
- The Cinematic Color is a major document, probably the best one but it is also very technical.
- Here are some documents that show the limit of OCIO. Hopefully OCIO 2.0 will be a huge improvement on these limitations.
- Substance explains a bit about color profile.When will OCIO happen ?
- A short video in French about mid gray.
- And finally a youtube video that explains sRGB and color gamut.
- Birds of a feather OCIO Siggraph 2019 and a demo from Siggraph 2018.
- The openEXR documentation is very interesting as well.
- Interesting posts about color spaces, gamma and HDR grading.
- A semantic scholar pdf about HDR television.