How to Create Ultra-Realistic Scenes in Three.js

Published on 09 Aug, 2024 | ~12 min read | Demo

Creating hyper-realistic objects or making the models you created or downloaded look ultra-realistic in a Three.js scene is easier than you might think and doesn't require hundreds of lines of code. However, understanding the meaning and concepts behind those lines can be a bit of a struggle, especially if you have no prior experience with image processing and related technical aspects.

With that being said, in this article, I'll explain some of the fundamental definitions and techniques you'll need to understand for your creative journey with Three.js, 3D graphics, or even image and video processing if you decide to explore those areas in the future.

The final project — A demo of what we'll be creating.

Dynamic Range

Dynamic range refers to the capability of a sensor (like a camera sensor) or a display device (like a monitor or screen) to represent the full spectrum of light in an image. This means accurately capturing or showing the darkest shadows and the brightest highlights within the same image.

So basically, we're talking about how many details a camera lens can capture in the shadows and highlighted areas.

Having said that, the wider the dynamic range of a camera, the more details it can capture. The same goes for monitors: the wider a monitor's dynamic range, the more details it can display.

Here's an example of different dynamic ranges.

As you can see in the picture on the left, the window area has excessive exposure, which hides details that could be important in the picture.

On the other hand, in the picture on the right, which is taken or displayed by a device with a higher dynamic range, we can see more details in that same area.

The same applies to dark areas. In the lower part of the left picture, some details are hidden in the shadows, which the device on the right can capture or display.

This is essentially where the term HDR comes from, if you've ever seen it in monitor or camera specifications.

HDR

HDR stands for High Dynamic Range. This feature allows an equipment to capture, process, or display a broader range of brightness and color, resulting in more vivid and detailed images.

Now, even though cameras have evolved significantly, they still can't capture tones as effectively as human eyes do. In other words, a camera can't capture as much detail in a single shot as human eyes can.

That being said, photographers use a technique to achieve similar results for their pictures by taking one photo at a mid-tone and several others at different light intensities or exposures. They then merge these images using photo editing software like Adobe Lightroom.

An interactive tool that showcases the set of pictures used to create one HDR image

The result of that process is stored in a file with an .hdr or .exr extension. You will use these types of files to create your ultra-realistic scenes in Three.js or Babylon.js.

Note: .exr and .hdr files are significantly larger than classic .jpg or .png files because they contain a substantial amount of uncompressed data, unlike the other types, which are compressed.

Now that we've learned that .hdr images are a special type of image that holds a large amount of data, we encounter a problem: most screens are unable to display such a high level of detail.

Tone Mapping

An HDR image is encoded in 16 or 32 bits, while most monitors can display only 8 or 10 bits. This discrepancy results in a loss of detail when the image is displayed.

HDR 1632 bits vs monitors 810 bits — HDR 16/32 bits vs monitors 8/10 bits

Fortunately, there is a solution to this problem called tone mapping.

Essentially, tone mapping is the process of adjusting an HDR image to make it look as close as possible to its original HDR form on screens. In other words, we're compressing the image from 16 or 32 bits to 8 or 10 bits while striving to preserve as many details as possible.

This can be done in a couple of ways. The first is by using photo editing software like Photoshop. The second is through a tone mapping algorithm, which is available in Three.js.

Gamma Correction

In reality, when we have a light source, there are x amount of photons traveling from that source to the sensor, whether it's human eyes or a camera lens.

Having said that, if we add another light source with the same characteristics as the first, we should have double the amount of photons reaching the sensor. This means we should perceive double the intensity of the light, right?

Well, that's physically correct, and that's exactly how cameras work. However, it's not the same with human eyes, as they are more sensitive to darkness than to light.

Double light sources don't necessarily mean double the intensity for the human eyes

So what does that mean?

That means that even if we double the light sources, our eyes will trick us into perceiving less than double the intensity.

Having said that, if we translate this into a graph, you'll see that the relationship between light intensity and how it should be perceived is linear. This is how cameras capture light.

However, when it comes to our eyes, the relationship follows a curve rather than a straight line. And this is just the first part of the story.

The way light is perceived by human eyes

The second part is that when a monitor emits light to display an image, it actually follows this curve.

That means monitors display images much darker than they actually are, and this is due to the application of what is known as gamma.

To solve this problem, engineers devised a simple solution: applying an inverse curve to the image before display. This adjustment restores the graph to a linear relationship at the time of display.

And again, that leaves the rest to the eyes to look at it the way we naturally do.

This process is called gamma correction, and it's performed internally in compressed image files like .jpg or .png. This means that images saved in such files are actually brighter than they appear.

.EXR and .HDR images, on the other hand, are saved in their linear format and do not have the inverse curve applied. This is why they are widely used in CGI and game development, as renderers require a linear workflow to work effectively with images.

This implies that ideally, the renderer in Three.js, for example, wants to work with the original colors of an image to make accurate calculations of pixel colors. It then handles gamma correction by itself using an encoding algorithm, typically sRGB encoding.

sRGB?! Aren't we done with these terms already?

Relax, this is the last and most important term, because you use it not only to create realistic scenes but also when working with simple textures.

Color Space

sRGB stands for Standard Red Green Blue. It is a subset of the colors that the human eye can perceive and is often referred to as a color space.

So basically, when we apply gamma correction to an image, we're trying to represent its original colors at the time of display using a subset of colors, usually within the sRGB color space.

There are various color spaces besides sRGB, but that's not something we need to explore—at least not in this tutorial.

Now that we've wrapped up the boring stuff, let's return to our main topic: how to create the realistic scene we saw earlier.

The Coding Side

First and foremost, create a basic scene. To save time, you can clone this Three.js boilerplate.

You'll also need an .hdr image. Download one and place it in the public folder.

Now, we need a special Three.js loader to load the HDR image, which is different from the classic TextureLoader. This one is called RGBELoader.

import { RGBELoader } from 'three/examples/jsm/loaders/RGBELoader.js';

Now, we'll create an instance of RGBELoader, passing the HDR file location as the first argument and a callback function as the second argument. The callback function will receive the loaded texture as its argument.

const rgbeLoader = new RGBELoader();

rgbeLoader.load('/env_map.hdr', function (texture) {
  
});

Then, we'll set that texture as the background for the scene.

scene.background = texture;

As you can see, the image looks strange because it's a 360-degree photo.

If you recall from the Three.js guide, we changed the background in a couple of ways.

The first method was by setting a normal image, and the second method involved setting an image for each of the six faces of the scene. I mentioned six faces because a scene is essentially represented as a cube, which has six faces.

Another way is to use a 360-degree texture and let the renderer create a surrounding environment from it.

To do this, we need to set the texture's mapping property to EquirectangularReflectionMapping.

texture.mapping = THREE.EquirectangularReflectionMapping;

And there you go—you should now have your background set correctly.

A couple of versions ago, the scene displayed incorrect colors because the color space wasn't set. We had to correct this manually with the following line.

renderer.outputColorSpace = THREE.SRGBColorSpace;

Now we don't need to, as it is set by default.

As you can see, the colors at the window appear too bright, obscuring details that might be important in the image.

Details are lost because the highlight is too bright

To fix this, we simply need to apply a tone mapping algorithm. This term should be familiar to you, right?

Three.js offers a handful of built-in tone mapping algorithms. The one we’ll use in our example is ACESFilmicToneMapping.

renderer.toneMapping = THREE.ACESFilmicToneMapping;

Details are revealed after applying tone mapping

Furthermore, we can adjust the exposure by modifying the toneMappingExposure property value.

renderer.toneMappingExposure = 1.8;

To create the spheres, we need to use MeshStandardMaterial or MeshPhysicalMaterial to achieve the realistic results we desire.

//renderer.outputColorSpace = THREE.SRGBColorSpace;
renderer.toneMapping = THREE.ACESFilmicToneMapping;
renderer.toneMappingExposure = 1.8;

const rgbeLoader = new RGBELoader(loadingManager);

rgbeLoader.load('/env_map.hdr', function (texture) {
  texture.mapping = THREE.EquirectangularReflectionMapping;
  scene.background = texture;

  const sphere = new THREE.Mesh(
    new THREE.SphereGeometry(2, 50, 50),
    new THREE.MeshStandardMaterial({
      color: 0xffea00
    })
  );
  sphere.position.x = -2.5;
  scene.add(sphere);
});

With this code, you should now see a black hole instead of a shining yellow sphere. This is because MeshStandardMaterial requires a light source to be visible, and since there isn't one in our scene, it appears this way.

Actually, this material requires either a light source or an environment map. This means we can skip adding a light source and instead use the environment map to provide lighting information, resulting in realistic reflections.

To do that, we’ll simply add this line:

scene.environment = texture;

This will set an environment map for the scene, affecting all materials within it. If you want the environment to affect only a specific material, you can skip the line above and set the environment map directly as a property of that material.

// scene.environment = texture;
  
const sphere = new THREE.Mesh(
    new THREE.SphereGeometry(2, 50, 50),
    new THREE.MeshStandardMaterial({
      color: 0xffea00,
      envMap: texture
    })
);
sphere.position.x = -2.5;
scene.add(sphere);

Now, the sphere appears with realistic shading.

To add reflections, we simply need to reduce the roughness value in the material.

new THREE.MeshStandardMaterial({
    roughness: 0,
    envMap: texture,
    color: 0xffea00,
  })

If that’s not enough reflection, we can enhance it by increasing the metalness.

To create a sphere that looks like it’s made of glass, we need to use MeshPhysicalMaterial.

In addition, we need to set both the metalness and roughness to 0. We also need to set the transmission property to 1.

const sphere2 = new THREE.Mesh(
    new THREE.SphereGeometry(2, 50, 50),
    new THREE.MeshPhysicalMaterial({
      roughness: 0,
      metalness: 0,
      transmission: 1,
      envMap: texture
    })
  );
sphere2.position.x = 2.5;
scene.add(sphere2);

With that done, we can enhance the visibility of the reflections on the sphere's surface by adjusting the ior (index of refraction) property, which should be set to a value between 1 and 2.333.

new THREE.MeshPhysicalMaterial({
    ior: 2.33,
    roughness: 0,
    metalness: 0,
    transmission: 1,
    envMap: texture
  })

Summary

In this article, we covered essential concepts for building realistic scenes in Three.js, including HDR, tone mapping, color space, and gamma correction.

We discussed how HDR captures a wide range of brightness, tone mapping adapts these images for standard displays, and gamma correction ensures accurate color representation.

Using these concepts, we demonstrated how to create realistic scenes with examples of material properties.