Increasing the number of lights in Qt 3D

While it is possible to draw scenes with almost unlimited numbers of lights using deferred rendering in Qt 3D, the default materials in Qt 3D Extras have been limited to eight lights because they are tied to forward rendering. Although we apply a few tricks that allow you to define more than eight lights in the scene, we only select the eight closest lights when rendering.

In the scene below, we have defined 100 lights that float around, but because only 8 of them are in use at the time and you can see how they pop in and out:

https://youtu.be/5k3Fd5t_KBo

Although it's possible to support more lights in other pipelines (this is for instance the case with the Qt 3D Studio Runtime), we also want the default materials in Qt 3D Extras to be able to use a larger number of lights, even when used with forward rendering.

The reason behind the current limit is that we need to support a range of OpenGL architectures, including some which are quite limited in terms of how much uniform data can be passed to the shader. We therefore need to pass a small array of light structs to the shader and loop over the lights in the fragment shader. Unfortunately, such loops are also very expensive on hardware with limited resources because they compile to a large number of shader program instructions. Sometimes so many that the resulting shader program won't compile at all.

We have therefore started work on a solution that will both allow us to increase the number of lights on beefy hardware and at the same time reduce the number of shader program instructions on weaker hardware. It is based on Qt 3D's new internal shader graph, which was recently introduced by Kevin Ottens from KDAB.

The shader graph is a game changer for Qt 3D. It allows us to build shaders dynamically with a structure similar to the material graphs you find in many modern 3D modeling applications. Each node in the graph can be thought of as a function - it is defined as a computation, takes a set of input values, and produces output values. For instance, a surface node can take the incoming and outgoing angles of light, the light color, and the surface color, and compute the resulting light contribution. By connecting nodes in the graph, you define the flow of computations in the shader program, which is translated to instructions in a programming language. Because all the computations are abstract, they can easily be translated to many different shader languages (or dialects), making it trivial to produce the same material definitions for different architectures.

In terms of increasing the number of lights, the great thing about the shader graph is that we can easily define a shader node for each of the lights in the scene. The graph then results in the right number of light calculations when converted to shader code. Instead of having a for loop inside the shader itself, we loop over the lights when the shader is generated. This is basically unrolling the old loop, which is way more efficient and allows us to have a much higher number of lights.

I am currently working on a prototype where we have moved the lighting system on top of the shader graph. While this work is still highly experimental, it is already producing some promising results. Here's the same scene as above, but this time we are actually seeing 100 lights in use at the same time:

https://youtu.be/zdxt_BRk-A4

Before this becomes part of the default materials in Qt 3D Extras, we also want to change how lights are gathered in a scene and passed on to the shaders. Currently, the light gathering is tightly coupled with the rendering code, but we want to reduce the number of assumptions we make and give you more control over how light information is passed to your shaders. In turn, this will also make it easier to create applications that can switch between rendering modes, for instance if you want to support both forward and deferred rendering of the same scene

If this sounds interesting for your applications, feel free to share thoughts on how you would like to work with lights in your code.