OpenGL and Vulkan comparison on rendering a CAD scene using various techniques
The "threaded cadscene" sample allows comparing various rendering approaches using core OpenGL, extended OpenGL via bindless graphics and NVcommandlist as well as Vulkan. It does make use of NVIDIA specific extensions to use Vulkan within an OpenGL context and display a Vulkan image.
Note This sample currently builds two executables: glvk... has both GL and Vulkan within the same GL window using a NVIDIA extension, the vk... one uses WSI and pure Vulkan. There is also a cmake build option to remove the glvk_ exe and build the only the vk_ exe. For Vulkan it is recommended to use the Vulkan stand-alone exe for Vulkan performance investigation and debugging.
The content being rendered in the sample is a CAD model which is made of many parts that have few triangles. Having such low complexity per draw-call can very often result into being CPU bound.
The sample is a fork of the public cadscene OpenGL sample and introduces the usage of multiple CPU threads.
By default a performance graph is drawn that illustrates the relative costs of rendering. The top green bar represents GPU time, and the lower blue bar CPU time. The bar that extends the furthest is the primary bottle-neck. Be aware that the GPU costs may appear higher than actual, when very CPU bound as discussed below in the performance section. When cloning the model multiple times the scene complexity is increased, however the sample does not use instancing for this, but generates actual bindings/drawcalls for the new objects. An animation effect can also be activated to "explode" the model by modifying the matrices on the GPU.
The sample loads a cadscene file (csf), which is a inspired by CAD applications' data organization, just that for simplicity everything is stored in a single RAW file.
The Scene is organized in:
TreeNodes: a tree consisting hierarchical information, mapping to Matrix indices
Materials: just classic two-sided OpenGL phong material parameters
Geometries: storing vertex and index information, and organized in
Objects, that reference Geometry and have corresponding
When animation is activated, the matrices will be animated through a compute shader, resulting in a "explosion view" of the model. One buffer stores all matrices, another all materials. Geometry is stored in multiple buffers for OpenGL, and only one in the Vulkan case (can be changed via
When the "min statechanges" option is enabled, we will draw first solid then edges, effectively reducing the number of shader changes dramatically.
These influence the number of drawcalls we generate for the hardware and software. The strategy is applied on a per-object level. Imagine an object whose parts use two materials, red and blue:
material: r b b r parts: A B C D- materialgroups Here we create a per-object cache of drawcall ranges based on the object's material and matrix assignments. We also "grow" drawcalls if subsequent ranges in the index buffer have the same assignments. Our sample object would be drawn using 2 states, which are creating 3 hardware drawcalls: red are ranges A, D and blue is B+C joined together as they are next to each other in the indexbuffer. - drawcall individual We render each piece individually: red A, blue B, C, red D.
Typically we do all rendering with basic state redundancy filtering so we don't setup a matrix/material change if the same is still active. To keep things simple for state redundancy filtering, you should not go too fine-grained, otherwise all the tracking causes too much memory hopping. In our case we have 3 indices we track: geometry (handles vertex / index buffer setup), material and matrix.
Global sorting of all items can be enabled with the "min statechanges" option. This will sort all items to reduce the amount of state changes.
Upon renderer activation the scene is traversed and encoded into a list of "drawitems". Optionally this list can be globally sorted once with the "min statechanges" option. All renderers operate from the same "drawitems" list.
By default all renderers also use the same principle state and binding sequences. VBO/IBO bind per geometry change, a uniform buffer range bind per material and a matrix change. Vulkan does allow alternative ways to pass the uniform data, which is discussed in a separate file.
default: we traverse the list and either directly render from it, or build temporary command-buffers every frame.
re-use: command-buffers are built only once and then re-used when rendering. This typically yields lowest CPU costs.
MT: multi-threaded, makes use of N worker-threads to build the command-buffers. The list is processed in chunks of workingset many items. Each thread grabs an available chunk. The global drawing order may be different every frame. When batched submission is active, the Vulkan renderers trigger their submission method at the end of their processing, i.e. once per frame.
glBindBufferRange(GL_UNIFORM_BUFFER, usageSlot, buffer, index * itemSize, itemSize). (Using only OpenGL 4.3 core features)
To reduce complexity, the renderers for
VK_NVX_device_generated_commandswere removed, they will get their own sample.
The sample allows to feed the material and matrix data in various ways to Vulkan by changing the
UNIFORMS_TECHNIQUEdefine. Please find much more detailed information here.
Preliminary Results: The Vulkan driver is still very new and the performance numbers are therefore not final. The benchmark was run on a Quadro M6000, with a first generation i7-860 CPU.
Important Note on GPU timings The GPU time is measured via timestamps. If the "begin" timestamp is in a different driver submission to the GPU (aka pushbuffer) than the "end" timestamp, then CPU bottlenecks will skew the GPU timings. The CPU time between submissions will impact the reported GPU time. This may be the case when very many GPU commands are created per-frame, although it typically affects OpenGL more than Vulkan.
To get more exact timings about the relative cost of each section, the application does measure the CPU time for the "flush" (Vulkan vkQueueSubmit, OpenGL glFlush) as well. The application does three flushes when animation is active, and two otherwise.
In this scenario we generate around 44k drawcalls all using the same shader. The cost of uniform and geometry buffer binds as well as drawcalls (average ~50 triangles) dominates.
|GPU time||CPU time [ms]|
|vk cmd 1 thread||1.0||1.8|
|vk cmd 2 threads||1.0||1.0|
|gl nvcmd 1 thread||1.2||0.9|
|gl nvcmd 2 threads||1.2||0.6|
|vk re-use cmd||1.0||0.03|
|vk re-use obj-level cmd||3.6||1.6|
|gl re-use nvcmd buffer||1.0||0.05|
|gl re-use nvcmd compiled||1.0||0.05|
Our CPU costs are nicely reduced with Vulkan and scale with more threads. By re-using command buffers we can remove the CPU costs almost entirely. We can also see that the re-use of many tiny command buffers is not recommended (in this scenario around 10k secondary buffers per-frame).
The NVcommandlist can be a tad faster than Vulkan on the CPU-side, as it benefits of having the API designed directly for the hardware in mind. There are no function calls and the data is formatted in a hardware friendly way. It is expected that Vulkan's performance will improve or that similar mechanisms will be exposed.
Now we render edges on top of the surfaces. We still generate around 44k drawcalls but half are for triangles, the other for lines. Within every object we toggle between the two and change the shader (not optimal but serves as stress test). The cost of those shader changes now dominates.
|GPU time||CPU time [ms]|
|vk cmd 1 thread||3.5||1.8|
|vk cmd 2 threads||3.5||1.0|
|gl nvcmd 1 thread||3.5||1.5|
|gl nvcmd 2 threads||3.5||1.5|
|vk re-use cmd||3.4||0.6|
|vk re-use obj-level cmd||3.6||1.0|
|gl re-use nvcmd buffer||3.0||1.1|
|gl re-use nvcmd compiled||2.8||0.05|
In this scenario we can see that the modern principles of Vulkan (and NVcommandlist) are on a whole new level when it comes to CPU efficiency (and NVIDIA's Quadro OpenGL driver is well-optimized). The shader toggles are just way too costly for a classic graphics API approach.
The NVcommandlist doesn't scale with CPU threads as the shader state switching is handled on the main submission thread in a serial fashion. Vulkan, however, still can scale by using more threads.
Overall Vulkan is a big improvement over unextended OpenGL, especially re-using CommandBuffers can result in greatly reduced CPU cost. In this particular sample NVcommandlist is often similar to Vulkan, which does show that OpenGL with proprietary extensions can be very fast. In pure draw-call limited scenarios core OpenGL's MultiDrawIndirect can also be very efficient as seen in another sample.
As this sample only uses two pipeline objects (or state-objects in OpenGL) and the rest of the scene's resources are static as well, Vulkan cannot make use of its greatest strength: generating and validating resources in parallel. Therefore, benchmarks in samples like this are very directed at a certain problem and only demonstrate a snapshot of an APIs capability.
Make sure to have installed the Vulkan-SDK. Always use 64-bit build configurations.
For best Vulkan performance use the vk exe (starting with vk_ ). If you are not interested in building the OpenGL & Vulkan combined exe then use the
Ideally clone this and other interesting nvpro-samples repositories into a common subdirectory. You will always need shared_sources and on Windows shared_external. The shared directories are searched either as subdirectory of the sample or one directory up.
If you are interested in multiple samples, you can use build_all CMAKE as entry point, it will also give you options to enable/disable individual samples when creating the solutions.
gl cadscene render techniques is most similar to this sample and covers various OpenGL approaches. gl commandlist basic illustrates the core principle of the NVcommandlist extension. gl occlusion cullling also uses the occlusion system of this sample, but in a simpler usage scenario.
When using classic scenegraphs, there is typically a lot of overhead in traversing the scene, it is highly recommended to use simpler representations for actual rendering. Flattened hierarchy, arrays... memory friendly data structures, data-oriented design. If you are still working with a classic scenegraph then nvpro-pipeline may provide some acceleration strategies to avoid full scenegraph traversal, of which some are also described in this GTC 2013 presentation.