OpenDLSS: A Vulkan-Based Recreation of Nvidia's DLSS 5 Neural Rendering

This project, hosted at maanHimself/OpenDLSS-NR, provides a high-fidelity Vulkan reimplementation of the DLSS 5 Neural Rendering (NR) network. The implementation is designed to be bit-exact compared to the original NVIDIA version.

🚀 Project Overview

The core of this project is a 71-block Swin/ViT network, mirroring the DLSS-NR build 310.8.0. It leverages tensor cores to execute using $\text{FP8}$ precision.

Note on Accuracy: This is not just a visual approximation. The intermediates are identical; all 75 block boundaries match the original byte-for-byte.

Implementation Variants

1. Vulkan Path: The primary version utilizing tensor cores and $\text{FP8}$ for maximum performance.
2. WebGPU Path: Located in ports/browser-webgpu/, this is a standalone implementation. It achieves the same bit-exact results in a browser environment without relying on $\text{FP8}$ or tensor cores.

---

🧠 Network Architecture

The system utilizes a U-net structure composed of shifted-window transformer blocks, anchored by a global ViT at the base.

Technical Specifications

• Structure: 71 blocks distributed across six pooling levels.
• Precision: $\text{FP8 (E4M3)}$ activations with $\text{FP16}$ accumulation.
• Weight Size: $\approx 141 \text{ MiB}$.
• Function: Upscaling $\rightarrow$ Generative Neural Rendering.

What is "Generative Neural Rendering"?

According to NVIDIA, this process re-renders a frame already produced by the engine. It injects noise to generate fine detail and modifies the skin, structure, and tone based on a specific style setting.

Input/Output Flow:

• Inputs:
  • A low-dynamic-range (LDR) proxy of the rendered frame.
  • Three channels of Gaussian noise.
  • A reprojected version of the previous frame's output.
  • Five scalar conditioning values.
• Outputs: Four $\text{f32}$ channels per pixel (consisting of an RGB residual and one temporal-blend logit).

Example: WebGPU port at $2048 \times 1152$ resolution.

---

🛠️ Build and Execution

Setup Checklist

• Run fetch_tools.ps1 to get glslang, Vulkan-Headers, volk, CMake, and Ninja.
• Run build.ps1 to compile shaders, PTX, and the dlss5vk.exe binary.
• Run fetch_filament.ps1 and build_filament.ps1 to set up the demo environment.
• Run build_demo.ps1 to generate dlss5-demo.exe.

Command Line Interface

You can interact with the tool using the following commands:

# Benchmark performance
build\dlss5vk.exe bench --model <dir> --width 768 --height 768

# Profile per-dispatch timings
build\dlss5vk.exe profile --model <dir> --width 768 --height 768

# Verify bit-exactness against a fixture
build\dlss5vk.exe parity --model <dir> --fixture <dir>

# Bisect kernels for block-0
build\dlss5vk.exe verify --model <dir> --fixture <dir>

The demo application can be launched via double-click, loading scenes from build\scenes or via a glTF command-line argument.

---

📈 Performance Metrics

Tested on an RTX 4070 SUPER (minimum values over 40 frames).

Note: Medians are slightly higher due to the GPU alternating between two clock states.

---

📂 Project Structure

---

📋 Requirements

Hardware & Software

• OS: Windows.
• GPU: NVIDIA Ada Lovelace or newer.
• Vulkan Extensions:
  • VK_KHR_cooperative_matrix
  • VK_NV_cooperative_matrix2
  • VK_EXT_shader_float8
  • VK_NV_cuda_kernel_launch
• Toolchain:
  • Visual Studio 2022+ (C++ x64 toolset).
  • Python 3 (for PTX generation).
  • Node.js + npm & Pillow (for scene conversion).
  • Git.

Tooling

The tools/ directory (git-ignored) manages:

• glslang 16.6.0
• Vulkan-Headers v1.4.363
• CMake 3.31 & Ninja 1.13