Upscaling, HDR and ray tracing

Matchstick turns on the newer GPU features for you when the hardware has them. Your module asks what is available, renders the inputs, and makes one call.

You will learn

  • How to find out what the player's GPU supports
  • How to upscale with DLSS
  • How to add frame generation and low latency
  • How to check for ray tracing
  • How to write HDR output

Check what is supported

You never request a feature. The host starts whatever the device supports and reports it through IMGDFRenderSettingsManager.

FeatureAskNot supported when
Upscaling (DLSS, DLAA)GetUpscaleModeCount
GetUpscaleMode
The count is 0.
Frame generationGetMaxGeneratedFramesIt returns 0.
Low latency (Reflex)GetLowLatencyModeCount
GetLowLatencyMode
The only mode is MGDF_LOW_LATENCY_OFF.
Ray tracingGetRayTracingSupportIt returns FALSE.
HDRGetCurrentOutputDisplayInfoSupportsHDR is FALSE.

Read the answers in RTBeforeFirstDraw. Read them again in RTDeviceReset, because a new device can support different things.

BOOL __stdcall Module::RTBeforeFirstDraw(IMGDFRenderHost *host) {
  MGDF::ComObject<IMGDFRenderSettingsManager> settings;
  host->GetRenderSettings(settings.Assign());

  // An empty list means the device cannot upscale
  _upscaleModes.clear();
  for (UINT64 i = 0; i < settings->GetUpscaleModeCount(); ++i) {
    MGDFUpscaleMode mode;
    if (settings->GetUpscaleMode(i, &mode)) _upscaleModes.push_back(mode);
  }
  // 0 means no frame generation
  _maxGeneratedFrames = settings->GetMaxGeneratedFrames();
  return TRUE;
}

Build your settings menu from the same lists, and show only what the device offers.

NoteDLSS, frame generation and Reflex need an NVIDIA GPU. Everywhere else the lists come back empty, so always keep a render path that works without them.

Upscale with DLSS

DLSS lets you render the scene at a fraction of the window size and still show a sharp, full-size image. The GPU shades far fewer pixels, so the frame rate goes up.

It works by combining many frames. A single small frame doesn't hold enough detail to fill the window, but the last several frames together do. You supply three things so the upscaler can line those frames up:

InputWhat it tells the upscaler
JitterWhere inside each pixel this frame sampled the scene.
Motion vectorsWhere each pixel was in the previous frame.
DepthWhich surface is in front when two overlap.

The upscaled image then goes back to you. Draw your post-processing effects, such as bloom and film grain, and then your UI on top of it. Both run at full size, so text and fine detail stay sharp.

Scene render size, jittered Upscale host Effects and UI output size Frame generation host, optional Present color ยท depth motion vectors

Choose a render size

Pick a mode from the supported list and ask the host how big to render. The answer depends on the window size, so ask again in RTBackBufferChange.

BOOL __stdcall Module::RTBackBufferChange(IMGDFRenderHost *host) {
  MGDFBackBufferInfo info = {};
  host->GetBackBufferInfo(&info);

  // Fails when the device cannot upscale in this mode
  MGDFUpscaleRenderSize size = {};
  _upscale = SUCCEEDED(host->GetUpscaleRenderSize(
      MGDF_UPSCALE_MODE_QUALITY, info.Width, info.Height, &size));

  // Scene targets at the render size, everything after at the window size
  return _upscale
      ? CreateRenderTargets(size.Width, size.Height, info.Width, info.Height)
      : CreateRenderTargets(info.Width, info.Height, info.Width, info.Height);
}

The modes run from MGDF_UPSCALE_MODE_DLAA, which renders at full size and only anti-aliases, down to MGDF_UPSCALE_MODE_ULTRA_PERFORMANCE. See MGDFUpscaleMode.

Jitter the camera

A small frame samples the scene once per pixel, always at the pixel center. Render it the same way every frame and the upscaler sees the same samples over and over, with nothing new to add.

Jitter fixes that. Each frame, nudge the projection by a different offset of less than a pixel. Every frame then samples a slightly different spot, and over a few frames those samples cover the gaps between pixels. That is where the extra detail comes from.

// A repeating low-discrepancy sequence spreads the samples evenly
constants.JitterOffset[0] = Halton(_frameIndex, 2) - 0.5f;
constants.JitterOffset[1] = Halton(_frameIndex, 3) - 0.5f;

The offset is in render pixels. Shift the projection you draw with by that amount, but pass the host the matrices without it. The host needs both to undo the nudge when it combines frames.

Upscale each frame

Give the host the camera and jitter for the frame, draw the scene, then record the upscale into your command list.

// Camera matrices and this frame's jitter offset
host->SetFrameConstants(&constants);

// Draws color, depth and motion vectors at the render size,
// using the jittered projection
DrawScene(list);

// The upscaler reads its inputs as shader resources and writes its
// output through unordered access, so move each one into that state.
// The host adds no barriers of its own.
D3D12_RESOURCE_BARRIER barriers[] = {
    Transition(_color.Get(), D3D12_RESOURCE_STATE_RENDER_TARGET,
               D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE),
    Transition(_depth.Get(), D3D12_RESOURCE_STATE_DEPTH_WRITE,
               D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE),
    Transition(_motion.Get(), D3D12_RESOURCE_STATE_RENDER_TARGET,
               D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE),
    Transition(_upscaled.Get(), D3D12_RESOURCE_STATE_PIXEL_SHADER_RESOURCE,
               D3D12_RESOURCE_STATE_UNORDERED_ACCESS),
};
list->ResourceBarrier(4, barriers);

MGDFUpscaleDesc upscale = {};
upscale.Mode = MGDF_UPSCALE_MODE_QUALITY;
upscale.RenderWidth = _renderWidth;
upscale.RenderHeight = _renderHeight;
upscale.OutputWidth = _outputWidth;
upscale.OutputHeight = _outputHeight;
upscale.ColorHDR = TRUE;   // Scene color is linear HDR, not yet tone mapped
// Each input pairs a resource with the state the barriers left it in
upscale.ColorIn = {_color.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
upscale.Depth = {_depth.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
upscale.MotionVectors = {_motion.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
upscale.ColorOut = {_upscaled.Get(), D3D12_RESOURCE_STATE_UNORDERED_ACCESS};
if (FAILED(host->Upscale(list, &upscale))) return FALSE;

// Upscale leaves the list's bindings undefined, so set them again
list->SetDescriptorHeaps(1, _heaps.AsArray());

// Reads _upscaled and draws at the output size
DrawEffectsAndUI(list);

Transition is a helper of your own that fills in a transition barrier. The states you move from depend on how your scene pass left each target.

PitfallUpscale leaves the command list's descriptor heaps, root signature, pipeline state, render targets and viewports undefined. Set them again before you draw anything else.

See MGDFFrameConstants and MGDFUpscaleDesc for every field.

Add frame generation

Frame generation inserts extra frames between the ones you render. It needs the same depth and motion vectors as the upscaler. Hand them over before RTDraw returns.

MGDFFrameGenerationDesc generation = {};
generation.Depth = {_depth.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
generation.MotionVectors = {_motion.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
// The finished frame before the UI is drawn
generation.HudlessColor = {_hudless.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
// The UI on its own, with alpha, so generated frames keep it steady
generation.UIColorAndAlpha = {_ui.Get(), D3D12_RESOURCE_STATE_NON_PIXEL_SHADER_RESOURCE};
host->SetFrameGenerationInputs(&generation);
InputRequiredWhat it is
DepthYesScene depth.
MotionVectorsYesPer-pixel motion.
HudlessColorNoThe final image without UI.
UIColorAndAlphaNoThe UI alone, with premultiplied alpha.

The two optional inputs are worth the effort. Without them, generated frames smear the UI.

Generated frames look wrong when the picture jumps. While the game loads, is paused, or cuts to a new scene, pass nullptr and the host skips generation for that frame:

// A null description turns generation off for this frame
if (_loading || _paused) host->SetFrameGenerationInputs(nullptr);

PitfallFrame generation doesn't work with an scRGB back buffer. If you want both HDR and frame generation, choose HDR10.

Measure and cut latency

Low latency mode shortens the delay between the player's input and the frame that shows it. The host measures that delay from markers you report, so mark four points in RTDraw:

MarkerReport it
MGDF_FRAME_MARKER_SIMULATION_STARTBefore you read the simulation state.
MGDF_FRAME_MARKER_SIMULATION_ENDOnce you have it.
MGDF_FRAME_MARKER_RENDER_SUBMIT_STARTBefore you record the first command list.
MGDF_FRAME_MARKER_RENDER_SUBMIT_ENDAfter the last ExecuteCommandLists.
// Marks where this frame's simulation starts and ends, for latency measurement
host->SetFrameMarker(MGDF_FRAME_MARKER_SIMULATION_START);
Snapshot frame = TakeSnapshot();
host->SetFrameMarker(MGDF_FRAME_MARKER_SIMULATION_END);

Report MGDF_FRAME_MARKER_INPUT_SAMPLE as well, at the moment you read controller input. The markers do nothing on a device without latency support, so you can call them unconditionally.

Let the player choose

Frame generation, low latency, ray tracing and HDR are player settings. Change them through a pending change, which applies when you release it.

MGDF::ComObject<IMGDFPendingRenderSettingsChange> change;
settings->CreatePendingSettingsChange(change.Assign());
change->SetFrameGeneration(1);
change->SetLowLatencyMode(MGDF_LOW_LATENCY_ON);
// Releasing the change applies both settings together
change.Clear();

The host saves each choice as a preference and restores it on the next run.

SettingChangeReadSaved as
Frame generationSetFrameGenerationGetFrameGenerationhost.frameGeneration
Low latencySetLowLatencyModeGetCurrentLowLatencyModehost.lowLatencyMode
Ray tracingSetRayTracingGetRayTracinghost.rayTracing
HDRSetHDREnabledGetHDREnabledhost.hdrEnabled

The first three setters return FALSE if the device can't do what you asked. Changing frame generation rebuilds the swap chain, so expect a back buffer change.

Check for ray tracing

If your renderer uses DXR, ask the host whether to turn it on. The answer combines what the device supports with what the player chose.

MGDFRayTracingSupport support = {};
// Supported by the device, and switched on by the player
_rayTracing = settings->GetRayTracingSupport(&support) && settings->GetRayTracing();
_reorder = _rayTracing && support.ShaderExecutionReordering;

Support means DXR tier 1.1 and shader model 6.9. ShaderExecutionReordering tells you whether the device reorders shader execution as well.

The host does nothing else with this setting. Your module reads it and picks its own render path. When more than one adapter meets your requirements, the host prefers one that supports ray tracing.

Write HDR output

The host switches to HDR when the player turns it on and the display supports it. You choose the format ahead of time, with HDRFormat in GetGraphicsRequirements.

scRGB (default)HDR10
HDRFormatMGDF_HDR_FORMAT_SCRGBMGDF_HDR_FORMAT_HDR10
Back buffer16-bit float10-bit UNORM
You writeLinear Rec.709, where 1.0 is 80 nitsPQ-encoded Rec.2020, where 1.0 is 10,000 nits
Frame generationNoYes
EffortNone. Write linear color.You encode the final image.

To find out whether HDR is on right now, compare the back buffer format with the HDR format in RTBackBufferChange:

MGDFBackBufferInfo info = {};
host->GetBackBufferInfo(&info);
// The back buffer only takes the HDR format while HDR output is active
_hdr = info.Format == settings->GetHDRBackBufferFormat();

Encode for HDR10

With HDR10 the back buffer holds encoded values, so do the encoding in your last pass:

  1. Composite the scene and UI in linear light.
  2. Tonemap to the display's luminance range.
  3. Convert from Rec.709 to Rec.2020 primaries.
  4. Apply the PQ curve.
// HLSL. nits is linear Rec.2020 color in nits.
float3 EncodePQ(float3 nits) {
  const float m1 = 0.1593017578125f, m2 = 78.84375f;
  const float c1 = 0.8359375f, c2 = 18.8515625f, c3 = 18.6875f;
  const float3 y = pow(saturate(nits / 10000.0f), m1);
  return pow((c1 + c2 * y) / (1.0f + c3 * y), m2);
}

PitfallDon't alpha-blend the UI straight into an HDR10 back buffer. That blends encoded values and the result is wrong. Blend in linear light, then encode.

Describe your content to the display

HDR10 displays tonemap better when they know how bright your content gets. Get the display's range from the host and send back the peak you tonemap to.

MGDFOutputDisplayInfo display = {};
settings->GetCurrentOutputDisplayInfo(&display);
// Some displays report no peak brightness. 1000 nits is a safe guess.
const float peak = display.MaxLuminance > 0.0f ? display.MaxLuminance : 1000.0f;

// Describes how bright your image gets, so the display can map it
MGDFHDRMetadata metadata = {};
metadata.MaxContentLightLevel = peak;
metadata.MaxFrameAverageLightLevel = (std::min)(display.MaxFullFrameLuminance, peak);
metadata.MaxMasteringLuminance = peak;
metadata.MinMasteringLuminance = display.MinLuminance;
host->SetHDRMetadata(&metadata);

The host keeps the metadata across resizes. See MGDFHDRMetadata for the valid ranges.

In VR

DLSS upscales each eye separately in VR. Frame generation and low latency switch off while the headset is active and come back when VR ends. See upscaling in VR.

Next steps