Major overhaul of Renderer abstraction, to support a new example (#624)

The original goal here was to bring up a second example program: `model-viewer`.
While the existing `hello-world` example is enough to get somebody up to speed with the basics of the Slang API (as a drop-in replacement for `D3DCompile` or similar), it doesn't really show any of the big-picture stuff that Slang is meant to enable.
There wasn't any use of D3D12/Vulkan descriptor tables/sets, and there wasn't any use of interfaces, generics, or `ParameterBlock`s in the shader code.

The `model-viewer` example addresses these issues. Its shader code involves generics, interfaces, and multiple `ParameterBlock`s, and the host-side code demonstrates a few key things for working with Slang:

* There is an application-level abstraction for parameter blocks, that combines the graphics-API descriptor set object with Slang type information

* There is a shader cache layer used to look up an appropriate variant of a rendering effect by using parameter block types to "plug in" global type variables

* There is a clear separation between the phases of compilation: a first phase that does semantic checking and enables reflection-based allocation of graphics API objects, followed by one or more code generation passes for specialized kernels.

This example is certainly not perfect, and it will need to be revamped more going forward. In particular:

* The output picture is ugly as sin. We need a plan for how to get this to load better content, perhaps even popping up an error message to note that the required input data isn't present in the basic repository.

* The shader code is too simplistic. There isn't any real material variety, and the `IMaterial` abstraction is completely wrong.

* The use of parameter blocks is facile because there are no resource parameters right now. Fixing that will likely expose issues around interfacing with Slang's reflection API.

* The whole example exposes the issue that Slang's current APIs aren't really designed for the benefit of two-phase compilation (since our many client application has been stuck on one-phase compilation).

* Global type parameters are actually a Bad Idea that we only did for compatibility with existing codebases. We should not be showing them off in an example of the Right Way to use Slang, but the language support for type parameters on entry points is still not complete.

Of course, the majority of the changes here are *not* inside the example applications, and instead involve a major overhaul of the `Renderer` abstraction that is used for both tests and examples. The main thrust of the change is to make the abstraction layer be closer to the D3D12/Vulkan model than to a D3D11-style model. This is important for the `model-viewer` example, since it aspires to show how Slang can be incorporated into a renderer that targets a modern API. The most important bit is actually the use of descriptor sets and "pipeline layouts" a la Vulkan, since without these Slang's `ParameterBlock` abstraction won't make a lot of sense.

Implementation of the abstraction for the various APIs has very much been on an as-needed basis. The current implementation is just enough for the two examples to work, plus enough to get all the tests to pass in both debug and release builds on Windows.

A big missing feature in the API abstraction right now is memory lifetime management. The code had been trending toward something D3D11-like where a constant buffer could be mapped per-frame with the implementation doing behind-the-scenes allocation for targets like D3D12/Vulkan. I'd like to shift more toward a model of just exposing "transient" allocations that are only valid for one frame, because these are more representation of how an efficient renderer for next-generation APIs will work. That transition isn't actually complete, though, so there are problems with the existing examples where `hello-world` is actually scribbling into memory that the GPU might still be using, while `model-viewer` is doing full-on heavy-weight allocations on a per-frame basis with no real concern for the performance implications.

All together, there are a lot of things here that need more work, but this branch has been way too long-lived already, and so I'd like to get this checked in as long as all the tests pass.

1 files changed, 306 insertions, 0 deletions

diff --git a/tools/gfx/d3d-util.cpp b/tools/gfx/d3d-util.cpp
new file mode 100644
index 000000000..19135707b
--- /dev/null
+++ b/tools/gfx/d3d-util.cpp
@@ -0,0 +1,306 @@
+// d3d-util.cpp
+#include "d3d-util.h"
+
+#include <d3dcompiler.h>
+
+// We will use the C standard library just for printing error messages.
+#include <stdio.h>
+
+namespace gfx {
+using namespace Slang;
+
+/* static */D3D_PRIMITIVE_TOPOLOGY D3DUtil::getPrimitiveTopology(PrimitiveTopology topology)
+{
+    switch (topology)
+    {
+        case PrimitiveTopology::TriangleList:
+        {
+            return D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST;
+        }
+        default: break;
+    }
+    return D3D11_PRIMITIVE_TOPOLOGY_UNDEFINED;
+}
+
+/* static */DXGI_FORMAT D3DUtil::getMapFormat(Format format)
+{
+    switch (format)
+    {
+        case Format::RGBA_Float32:          return DXGI_FORMAT_R32G32B32A32_FLOAT;
+        case Format::RGB_Float32:           return DXGI_FORMAT_R32G32B32_FLOAT;
+        case Format::RG_Float32:            return DXGI_FORMAT_R32G32_FLOAT;
+        case Format::R_Float32:             return DXGI_FORMAT_R32_FLOAT;
+        case Format::RGBA_Unorm_UInt8:      return DXGI_FORMAT_R8G8B8A8_UNORM;
+        case Format::R_UInt32:              return DXGI_FORMAT_R32_UINT;
+
+        case Format::D_Float32:             return DXGI_FORMAT_D32_FLOAT;
+        case Format::D_Unorm24_S8:          return DXGI_FORMAT_D24_UNORM_S8_UINT;
+
+        default:                            return DXGI_FORMAT_UNKNOWN;
+    }
+}
+
+/* static */DXGI_FORMAT D3DUtil::calcResourceFormat(UsageType usage, Int usageFlags, DXGI_FORMAT format)
+{
+    SLANG_UNUSED(usage);
+    if (usageFlags)
+    {
+        switch (format)
+        {
+            case DXGI_FORMAT_R32_FLOAT:     /* fallthru */
+            case DXGI_FORMAT_R32_UINT:
+            case DXGI_FORMAT_D32_FLOAT:
+            {
+                return DXGI_FORMAT_R32_TYPELESS;
+            }
+            case DXGI_FORMAT_D24_UNORM_S8_UINT:    return DXGI_FORMAT_R24G8_TYPELESS;
+            default: break;
+        }
+        return format;
+    }
+    return format;
+}
+
+/* static */DXGI_FORMAT D3DUtil::calcFormat(UsageType usage, DXGI_FORMAT format)
+{
+    switch (usage)
+    {
+        case USAGE_COUNT_OF:
+        case USAGE_UNKNOWN:
+        {
+            return DXGI_FORMAT_UNKNOWN;
+        }
+        case USAGE_DEPTH_STENCIL:
+        {
+            switch (format)
+            {
+                case DXGI_FORMAT_D32_FLOAT:     /* fallthru */
+                case DXGI_FORMAT_R32_TYPELESS:
+                {
+                    return DXGI_FORMAT_D32_FLOAT;
+                }
+                case DXGI_FORMAT_R24_UNORM_X8_TYPELESS:    return DXGI_FORMAT_D24_UNORM_S8_UINT;
+                case DXGI_FORMAT_R24G8_TYPELESS:        return DXGI_FORMAT_D24_UNORM_S8_UINT;
+                default: break;
+            }
+            return format;
+        }
+        case USAGE_TARGET:
+        {
+            switch (format)
+            {
+                case DXGI_FORMAT_D32_FLOAT:     /* fallthru */
+                case DXGI_FORMAT_D24_UNORM_S8_UINT:
+                {
+                    return DXGI_FORMAT_UNKNOWN;
+                }
+                case DXGI_FORMAT_R32_TYPELESS:        return DXGI_FORMAT_R32_FLOAT;
+                default: break;
+            }
+            return format;
+        }
+        case USAGE_SRV:
+        {
+            switch (format)
+            {
+                case DXGI_FORMAT_D32_FLOAT:     /* fallthru */
+                case DXGI_FORMAT_R32_TYPELESS:
+                {
+                    return DXGI_FORMAT_R32_FLOAT;
+                }
+                case DXGI_FORMAT_R24_UNORM_X8_TYPELESS:    return DXGI_FORMAT_R24_UNORM_X8_TYPELESS;
+                default: break;
+            }
+
+            return format;
+        }
+    }
+
+    assert(!"Not reachable");
+    return DXGI_FORMAT_UNKNOWN;
+}
+
+bool D3DUtil::isTypeless(DXGI_FORMAT format)
+{
+    switch (format)
+    {
+        case DXGI_FORMAT_R32G32B32A32_TYPELESS:
+        case DXGI_FORMAT_R32G32B32_TYPELESS:
+        case DXGI_FORMAT_R16G16B16A16_TYPELESS:
+        case DXGI_FORMAT_R32G32_TYPELESS:
+        case DXGI_FORMAT_R32G8X24_TYPELESS:
+        case DXGI_FORMAT_R32_FLOAT_X8X24_TYPELESS:
+        case DXGI_FORMAT_R10G10B10A2_TYPELESS:
+        case DXGI_FORMAT_R8G8B8A8_TYPELESS:
+        case DXGI_FORMAT_R16G16_TYPELESS:
+        case DXGI_FORMAT_R32_TYPELESS:
+        case DXGI_FORMAT_R24_UNORM_X8_TYPELESS:
+        case DXGI_FORMAT_R24G8_TYPELESS:
+        case DXGI_FORMAT_R8G8_TYPELESS:
+        case DXGI_FORMAT_R16_TYPELESS:
+        case DXGI_FORMAT_R8_TYPELESS:
+        case DXGI_FORMAT_BC1_TYPELESS:
+        case DXGI_FORMAT_BC2_TYPELESS:
+        case DXGI_FORMAT_BC3_TYPELESS:
+        case DXGI_FORMAT_BC4_TYPELESS:
+        case DXGI_FORMAT_BC5_TYPELESS:
+        case DXGI_FORMAT_B8G8R8A8_TYPELESS:
+        case DXGI_FORMAT_BC6H_TYPELESS:
+        case DXGI_FORMAT_BC7_TYPELESS:
+        {
+            return true;
+        }
+        default: break;
+    }
+    return false;
+}
+
+/* static */Int D3DUtil::getNumColorChannelBits(DXGI_FORMAT fmt)
+{
+    switch (fmt)
+    {
+        case DXGI_FORMAT_R32G32B32A32_TYPELESS:
+        case DXGI_FORMAT_R32G32B32A32_FLOAT:
+        case DXGI_FORMAT_R32G32B32A32_UINT:
+        case DXGI_FORMAT_R32G32B32A32_SINT:
+        case DXGI_FORMAT_R32G32B32_TYPELESS:
+        case DXGI_FORMAT_R32G32B32_FLOAT:
+        case DXGI_FORMAT_R32G32B32_UINT:
+        case DXGI_FORMAT_R32G32B32_SINT:
+        {
+            return 32;
+        }
+        case DXGI_FORMAT_R16G16B16A16_TYPELESS:
+        case DXGI_FORMAT_R16G16B16A16_FLOAT:
+        case DXGI_FORMAT_R16G16B16A16_UNORM:
+        case DXGI_FORMAT_R16G16B16A16_UINT:
+        case DXGI_FORMAT_R16G16B16A16_SNORM:
+        case DXGI_FORMAT_R16G16B16A16_SINT:
+        {
+            return 16;
+        }
+        case DXGI_FORMAT_R10G10B10A2_TYPELESS:
+        case DXGI_FORMAT_R10G10B10A2_UNORM:
+        case DXGI_FORMAT_R10G10B10A2_UINT:
+        case DXGI_FORMAT_R10G10B10_XR_BIAS_A2_UNORM:
+        {
+            return 10;
+        }
+        case DXGI_FORMAT_R8G8B8A8_TYPELESS:
+        case DXGI_FORMAT_R8G8B8A8_UNORM:
+        case DXGI_FORMAT_R8G8B8A8_UNORM_SRGB:
+        case DXGI_FORMAT_R8G8B8A8_UINT:
+        case DXGI_FORMAT_R8G8B8A8_SNORM:
+        case DXGI_FORMAT_R8G8B8A8_SINT:
+        case DXGI_FORMAT_B8G8R8A8_UNORM:
+        case DXGI_FORMAT_B8G8R8X8_UNORM:
+        case DXGI_FORMAT_B8G8R8A8_TYPELESS:
+        case DXGI_FORMAT_B8G8R8A8_UNORM_SRGB:
+        case DXGI_FORMAT_B8G8R8X8_TYPELESS:
+        case DXGI_FORMAT_B8G8R8X8_UNORM_SRGB:
+        {
+            return 8;
+        }
+        case DXGI_FORMAT_B5G6R5_UNORM:
+        case DXGI_FORMAT_B5G5R5A1_UNORM:
+        {
+            return 5;
+        }
+        case DXGI_FORMAT_B4G4R4A4_UNORM:
+            return 4;
+
+        default:
+            return 0;
+    }
+}
+
+// Note: this subroutine is now only used by D3D11 for generating bytecode to go into input layouts.
+//
+// TODO: we can probably remove that code completely by switching to a PSO-like model across all APIs.
+//
+/* static */Result D3DUtil::compileHLSLShader(char const* sourcePath, char const* source, char const* entryPointName, char const* dxProfileName, ComPtr<ID3DBlob>& shaderBlobOut)
+{
+    // Rather than statically link against the `d3dcompile` library, we
+    // dynamically load it.
+    //
+    // Note: A more realistic application would compile from HLSL text to D3D
+    // shader bytecode as part of an offline process, rather than doing it
+    // on-the-fly like this
+    //
+    static pD3DCompile compileFunc = nullptr;
+    if (!compileFunc)
+    {
+        // TODO(tfoley): maybe want to search for one of a few versions of the DLL
+        HMODULE compilerModule = LoadLibraryA("d3dcompiler_47.dll");
+        if (!compilerModule)
+        {
+            fprintf(stderr, "error: failed load 'd3dcompiler_47.dll'\n");
+            return SLANG_FAIL;
+        }
+
+        compileFunc = (pD3DCompile)GetProcAddress(compilerModule, "D3DCompile");
+        if (!compileFunc)
+        {
+            fprintf(stderr, "error: failed load symbol 'D3DCompile'\n");
+            return SLANG_FAIL;
+        }
+    }
+
+    // For this example, we turn on debug output, and turn off all
+    // optimization. A real application would only use these flags
+    // when shader debugging is needed.
+    UINT flags = 0;
+    flags |= D3DCOMPILE_DEBUG;
+    flags |= D3DCOMPILE_OPTIMIZATION_LEVEL0 | D3DCOMPILE_SKIP_OPTIMIZATION;
+
+    // We will always define `__HLSL__` when compiling here, so that
+    // input code can react differently to being compiled as pure HLSL.
+    D3D_SHADER_MACRO defines[] = {
+        { "__HLSL__", "1" },
+        { nullptr, nullptr },
+    };
+
+    // The `D3DCompile` entry point takes a bunch of parameters, but we
+    // don't really need most of them for Slang-generated code.
+    ComPtr<ID3DBlob> shaderBlob;
+    ComPtr<ID3DBlob> errorBlob;
+
+    HRESULT hr = compileFunc(source, strlen(source), sourcePath, &defines[0], nullptr, entryPointName, dxProfileName, flags, 0,
+        shaderBlob.writeRef(), errorBlob.writeRef());
+
+    // If the HLSL-to-bytecode compilation produced any diagnostic messages
+    // then we will print them out (whether or not the compilation failed).
+    if (errorBlob)
+    {
+        ::fputs((const char*)errorBlob->GetBufferPointer(), stderr);
+        ::fflush(stderr);
+        ::OutputDebugStringA((const char*)errorBlob->GetBufferPointer());
+    }
+
+    SLANG_RETURN_ON_FAIL(hr);
+    shaderBlobOut.swap(shaderBlob);
+    return SLANG_OK;
+}
+
+/* static */void D3DUtil::appendWideChars(const char* in, List<wchar_t>& out)
+{
+    size_t len = ::strlen(in);
+
+    const DWORD dwFlags = 0;
+    int outSize = ::MultiByteToWideChar(CP_UTF8, dwFlags, in, int(len), nullptr, 0);
+
+    if (outSize > 0)
+    {
+        const UInt prevSize = out.Count();
+        out.SetSize(prevSize + len + 1);
+
+        WCHAR* dst = out.Buffer() + prevSize;
+        ::MultiByteToWideChar(CP_UTF8, dwFlags, in, int(len), dst, outSize);
+        // Make null terminated
+        dst[outSize] = 0;
+        // Remove terminating 0 from array
+        out.UnsafeShrinkToSize(prevSize + outSize);
+    }
+}
+
+} // renderer_test