Add support for unbounded arrays as shader parameters (#725)

* Add support for unbounded arrays as shader parameters

With this change, Slang shaders can use unbounded-size arrays as parameters, e.g.:

```hlsl
Texture2D t[] : register(t3, space2);
SamplerState s[];
```

As shown in the above example, Slang supports both explicit `register` declarations on unbounded-size arrays and also implicit binding.
When doing automatic parmaeter binding, Slang will allocate a full register space to an unbounded-size array of textures/smaplers, starting at register zero.

Note that for the Vulkan target, an array of descriptors of any size (including unbounded size) consumes only a single `bindign`, so much of this logic is specific to D3D targets.

Details on the changes made:

* The single biggest change is a new `LayoutSize` type that is used to store a value that can either be a finite unsigned integer or a dedicated "infinite" value (which is stored as the all-bits-set `-1` value). This is used in places where a size could either be a finite value or an "unbounded" value, to both try to make standard math robust against the infinite case, and also to force code to deal with both the finite and infinite cases more explicitly when they care about the difference.

* The public API was documented so that unbounded-size arrays report their size as `-1`. We should probably change this function to return a signed value instead of `size_t`, but that would technically be a source-breaking change, so we want to make sure we stage it appropriately.

* The code that invokes fxc was updated so that it passes the appropriate flag to enable unbounded arrays of descriptors. I haven't looked yet at whether dxc needs such a flag, so there may need to be a follow-on change to add that.

* The logic in the `UsedRanges::Add` method for tracking what registers have been claimed was rewritten because the previous version had some subtle bugs. The new version includes more detailed comments that attempt to explain why I think the new logic works.

* The top-level logic for auto-assigning bindings to parameters has been overhauled to deal with the fact that a parameter that needs "infinite" amounts of a resource should be claiming a full register space for those resources instead. Whenever a parameter allocates any register spaces we want them all to be contiguous, so we have a loop that counts the requirements and allocates the spaces before we go along and dole them out.

* When computing the layout for an array type, we need to carefully deal with unbounded-size arrays. In the case of an unbounded array of a "simple" resource type (e.g., `Texture2D[]`), we opt to expose the type layout as consuming an infinite number of the appropriate register, while in the case of a complex type (say, a `struct` with two texture fields), we need to instead allocate whole spaces for those fields. The logic here is more subtle than I would like, and interacts with the existing code that "adjusts" the element type of an array in order to make standard indexing math Just Work.

* Similarly, when a `struct` type has unbounded-array fields, then we need to transform any field with infinite register requirements to instead consume a space in the resulting aggregate type. This case is comparatively easier than the array case.

* The test case for unbounded arrays covers both explicit and implicit bindings, and also the case of an unbounded array over a `struct` type (it does not cover the case of a `struct` contianing unbounded arrays, so that will need to be added later). For this test we are both validation the output reflection data and that we produce the same code as fxc (with explicit bindings in the fxc case).

* The reflection test app was modified to use the new API contract and detect when a parameter consumes `SLANG_UNBOUNDED_SIZE` resources.

* Fixup: ensure unbounded size is defined at right bit width

1 files changed, 15 insertions, 4 deletions

diff --git a/source/slang/reflection.cpp b/source/slang/reflection.cpp
index 6e635a8ce..73b616377 100644
--- a/source/slang/reflection.cpp
+++ b/source/slang/reflection.cpp
@@ -513,6 +513,17 @@ SLANG_API SlangReflectionType* spReflectionTypeLayout_GetType(SlangReflectionTyp
     return (SlangReflectionType*) typeLayout->type.Ptr();
 }
 
+namespace
+{
+    static size_t getReflectionSize(LayoutSize size)
+    {
+        if(size.isFinite())
+            return size.getFiniteValue();
+
+        return SLANG_UNBOUNDED_SIZE;
+    }
+}
+
 SLANG_API size_t spReflectionTypeLayout_GetSize(SlangReflectionTypeLayout* inTypeLayout, SlangParameterCategory category)
 {
     auto typeLayout = convert(inTypeLayout);
@@ -521,7 +532,7 @@ SLANG_API size_t spReflectionTypeLayout_GetSize(SlangReflectionTypeLayout* inTyp
     auto info = typeLayout->FindResourceInfo(LayoutResourceKind(category));
     if(!info) return 0;
 
-    return info->count;
+    return getReflectionSize(info->count);
 }
 
 SLANG_API SlangReflectionVariableLayout* spReflectionTypeLayout_GetFieldByIndex(SlangReflectionTypeLayout* inTypeLayout, unsigned index)
@@ -558,10 +569,10 @@ SLANG_API size_t spReflectionTypeLayout_GetElementStride(SlangReflectionTypeLayo
                 auto elementTypeLayout = arrayTypeLayout->elementTypeLayout;
                 auto info = elementTypeLayout->FindResourceInfo(LayoutResourceKind(category));
                 if(!info) return 0;
-                return info->count;
+                return getReflectionSize(info->count);
             }
 
-        // An import special case, though, is Vulkan descriptor-table slots,
+        // An important special case, though, is Vulkan descriptor-table slots,
         // where an entire array will use a single `binding`, so that the
         // effective stride is zero:
         case SLANG_PARAMETER_CATEGORY_DESCRIPTOR_TABLE_SLOT:
@@ -1210,5 +1221,5 @@ SLANG_API size_t spReflection_getGlobalConstantBufferSize(SlangReflection* inPro
     auto structLayout = getGlobalStructLayout(program);
     auto uniform = structLayout->FindResourceInfo(LayoutResourceKind::Uniform);
     if (!uniform) return 0;
-    return uniform->count;
+    return getReflectionSize(uniform->count);
 }