Add support for unbounded arrays as shader parameters (#725)

* Add support for unbounded arrays as shader parameters

With this change, Slang shaders can use unbounded-size arrays as parameters, e.g.:

```hlsl
Texture2D t[] : register(t3, space2);
SamplerState s[];
```

As shown in the above example, Slang supports both explicit `register` declarations on unbounded-size arrays and also implicit binding.
When doing automatic parmaeter binding, Slang will allocate a full register space to an unbounded-size array of textures/smaplers, starting at register zero.

Note that for the Vulkan target, an array of descriptors of any size (including unbounded size) consumes only a single `bindign`, so much of this logic is specific to D3D targets.

Details on the changes made:

* The single biggest change is a new `LayoutSize` type that is used to store a value that can either be a finite unsigned integer or a dedicated "infinite" value (which is stored as the all-bits-set `-1` value). This is used in places where a size could either be a finite value or an "unbounded" value, to both try to make standard math robust against the infinite case, and also to force code to deal with both the finite and infinite cases more explicitly when they care about the difference.

* The public API was documented so that unbounded-size arrays report their size as `-1`. We should probably change this function to return a signed value instead of `size_t`, but that would technically be a source-breaking change, so we want to make sure we stage it appropriately.

* The code that invokes fxc was updated so that it passes the appropriate flag to enable unbounded arrays of descriptors. I haven't looked yet at whether dxc needs such a flag, so there may need to be a follow-on change to add that.

* The logic in the `UsedRanges::Add` method for tracking what registers have been claimed was rewritten because the previous version had some subtle bugs. The new version includes more detailed comments that attempt to explain why I think the new logic works.

* The top-level logic for auto-assigning bindings to parameters has been overhauled to deal with the fact that a parameter that needs "infinite" amounts of a resource should be claiming a full register space for those resources instead. Whenever a parameter allocates any register spaces we want them all to be contiguous, so we have a loop that counts the requirements and allocates the spaces before we go along and dole them out.

* When computing the layout for an array type, we need to carefully deal with unbounded-size arrays. In the case of an unbounded array of a "simple" resource type (e.g., `Texture2D[]`), we opt to expose the type layout as consuming an infinite number of the appropriate register, while in the case of a complex type (say, a `struct` with two texture fields), we need to instead allocate whole spaces for those fields. The logic here is more subtle than I would like, and interacts with the existing code that "adjusts" the element type of an array in order to make standard indexing math Just Work.

* Similarly, when a `struct` type has unbounded-array fields, then we need to transform any field with infinite register requirements to instead consume a space in the resulting aggregate type. This case is comparatively easier than the array case.

* The test case for unbounded arrays covers both explicit and implicit bindings, and also the case of an unbounded array over a `struct` type (it does not cover the case of a `struct` contianing unbounded arrays, so that will need to be added later). For this test we are both validation the output reflection data and that we produce the same code as fxc (with explicit bindings in the fxc case).

* The reflection test app was modified to use the new API contract and detect when a parameter consumes `SLANG_UNBOUNDED_SIZE` resources.

* Fixup: ensure unbounded size is defined at right bit width

1 files changed, 155 insertions, 15 deletions

diff --git a/source/slang/type-layout.h b/source/slang/type-layout.h
index 7f662bdee..32ee41784 100644
--- a/source/slang/type-layout.h
+++ b/source/slang/type-layout.h
@@ -36,12 +36,152 @@ enum class LayoutRulesFamily
 };
 #endif
 
+// A "size" that can either be a simple finite size or
+// the special case of an infinite/unbounded size.
+//
+struct LayoutSize
+{
+    typedef size_t RawValue;
+
+    LayoutSize()
+        : raw(0)
+    {}
+
+    LayoutSize(RawValue size)
+        : raw(size)
+    {
+        SLANG_ASSERT(size != RawValue(-1));
+    }
+
+    static LayoutSize infinite()
+    {
+        LayoutSize result;
+        result.raw = RawValue(-1);
+        return result;
+    }
+
+    bool isInfinite() const { return raw == RawValue(-1); }
+
+    bool isFinite() const { return raw != RawValue(-1); }
+    RawValue getFiniteValue() const { SLANG_ASSERT(isFinite()); return raw; }
+
+    bool operator==(LayoutSize that) const
+    {
+        return raw == that.raw;
+    }
+
+    bool operator!=(LayoutSize that) const
+    {
+        return raw != that.raw;
+    }
+
+    void operator+=(LayoutSize right)
+    {
+        if( isInfinite() ) {}
+        else if( right.isInfinite() )
+        {
+            *this = LayoutSize::infinite();
+        }
+        else
+        {
+            *this = LayoutSize(raw + right.raw);
+        }
+    }
+
+    void operator*=(LayoutSize right)
+    {
+        // Deal with zero first, so that anything (even the "infinite" value) times zero is zero.
+        if( raw == 0 )
+        {
+            return;
+        }
+
+        if( right.raw == 0 )
+        {
+            raw = 0;
+            return;
+        }
+
+        // Next we deal with infinite cases, so that infinite times anything non-zero is infinite
+        if( isInfinite() )
+        {
+            return;
+        }
+
+        if( right.isInfinite() )
+        {
+            *this = LayoutSize::infinite();
+            return;
+        }
+
+        // Finally deal with the case where both sides are finite
+        *this = LayoutSize(raw * right.raw);
+    }
+
+    void operator-=(RawValue right)
+    {
+        if( isInfinite() ) {}
+        else
+        {
+            *this = LayoutSize(raw - right);
+        }
+    }
+
+    void operator/=(RawValue right)
+    {
+        if( isInfinite() ) {}
+        else
+        {
+            *this = LayoutSize(raw / right);
+        }
+    }
+    RawValue raw;
+};
+
+inline LayoutSize operator+(LayoutSize left, LayoutSize right)
+{
+    LayoutSize result(left);
+    result += right;
+    return result;
+}
+
+inline LayoutSize operator*(LayoutSize left, LayoutSize right)
+{
+    LayoutSize result(left);
+    result *= right;
+    return result;
+}
+
+inline LayoutSize operator-(LayoutSize left, LayoutSize::RawValue right)
+{
+    LayoutSize result(left);
+    result -= right;
+    return result;
+}
+
+inline LayoutSize operator/(LayoutSize left, LayoutSize::RawValue right)
+{
+    LayoutSize result(left);
+    result /= right;
+    return result;
+}
+
+inline LayoutSize maximum(LayoutSize left, LayoutSize right)
+{
+    if(left.isInfinite() || right.isInfinite())
+        return LayoutSize::infinite();
+
+    return LayoutSize(Math::Max(
+            left.getFiniteValue(),
+            right.getFiniteValue()));
+}
+
 // Layout appropriate to "just memory" scenarios,
 // such as laying out the members of a constant buffer.
 struct UniformLayoutInfo
 {
-    size_t size;
-    size_t alignment;
+    LayoutSize  size;
+    size_t      alignment;
 
     UniformLayoutInfo()
         : size(0)
@@ -49,8 +189,8 @@ struct UniformLayoutInfo
     {}
 
     UniformLayoutInfo(
-        size_t size,
-        size_t alignment)
+        LayoutSize  size,
+        size_t      alignment)
         : size(size)
         , alignment(alignment)
     {}
@@ -68,9 +208,9 @@ struct UniformArrayLayoutInfo : UniformLayoutInfo
     {}
 
     UniformArrayLayoutInfo(
-        size_t size,
-        size_t alignment,
-        size_t elementStride)
+        LayoutSize  size,
+        size_t      alignment,
+        size_t      elementStride)
         : UniformLayoutInfo(size, alignment)
         , elementStride(elementStride)
     {}
@@ -86,7 +226,7 @@ struct SimpleLayoutInfo
     LayoutResourceKind kind;
 
     // How many resources of that kind?
-    size_t size;
+    LayoutSize size;
 
     // only useful in the uniform case
     size_t alignment;
@@ -104,7 +244,7 @@ struct SimpleLayoutInfo
         , alignment(uniformInfo.alignment)
     {}
 
-    SimpleLayoutInfo(LayoutResourceKind kind, size_t size, size_t alignment=1)
+    SimpleLayoutInfo(LayoutResourceKind kind, LayoutSize size, size_t alignment=1)
         : kind(kind)
         , size(size)
         , alignment(alignment)
@@ -168,7 +308,7 @@ public:
         LayoutResourceKind  kind = LayoutResourceKind::None;
 
         // How many registers of the above kind did we use?
-        UInt                 count;
+        LayoutSize          count;
     };
 
     List<ResourceInfo>      resourceInfos;
@@ -207,7 +347,7 @@ public:
         findOrAddResourceInfo(info.kind)->count += info.count;
     }
 
-    void addResourceUsage(LayoutResourceKind kind, UInt count)
+    void addResourceUsage(LayoutResourceKind kind, LayoutSize count)
     {
         ResourceInfo info;
         info.kind = kind;
@@ -507,7 +647,7 @@ struct SimpleLayoutRulesImpl
     virtual SimpleLayoutInfo GetScalarLayout(BaseType baseType) = 0;
 
     // Get size and alignment for an array of elements
-    virtual SimpleArrayLayoutInfo GetArrayLayout(SimpleLayoutInfo elementInfo, size_t elementCount) = 0;
+    virtual SimpleArrayLayoutInfo GetArrayLayout(SimpleLayoutInfo elementInfo, LayoutSize elementCount) = 0;
 
     // Get layout for a vector or matrix type
     virtual SimpleLayoutInfo GetVectorLayout(SimpleLayoutInfo elementInfo, size_t elementCount) = 0;
@@ -517,7 +657,7 @@ struct SimpleLayoutRulesImpl
     virtual UniformLayoutInfo BeginStructLayout() = 0;
 
     // Add a field to a `struct` type, and return the offset for the field
-    virtual size_t AddStructField(UniformLayoutInfo* ioStructInfo, UniformLayoutInfo fieldInfo) = 0;
+    virtual LayoutSize AddStructField(UniformLayoutInfo* ioStructInfo, UniformLayoutInfo fieldInfo) = 0;
 
     // End layout for a struct, and finalize its size/alignment.
     virtual void EndStructLayout(UniformLayoutInfo* ioStructInfo) = 0;
@@ -542,7 +682,7 @@ struct LayoutRulesImpl
         return simpleRules->GetScalarLayout(baseType);
     }
 
-    SimpleArrayLayoutInfo GetArrayLayout(SimpleLayoutInfo elementInfo, size_t elementCount)
+    SimpleArrayLayoutInfo GetArrayLayout(SimpleLayoutInfo elementInfo, LayoutSize elementCount)
     {
         return simpleRules->GetArrayLayout(elementInfo, elementCount);
     }
@@ -562,7 +702,7 @@ struct LayoutRulesImpl
         return simpleRules->BeginStructLayout();
     }
 
-    size_t AddStructField(UniformLayoutInfo* ioStructInfo, UniformLayoutInfo fieldInfo)
+    LayoutSize AddStructField(UniformLayoutInfo* ioStructInfo, UniformLayoutInfo fieldInfo)
     {
         return simpleRules->AddStructField(ioStructInfo, fieldInfo);
     }