Add support for unbounded arrays as shader parameters (#725)

* Add support for unbounded arrays as shader parameters

With this change, Slang shaders can use unbounded-size arrays as parameters, e.g.:

```hlsl
Texture2D t[] : register(t3, space2);
SamplerState s[];
```

As shown in the above example, Slang supports both explicit `register` declarations on unbounded-size arrays and also implicit binding.
When doing automatic parmaeter binding, Slang will allocate a full register space to an unbounded-size array of textures/smaplers, starting at register zero.

Note that for the Vulkan target, an array of descriptors of any size (including unbounded size) consumes only a single `bindign`, so much of this logic is specific to D3D targets.

Details on the changes made:

* The single biggest change is a new `LayoutSize` type that is used to store a value that can either be a finite unsigned integer or a dedicated "infinite" value (which is stored as the all-bits-set `-1` value). This is used in places where a size could either be a finite value or an "unbounded" value, to both try to make standard math robust against the infinite case, and also to force code to deal with both the finite and infinite cases more explicitly when they care about the difference.

* The public API was documented so that unbounded-size arrays report their size as `-1`. We should probably change this function to return a signed value instead of `size_t`, but that would technically be a source-breaking change, so we want to make sure we stage it appropriately.

* The code that invokes fxc was updated so that it passes the appropriate flag to enable unbounded arrays of descriptors. I haven't looked yet at whether dxc needs such a flag, so there may need to be a follow-on change to add that.

* The logic in the `UsedRanges::Add` method for tracking what registers have been claimed was rewritten because the previous version had some subtle bugs. The new version includes more detailed comments that attempt to explain why I think the new logic works.

* The top-level logic for auto-assigning bindings to parameters has been overhauled to deal with the fact that a parameter that needs "infinite" amounts of a resource should be claiming a full register space for those resources instead. Whenever a parameter allocates any register spaces we want them all to be contiguous, so we have a loop that counts the requirements and allocates the spaces before we go along and dole them out.

* When computing the layout for an array type, we need to carefully deal with unbounded-size arrays. In the case of an unbounded array of a "simple" resource type (e.g., `Texture2D[]`), we opt to expose the type layout as consuming an infinite number of the appropriate register, while in the case of a complex type (say, a `struct` with two texture fields), we need to instead allocate whole spaces for those fields. The logic here is more subtle than I would like, and interacts with the existing code that "adjusts" the element type of an array in order to make standard indexing math Just Work.

* Similarly, when a `struct` type has unbounded-array fields, then we need to transform any field with infinite register requirements to instead consume a space in the resulting aggregate type. This case is comparatively easier than the array case.

* The test case for unbounded arrays covers both explicit and implicit bindings, and also the case of an unbounded array over a `struct` type (it does not cover the case of a `struct` contianing unbounded arrays, so that will need to be added later). For this test we are both validation the output reflection data and that we produce the same code as fxc (with explicit bindings in the fxc case).

* The reflection test app was modified to use the new API contract and detect when a parameter consumes `SLANG_UNBOUNDED_SIZE` resources.

* Fixup: ensure unbounded size is defined at right bit width

2 files changed, 284 insertions, 0 deletions

diff --git a/tests/reflection/unbounded-arrays.hlsl b/tests/reflection/unbounded-arrays.hlsl
new file mode 100644
index 000000000..2c3b7a7bb
--- /dev/null
+++ b/tests/reflection/unbounded-arrays.hlsl
@@ -0,0 +1,157 @@
+// unbounded-arrays.hlsl
+
+//TEST:COMPARE_HLSL:-profile cs_5_1 -entry main
+//TEST:REFLECTION:-profile cs_5_1 -target hlsl -D__SLANG__
+
+//
+// This test is trying to make sure that we correctly compute
+// reflection/layout information for shaders that make use
+// of unbounded arrays of resources.
+//
+// We will begin by declaring various "simple" global arrays
+// of resource/sampler types and try out variations on binding
+// them to registers/spaces or not.
+//
+// We will want to confirm that Slang generates the bindings
+// we expect, and we will do this by enforcing explicit
+// bindings on all parameters in the HLSL baseline we'll
+// compare against:
+//
+
+    #ifdef __SLANG__
+        #define REGISTER(x, y) /* empty */
+    #else
+        #define REGISTER(x,y) : register(x,y)
+        #define aa aa_0
+        #define b0 b0_0
+        #define b1 b1_0
+        #define bb bb_0
+        #define c0 c0_0
+        #define cc cc_0
+        #define data data_0
+    #endif
+
+// First, let's just declare a simple unbounded array of samplers.
+// We expect this to be given its own register and space (for D3D12)
+//
+
+    SamplerState aa[] REGISTER(s0, space2);
+
+//
+// Next, we will try to declare an array of resources with an explicit
+// `register` binding. This should be set to start at that register in
+// space zero (the default space), and should therefore "claim" all
+// registers from that point on.
+//
+
+    Texture2D bb[] : register(t2);
+
+//
+// If we have assigned register t2 and beyond in space zero to `bb`,
+// then we should still be able to put other resources in there explicitly:
+//
+
+    Texture2D b0 : register(t0);
+    Texture2D b1 : register(t1, space0);
+
+//
+// It should also be possible to give an unbounded array an explicit
+// register and space, and again it should be poossible to fill
+// in the space before the unbounded array:
+//
+
+    TextureCube cc[] : register(t1, space1);
+    Texture2D c0 : register(t0, space1);
+
+//
+// As a final detail, we should allow the user to specify the space
+// and no register, which should be interpreted as requesting *any*
+// register in the given space.
+//
+// TODO: Implement support for this case.
+//
+
+//    SamplerState dd[] : register(space5);
+
+//
+// With the simple cases out of the way, we will look at cases
+// that involve structures and nested arrays.
+//
+// The first case we'll test is a structure type that contains
+// two or more resources:
+//
+
+    struct X
+    {
+        Texture3D t;
+        SamplerState s;
+    };
+
+//
+// The simple case should Just Work, even though the same
+// syntax will fail when used with fxc (so we have to
+// provide a hand-written expansion for the baseline).
+//
+
+    #ifdef __SLANG__
+        X ee[];
+    #else
+        Texture3D ee_t_0[] REGISTER(t0, space3);
+        SamplerState ee_s_0[] REGISTER(s0, space4);
+    #endif
+
+//
+// TODO: we should probably test interactions with explicit
+// bindings for a structrure.
+//
+// TODO: we should also test cases that mix resource and
+// non-resource types, but we can't currently have an unbounded
+// array of uniform data (in HLSL at least).
+//
+// TODO: should test arrays-of-arrays cases.
+//
+
+
+//
+// We'll close things out with a dummy entry point just
+// to allow this file to be compiled with fxc/dxc.
+//
+
+    float4 use(Texture2D t, SamplerState s, float4 u)
+    {
+        return t.SampleLevel(s, u.xy + u.z, u.w);
+    }
+
+    float4 use(Texture3D t, SamplerState s, float4 u)
+    {
+        return t.SampleLevel(s, u.xyz, u.w);
+    }
+
+    float4 use(TextureCube t, SamplerState s, float4 u)
+    {
+        return t.SampleLevel(s, u.xyz, u.w);
+    }
+
+    RWStructuredBuffer<float4> data;
+
+    [numthreads(4,1,1)]
+    void main(uint3 tid : SV_DispatchThreadID)
+    {
+        int idx = tid.x;
+        float4 tmp = data[idx];
+
+        SamplerState s = aa[idx];
+
+        tmp = use(bb[idx],  s, tmp);
+        tmp = use(b0,       s, tmp);
+        tmp = use(b1,       s, tmp);
+        tmp = use(cc[idx],  s, tmp);
+        tmp = use(c0,       s, tmp);
+
+#ifdef __SLANG__
+        tmp = use(ee[idx].t, ee[idx].s, tmp);
+#else
+        tmp = use(ee_t_0[idx], ee_s_0[idx], tmp);
+#endif
+        data[idx] = tmp;
+    }
diff --git a/tests/reflection/unbounded-arrays.hlsl.1.expected b/tests/reflection/unbounded-arrays.hlsl.1.expected
new file mode 100644
index 000000000..382fc25bc
--- /dev/null
+++ b/tests/reflection/unbounded-arrays.hlsl.1.expected
@@ -0,0 +1,127 @@
+result code = 0
+standard error = {
+}
+standard output = {
+{
+    "parameters": [
+        {
+            "name": "aa",
+            "binding": {"kind": "samplerState", "space": 2, "index": 0, "count": "unbounded"},
+            "type": {
+                "kind": "array",
+                "elementCount": 0,
+                "elementType": {
+                    "kind": "samplerState"
+                }
+            }
+        },
+        {
+            "name": "bb",
+            "binding": {"kind": "shaderResource", "index": 2, "count": "unbounded"},
+            "type": {
+                "kind": "array",
+                "elementCount": 0,
+                "elementType": {
+                    "kind": "resource",
+                    "baseShape": "texture2D"
+                }
+            }
+        },
+        {
+            "name": "b0",
+            "binding": {"kind": "shaderResource", "index": 0},
+            "type": {
+                "kind": "resource",
+                "baseShape": "texture2D"
+            }
+        },
+        {
+            "name": "b1",
+            "binding": {"kind": "shaderResource", "index": 1},
+            "type": {
+                "kind": "resource",
+                "baseShape": "texture2D"
+            }
+        },
+        {
+            "name": "cc",
+            "binding": {"kind": "shaderResource", "space": 1, "index": 1, "count": "unbounded"},
+            "type": {
+                "kind": "array",
+                "elementCount": 0,
+                "elementType": {
+                    "kind": "resource",
+                    "baseShape": "textureCube"
+                }
+            }
+        },
+        {
+            "name": "c0",
+            "binding": {"kind": "shaderResource", "space": 1, "index": 0},
+            "type": {
+                "kind": "resource",
+                "baseShape": "texture2D"
+            }
+        },
+        {
+            "name": "ee",
+            "binding": {"kind": "registerSpace", "index": 3, "count": 2},
+            "type": {
+                "kind": "array",
+                "elementCount": 0,
+                "elementType": {
+                    "kind": "struct",
+                    "name": "X",
+                    "fields": [
+                        {
+                            "name": "t",
+                            "type": {
+                                "kind": "resource",
+                                "baseShape": "texture3D"
+                            },
+                            "binding": {"kind": "shaderResource", "index": 0}
+                        },
+                        {
+                            "name": "s",
+                            "type": {
+                                "kind": "samplerState"
+                            },
+                            "binding": {"kind": "samplerState", "space": 1, "index": 0}
+                        }
+                    ]
+                }
+            }
+        },
+        {
+            "name": "data",
+            "binding": {"kind": "unorderedAccess", "index": 0},
+            "type": {
+                "kind": "resource",
+                "baseShape": "structuredBuffer",
+                "access": "readWrite"
+            }
+        }
+    ],
+    "entryPoints": [
+        {
+            "name": "main",
+            "stage:": "compute",
+            "parameters": [
+                {
+                    "name": "tid",
+                    "semanticName": "SV_DISPATCHTHREADID",
+                    "type": {
+                        "kind": "vector",
+                        "elementCount": 3,
+                        "elementType": {
+                            "kind": "scalar",
+                            "scalarType": "uint32"
+                        }
+                    }
+                }
+            ],
+            "threadGroupSize": [4, 1, 1]
+        }
+    ]
+}
+}