Add Vulkan/SPIR-V support for TraceRayInline() (#1737)

For the most part, this translation is straightforward because the `GL_EXT_ray_query` extension is well aligned with the DXR 1.1 `RayQuery` feature. Many function map one-to-one from one extension to the other.

A few notable details:

* The equivalent of the `RayQuery<Flags>` type is non-generic in GLSL, and the GLSL path previously didn't have support for trying to look up an intrinsic type name on an IR type declaration, so that required some tweaks to the emit logic.

* All the GLSL functions are free functions instead of member functions, but our IR doesn't recognize that distinction anyway

* The main `TraceRayInline()` call is the one that took the most tweaking, just because it takes a `RayDesc` structure for D3D/HLSL but takes individual vector sand scalars for VK/GLSL. The approach here is a standard one for how we manage this stuff in the stdlib (and I wanted to avoid adding even more `$` magic for intrinsics).

* For several other calls, the HLSL API had distinct `Candidate***()` and `Committed***()` calls that return information about a candidate hit vs. the one committed into the query. In contrast, the GLSL API uses a single call that takes an additional "must be compile-time constant" `bool` parameter to select between the two behaviors. This is even the case for one call that basically returns a value of a different `enum` type depending on the state of that `bool`. The D3D API model here seems almost strictly better and I have no idea why the GLSL extension was defined this way.

* Because both the `GL_EXT_ray_query` and `GL_EXT_ray_tracing` extensions declare the `accelerationStructureEXT` type, we can no longer infer what extension is supposed to be used based only on the presene of such a type. The logic right now is a bit slippery, because in theory a program that declares an acceleration structure but never traces into it could end up getting a compilation error now. We will have to see if that corner case comes up in practice. :(

The one big detail that is looming after doing this work is that both the HLSL and GLSL exposures of ray queries are extremely "slippery" about the actual identity of queries (e.g., when is one query a copy of another, vs. just being a new variable that references the existing query). Somehow queries get their identity from the original declaration, and as such our "default constructor" approach to them seems semanticay correct, but the whole thing is kind of slippery at a foundational level and I don't know how to fix it with the API as defined. Oh well; just something to keep an eye on.

Co-authored-by: Yong He <yonghe@outlook.com>

3 files changed, 629 insertions, 0 deletions

diff --git a/tests/pipeline/ray-tracing/trace-ray-inline.slang b/tests/pipeline/ray-tracing/trace-ray-inline.slang
new file mode 100644
index 000000000..30c32ee4f
--- /dev/null
+++ b/tests/pipeline/ray-tracing/trace-ray-inline.slang
@@ -0,0 +1,172 @@
+// trace-ray-inline.slang
+
+//TEST:CROSS_COMPILE:-target dxil-asm -stage compute -profile sm_6_5 -entry main
+//TEST:CROSS_COMPILE:-target spirv-asm -stage compute -profile sm_6_5 -entry main
+
+// The goal of this shader is to use all the main pieces
+// of functionality in DXR 1.1's `TraceRayInline` feature,
+// to ensure that they survive translation to HLSL.
+
+// In order to trace rays, we need an acceleration structure.
+//
+RaytracingAccelerationStructure myAccelerationStructure;
+
+// We also need to decide what to do with hits/misses.
+// The `TraceRayInline` approach eschews separate shader
+// stages for RT, and instead expects users to write
+// those operations as subroutines instead.
+//
+// We will mimic the style and naming of DXR 1.0 here
+// to try and make the parallels clear.
+//
+// We start with a ray "payload" type that will be
+// used for input/output on hit and miss shaders
+//
+
+struct MyRayPayload
+{
+	int value;
+};
+
+// The first and simplest shader is the miss shader.
+//
+void myMiss(inout MyRayPayload payload)
+{
+	payload.value = 0;
+}
+
+// Next, up is a closest hit shader for opaque triangles.
+//
+void myTriangleClosestHit(inout MyRayPayload payload)
+{
+	payload.value = 1;
+}
+
+// In order to support alpha testing, we need an any-hit
+// shader for triangles.
+//
+// In this case, the return value is used to specify
+// whether the hit should be accepted (true) or ignored (false).
+//
+bool myTriangleAnyHit(inout MyRayPayload payload)
+{
+	return true;
+}
+
+// Procedural primitives are different than triangles
+// in that they need user-defined hit attributes.
+//
+struct MyProceduralHitAttrs { int value; }
+
+// Otherwise, the closest- and any-hit shaders
+// for procedural primitives are similar to those
+// for triangles.
+//
+void myProceduralClosestHit(inout MyRayPayload payload, MyProceduralHitAttrs attrs)
+{
+	payload.value = attrs.value;
+}
+bool myProceduralAnyHit(inout MyRayPayload payload)
+{
+	return true;
+}
+
+// The new piece of the puzzle for procedural primitives
+// is the intersection shader, which should be able to
+// report zero or more intersections.
+//
+// For now we will only deal with the single-intersection
+// case.
+//
+bool myProceduralIntersection(inout float tHit, inout MyProceduralHitAttrs hitAttrs)
+{
+	return true;
+}
+
+// In order to kick of tracing we need the properties of a ray
+// query to trace, so we will pipe those in via a constant buffer.
+//
+cbuffer C
+{
+	float3 origin;
+	float tMin;
+	float3 direction;
+	float tMax;
+	uint rayFlags;
+	uint instanceMask;
+	uint shouldStopAtFirstHit;
+}
+
+// The actual tracing is handled by a compute shader,
+// which here takes on the role of a ray generation shader.
+//
+void main(uint3 tid : SV_DispatchThreadID)
+{
+	uint index = tid.x;
+
+	RayQuery<RAY_FLAG_NONE> query;
+	MyProceduralHitAttrs committedProceduralAttrs;
+
+	MyRayPayload payload = { -1 };
+	RayDesc ray = { origin, tMin, direction, tMax };
+	query.TraceRayInline(
+		myAccelerationStructure,
+		rayFlags,
+		instanceMask,
+		ray);
+
+
+	for(;;)
+	{
+		if(!query.Proceed()) break;
+
+		switch(query.CandidateType())
+		{
+		case CANDIDATE_PROCEDURAL_PRIMITIVE:
+			{
+				MyProceduralHitAttrs candidateProceduralAttrs = { 0 };
+				float tHit = 0.0f;
+				if(myProceduralIntersection(tHit, candidateProceduralAttrs))
+				{
+					if(myProceduralAnyHit(payload))
+					{
+						query.CommitProceduralPrimitiveHit(tHit);
+						committedProceduralAttrs = candidateProceduralAttrs;
+						if(shouldStopAtFirstHit)
+							query.Abort();
+					}
+				}
+			}
+			break;
+
+		case CANDIDATE_NON_OPAQUE_TRIANGLE:
+			{
+				if(myTriangleAnyHit(payload))
+				{
+					query.CommitNonOpaqueTriangleHit();
+					if(shouldStopAtFirstHit)
+						query.Abort();
+				}
+			}
+			break;
+
+		}
+
+
+	}
+
+	switch(query.CommittedStatus())
+	{
+	case COMMITTED_TRIANGLE_HIT:
+		myTriangleClosestHit(payload);
+		break;
+
+	case COMMITTED_PROCEDURAL_PRIMITIVE_HIT:
+		myProceduralClosestHit(payload, committedProceduralAttrs);
+		break;
+
+	case COMMITTED_NOTHING:
+		myMiss(payload);
+		break;
+	}
+}
\ No newline at end of file
diff --git a/tests/pipeline/ray-tracing/trace-ray-inline.slang.glsl b/tests/pipeline/ray-tracing/trace-ray-inline.slang.glsl
new file mode 100644
index 000000000..883742020
--- /dev/null
+++ b/tests/pipeline/ray-tracing/trace-ray-inline.slang.glsl
@@ -0,0 +1,219 @@
+// trace-ray-inline.slang.glsl
+//TEST_IGNORE_FILE:
+
+#version 460
+#extension GL_EXT_ray_query : require
+
+struct SLANG_ParameterGroup_C_0
+{
+    vec3 origin_0;
+    float tMin_0;
+    vec3 direction_0;
+    float tMax_0;
+    uint rayFlags_0;
+    uint instanceMask_0;
+    uint shouldStopAtFirstHit_0;
+};
+
+layout(binding = 1)
+layout(std140) uniform _S1
+{
+    SLANG_ParameterGroup_C_0 _data;
+} C_0;
+
+struct RayDesc_0
+{
+    vec3 Origin_0;
+    float TMin_0;
+    vec3 Direction_0;
+    float TMax_0;
+};
+
+void RayQuery_TraceRayInline_0(rayQueryEXT this_0, accelerationStructureEXT accelerationStructure_0, uint rayFlags_1, uint instanceInclusionMask_0, RayDesc_0 ray_0)
+{
+    rayQueryInitializeEXT((this_0), (accelerationStructure_0), (rayFlags_1), (instanceInclusionMask_0), (ray_0.Origin_0), (ray_0.TMin_0), (ray_0.Direction_0), (ray_0.TMax_0));
+    return;
+}
+
+layout(binding = 0)
+uniform accelerationStructureEXT myAccelerationStructure_0;
+
+struct MyProceduralHitAttrs_0
+{
+    int value_0;
+};
+
+bool myProceduralIntersection_0(inout float tHit_0, inout MyProceduralHitAttrs_0 hitAttrs_0)
+{
+    return true;
+}
+
+struct MyRayPayload_0
+{
+    int value_1;
+};
+
+bool myProceduralAnyHit_0(inout MyRayPayload_0 payload_0)
+{
+    return true;
+}
+
+bool myTriangleAnyHit_0(inout MyRayPayload_0 payload_1)
+{
+    return true;
+}
+
+void myTriangleClosestHit_0(inout MyRayPayload_0 payload_2)
+{
+    payload_2.value_1 = 1;
+    return;
+}
+
+void myProceduralClosestHit_0(inout MyRayPayload_0 payload_3, MyProceduralHitAttrs_0 attrs_0)
+{
+    payload_3.value_1 = attrs_0.value_0;
+    return;
+}
+
+void myMiss_0(inout MyRayPayload_0 payload_4)
+{
+    payload_4.value_1 = 0;
+    return;
+}
+
+layout(local_size_x = 1, local_size_y = 1, local_size_z = 1) in;
+void main()
+{
+    MyRayPayload_0 payload_5;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_0;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_1;
+    MyRayPayload_0 payload_6;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_2;
+    MyRayPayload_0 payload_7;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_3;
+    rayQueryEXT query_0;
+    MyRayPayload_0 _S2 = { -1 };
+    RayDesc_0 ray_1 = { C_0._data.origin_0, C_0._data.tMin_0, C_0._data.direction_0, C_0._data.tMax_0 };
+    RayQuery_TraceRayInline_0(query_0, myAccelerationStructure_0, C_0._data.rayFlags_0, C_0._data.instanceMask_0, ray_1);
+    MyProceduralHitAttrs_0 _S3;
+    payload_5 = _S2;
+    committedProceduralAttrs_0 = _S3;
+    for(;;)
+    {
+        bool _S4 = rayQueryProceedEXT(query_0);
+        if(!_S4)
+        {
+            break;
+        }
+        uint _S5 = (rayQueryGetIntersectionTypeEXT((query_0), false));
+        switch(_S5)
+        {
+        case uint(1):
+            {
+                MyProceduralHitAttrs_0 candidateProceduralAttrs_0 = { 0 };
+                float _S6;
+                _S6 = 0.00000000000000000000;
+                MyProceduralHitAttrs_0 _S7;
+                _S7 = candidateProceduralAttrs_0;
+                bool _S8 = myProceduralIntersection_0(_S6, _S7);
+                float tHit_1 = _S6;
+                MyProceduralHitAttrs_0 candidateProceduralAttrs_1 = _S7;
+                if(_S8)
+                {
+                    MyRayPayload_0 _S9;
+                    _S9 = payload_5;
+                    bool _S10 = myProceduralAnyHit_0(_S9);
+                    MyRayPayload_0 _S11 = _S9;
+                    if(_S10)
+                    {
+                        rayQueryGenerateIntersectionEXT(query_0, tHit_1);
+                        if(bool(C_0._data.shouldStopAtFirstHit_0))
+                        {
+                            rayQueryTerminateEXT(query_0);
+                        }
+                        else
+                        {
+                        }
+                        committedProceduralAttrs_1 = candidateProceduralAttrs_1;
+                    }
+                    else
+                    {
+                        committedProceduralAttrs_1 = committedProceduralAttrs_0;
+                    }
+                    payload_6 = _S11;
+                    committedProceduralAttrs_2 = committedProceduralAttrs_1;
+                }
+                else
+                {
+                    payload_6 = payload_5;
+                    committedProceduralAttrs_2 = committedProceduralAttrs_0;
+                }
+                payload_7 = payload_6;
+                committedProceduralAttrs_3 = committedProceduralAttrs_2;
+                break;
+            }
+        case uint(0):
+            {
+                MyRayPayload_0 _S12;
+                _S12 = payload_5;
+                bool _S13 = myTriangleAnyHit_0(_S12);
+                MyRayPayload_0 _S14 = _S12;
+                if(_S13)
+                {
+                    rayQueryConfirmIntersectionEXT(query_0);
+                    if(bool(C_0._data.shouldStopAtFirstHit_0))
+                    {
+                        rayQueryTerminateEXT(query_0);
+                    }
+                    else
+                    {
+                    }
+                }
+                else
+                {
+                }
+                payload_7 = _S14;
+                committedProceduralAttrs_3 = committedProceduralAttrs_0;
+                break;
+            }
+        default:
+            {
+                payload_7 = payload_5;
+                committedProceduralAttrs_3 = committedProceduralAttrs_0;
+                break;
+            }
+        }
+        payload_5 = payload_7;
+        committedProceduralAttrs_0 = committedProceduralAttrs_3;
+    }
+    uint _S15 = (rayQueryGetIntersectionTypeEXT((query_0), true));
+    switch(_S15)
+    {
+    case uint(1):
+        {
+            MyRayPayload_0 _S16;
+            _S16 = payload_5;
+            myTriangleClosestHit_0(_S16);
+            break;
+        }
+    case uint(2):
+        {
+            MyRayPayload_0 _S17;
+            _S17 = payload_5;
+            myProceduralClosestHit_0(_S17, committedProceduralAttrs_0);
+            break;
+        }
+    case uint(0):
+        {
+            MyRayPayload_0 _S18;
+            _S18 = payload_5;
+            myMiss_0(_S18);
+            break;
+        }
+    default:
+        {
+            break;
+        }
+    }
+    return;
+}
diff --git a/tests/pipeline/ray-tracing/trace-ray-inline.slang.hlsl b/tests/pipeline/ray-tracing/trace-ray-inline.slang.hlsl
new file mode 100644
index 000000000..cdea1668a
--- /dev/null
+++ b/tests/pipeline/ray-tracing/trace-ray-inline.slang.hlsl
@@ -0,0 +1,238 @@
+// trace-ray-inline.slang.hlsl
+//TEST_IGNORE_FILE:
+
+struct SLANG_ParameterGroup_C_0
+{
+    vector<float,3> origin_0;
+    float tMin_0;
+    vector<float,3> direction_0;
+    float tMax_0;
+    uint rayFlags_0;
+    uint instanceMask_0;
+    uint shouldStopAtFirstHit_0;
+};
+
+cbuffer C_0 : register(b0)
+{
+    SLANG_ParameterGroup_C_0 C_0;
+}
+
+RaytracingAccelerationStructure myAccelerationStructure_0 : register(t0);
+
+struct MyProceduralHitAttrs_0
+{
+    int value_0;
+};
+
+bool myProceduralIntersection_0(inout float tHit_0, inout MyProceduralHitAttrs_0 hitAttrs_0)
+{
+    return true;
+}
+
+struct MyRayPayload_0
+{
+    int value_1;
+};
+
+bool myProceduralAnyHit_0(inout MyRayPayload_0 payload_0)
+{
+    return true;
+}
+
+bool myTriangleAnyHit_0(inout MyRayPayload_0 payload_1)
+{
+    return true;
+}
+
+void myTriangleClosestHit_0(inout MyRayPayload_0 payload_2)
+{
+    payload_2.value_1 = int(1);
+    return;
+}
+
+void myProceduralClosestHit_0(inout MyRayPayload_0 payload_3, MyProceduralHitAttrs_0 attrs_0)
+{
+    payload_3.value_1 = attrs_0.value_0;
+    return;
+}
+
+void myMiss_0(inout MyRayPayload_0 payload_4)
+{
+    payload_4.value_1 = int(0);
+    return;
+}
+
+
+[shader("compute")]
+[numthreads(1, 1, 1)]
+void main(vector<uint,3> tid_0 : SV_DISPATCHTHREADID)
+{
+    MyRayPayload_0 payload_5;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_0;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_1;
+    MyRayPayload_0 payload_6;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_2;
+    MyRayPayload_0 payload_7;
+    MyProceduralHitAttrs_0 committedProceduralAttrs_3;
+
+    RayQuery<int(0) > query_0;
+
+    MyRayPayload_0 _S1 = { int(-1) };
+    RayDesc ray_0 = { C_0.origin_0, C_0.tMin_0, C_0.direction_0, C_0.tMax_0 };
+    query_0.TraceRayInline(myAccelerationStructure_0, C_0.rayFlags_0, C_0.instanceMask_0, ray_0);
+
+    MyProceduralHitAttrs_0 _S2;
+
+    payload_5 = _S1;
+    committedProceduralAttrs_0 = _S2;
+    for(;;)
+    {
+        bool _S3 = query_0.Proceed();
+
+        if(!_S3)
+        {
+            break;
+        }
+        uint _S4 = query_0.CandidateType();
+
+        switch(_S4)
+        {
+        case (uint) int(1):
+            {
+                MyProceduralHitAttrs_0 candidateProceduralAttrs_0 = { int(0) };
+
+                float _S5;
+
+                _S5 = 0.00000000000000000000;
+
+                MyProceduralHitAttrs_0 _S6;
+
+                _S6 = candidateProceduralAttrs_0;
+
+                bool _S7 = myProceduralIntersection_0(_S5, _S6);
+
+                float tHit_1 = _S5;
+
+                MyProceduralHitAttrs_0 candidateProceduralAttrs_1 = _S6;
+
+                if(_S7)
+                {
+                    MyRayPayload_0 _S8;
+
+                    _S8 = payload_5;
+
+                    bool _S9 = myProceduralAnyHit_0(_S8);
+
+                    MyRayPayload_0 _S10 = _S8;
+
+                    if(_S9)
+                    {
+                        query_0.CommitProceduralPrimitiveHit(tHit_1);
+
+                        if((bool) C_0.shouldStopAtFirstHit_0)
+                        {
+
+                            query_0.Abort();
+                        }
+                        else
+                        {
+                        }
+
+                        committedProceduralAttrs_1 = candidateProceduralAttrs_1;
+                    }
+                    else
+                    {
+                        committedProceduralAttrs_1 = committedProceduralAttrs_0;
+                    }
+
+                    payload_6 = _S10;
+                    committedProceduralAttrs_2 = committedProceduralAttrs_1;
+                }
+                else
+                {
+                    payload_6 = payload_5;
+                    committedProceduralAttrs_2 = committedProceduralAttrs_0;
+                }
+
+                payload_7 = payload_6;
+                committedProceduralAttrs_3 = committedProceduralAttrs_2;
+                break;
+            }
+        case (uint) int(0):
+            {
+                MyRayPayload_0 _S11;
+                _S11 = payload_5;
+
+                bool _S12 = myTriangleAnyHit_0(_S11);
+                MyRayPayload_0 _S13 = _S11;
+
+                if(_S12)
+                {
+                    query_0.CommitNonOpaqueTriangleHit();
+                    if((bool) C_0.shouldStopAtFirstHit_0)
+                    {
+                        query_0.Abort();
+                    }
+                    else
+                    {
+                    }
+                }
+                else
+                {
+                }
+
+                payload_7 = _S13;
+                committedProceduralAttrs_3 = committedProceduralAttrs_0;
+                break;
+            }
+        default:
+            {
+                payload_7 = payload_5;
+                committedProceduralAttrs_3 = committedProceduralAttrs_0;
+                break;
+            }
+        }
+
+        payload_5 = payload_7;
+        committedProceduralAttrs_0 = committedProceduralAttrs_3;
+    }
+
+    uint _S14 = query_0.CommittedStatus();
+
+    switch(_S14)
+    {
+    case (uint) int(1):
+        {
+            MyRayPayload_0 _S15;
+
+            _S15 = payload_5;
+
+            myTriangleClosestHit_0(_S15);
+            break;
+        }
+    case (uint) int(2):
+        {
+
+            MyRayPayload_0 _S16;
+            _S16 = payload_5;
+
+            myProceduralClosestHit_0(_S16, committedProceduralAttrs_0);
+            break;
+        }
+    case (uint) int(0):
+        {
+            MyRayPayload_0 _S17;
+
+            _S17 = payload_5;
+
+            myMiss_0(_S17);
+            break;
+        }
+    default:
+        {
+            break;
+        }
+    }
+
+    return;
+}