Support cross-compilation of ray tracing shaders to Vulkan (#663)

* Move to newer glslang

* Support cross-compilation of ray tracing shaders to Vulkan

This change allows HLSL shaders authored for DirectX Raytracing (DXR) to be cross-compiled to run with the experimental `GL_NVX_raytracing` extension (aka "VKRay").

* The GLSL extension spec is marked as experimental, so that any shaders written using this support should be ready for breaking changes when the spec is finalized.
* "Callable shaders" are not exposed throug the GLSL extension, so this feature of DXR will not be cross-compiled.
* The experimental Vulkan raytracing extension does not have an equivalent to DXR's "local root signature" concept. This does not visibly impact shader translation (because the local/global root signature mapping is handled outside of the HLSL code), but in practice it means that applications which rely on local root signatures on their DXR path will not be able to use the translation in this change as-is; more work will be needed.

The simplest part of the implementation was to go into the Slang standard library and start adding GLSL translations for the various DXR operations.
In some cases, like mapping `IgnoreHit()` to `ignoreIntersectionNVX()` this is almost trivial.
The various functions to query system-provided values (e.g., `RayTMin()`) were also easy, with the only gotcha being that they map to variables rather than function calls in GLSL, and our handling of `__target_intrinsic` assumes that a bare identifier represents a replacement function name, and not a full expression, so we have to wrap these definitions in parentheses.

The tricky operations are then `TraceRay<P>()` and `ReportHit<A>()`, because these two are generics/templates in HLSL.
GLSL doesn't support generics, even for "standard library" functions, so the raytracing extension implements a slightly complex workaround: the matching operations `traceNVX()` and `reportIntersectionNVX()` pass the payload/attributes argument data via a global variable.
That is, shader code for the GLSL extensions writes to the global variable and then calls the intrinsic function.
The linkage between the call site and the global is established by a modifier keyword (`rayPayloadNVX` and `hitAttributeNVX`, respectively) and in the case of ray payload also uses `location` number to identify which payload global to use (since a single shader can trace rays with multiple payload types).

Our translation strategy in Slang tries to leverage standard language mechanisms instead of special-case logic.
For example, to translate the `ReportHit<A>()` function, we provide both a default declaration that will work for HLSL (where the operation is built-in with the signature given), and a *definition* marked with the `__specialized_for_target(glsl)` modifier.
The GLSL definition declares a function `static` variable that will fill the role of the required global, and then does what the GLSL spec requires: assigns to the global, and then calls the `reportIntersectionNVX` builtin (which we declare as a separate builtin).
Our ordinary lowering process will turn that `static` variable into an ordinary global in the IR, and the `[__vulkanHitAttributes]` attribute on the variable will be emitted as `hitAttributeNVX` in the output.
There is no additional cross-compilation logic in Slang specific to `ReportHit<A>()` - the target-specific definition in the standard library Just Works.

The case for `TraceRay<P>()` is a bit more complicated, simply because the GLSL `traceNVX()` function needs to be passed the `location` for the payload global.
We implement the payload global as a function-`static` variable, with the knowledge that every unique specialization of `TraceRay<P>()` will generate a unique global variable of type `P` to implement our function-`static` variable.
We then add a slightly magical builtin function `__rayPayloadLocation()` that can map such a variable to its generated `location`; the logic for this is implemented in `emit.cpp` and described below.

We also changed the `RayDesc` and `BuiltinTriangleIntersectionAttributes` types from "magic" intrinsic types over to ordinary types (because the GLSL output needs to declare them as ordinary `struct` types).
This ends up removing some cases in the AST and IR type representations.
By itself this change would break HLSL emit, because in that case the types really are intrinsic.
We added a `__target_intrinsic` modifier to these types to make them intrinsic for HLSL, and then updated the downstream passes to handle the notion of target-intrinsic types.

The logic for binding/layout of entry point inputs and outputs was updated so that raytracing stages don't follow the default logic for varying input/output parameters.
This is because the input/output parameters of a raytracing entry point aren't really "varying" in the same sense as those in the rasterization pipeline.
In particular, the SPIR-V model for raytracing input and output treats "ray payload" and "hit attributes" parameters as being in a distinct storage class from `in` or `out` parameters.

We also detect cases where a ray tracing stage declares inputs/outputs that it shouldn't have. This logic could conceivably be extended to other stages (e.g., to give an error on a compute shader with user-defined varying input/output).

The type layout logic added cases for handling raytracing payload and hit-attribute data, but this is currently just a stub implementation that follows the same logic as for varying `in` and `out` parameters (it cannot give meaningful byte sizes/offsets right now).
To my knowledge the GLSL spec doesn't currently specify anything about layout, and I haven't read the DXR spec language carefully enough to know what it says about layout.
A future change should update the layout logic to allow for byte-based layout of ray payloads, etc. so that we can query this information via reflection.

The GLSL legalization logic in `ir.cpp` was updated to factor out the per-entry-point-parameter code into its own function, and then that function was updated to special-case the input/output of a ray-tracing shader.

While for rasterization stages we typically want to take the user-declared input/output and "scalarize" it for use in GLSL (in part to deal with language limitations, and in part to tease system values apart from user-defined input/output), the GLSL spec for raytracing requires payload and hit attribute parameters to be declared as single variables. There is also the issue that even for an `in out` parameter, a ray payload parameter should only turn into a single global, whereas the handling for varying `in out` parameters generates both an `in` and an `out` global for the GLSL case.

Other than the handling of entry point parameters, the GLSL legalization pass doesn't need to do anything special for ray tracing shaders.

The trickiest change in the `emit.cpp` logic is that we now generate `location`s for ray payload arguments (the outgoing from a `TraceRay()` call) on demand during code generation.
This is a bit hacky, and it would be nice to handle it as a separate pass on the IR rather than clutter up the emit logic, but this approach was expedient.
Basically, any of the global variables that got generated from the `static` declarations in the standard library implementation of `TraceRay()` will trigger the logic to assign them a `location`.

The logic for emitting intrinsic operations added a few new `$`-based escape sequences. The `$XP` case handles emitting the location of a generated ray payload variable; this is how we emit the matching location at the site where we call `traceNVX`. The `$XT` case emits the appropriate translation for `RayTCurrent()` in HLSL, because it maps to something different depending on the target stage.

All of the test cases here consist of a pair of an HLSL/Slang shader written to the DXR spec, plus a matching GLSL shader for a baseline.
The GLSL shaders are carefully designed so that when fed into glslang they will produce the same SPIR-V as our cross-compilation process.
This kind of testing is quite fragile, but it seems to be the best we can do until our testing framework code supports *both* DXR and VKRay.

A bunch of the core changes ended up being blocked on issues in the rest of the compiler, so some additional features go implemented or fixed along the way:

The first big wall this work ran into was that the `__specialized_for_target` modifier hasn't actually been working correctly for a while.
It turns out that for the one function that is using it, `saturate()`, we have been outputting the workaround GLSL function in *all* cases (including for HLSL output) rather than only on GLSL targets.

The problem here is that for a generic function with a `__specialized_for_target` modifier or a `__target_intrinsic` modifier, the IR-level decoration will end up attached to the `IRFunc` instruction nested in the `IRGeneric`, but the logic for comparing IR declarations to see which is more specialized (via `getTargetSpecializationLevel()`) was looking only at decorations on the top-level value (the generic).

The quick (hacky) fix here is to make `getTargetSpecializationLevel()` try to look at the return value of a generic rather than the generic itself, so that it can see the decorations that indicate target-specific functions.
A more refined fix would be to attach target-specificity decorations to the outer-most generic (to simplify the "linking" logic).
The only reason not to fold that into the current fix is that the `__target_intrinsic` modifier currently serves double-duty as a marker of target specialization *and* information to drive emit logic. The latter (the emit-related stuff) currently needs to live on the `IRFunc`, and moving it to the generic could easily break a lot of code.

This needs more work in a follow-on fix, but for now target specialization should again be working.

The other big gotcha that the simple "just use the standard library" strategy ran into was that function-`static` variables weren't actually implemented yet, and in particular function-`static` variables inside of generic functions required some careful coding.

The logic in `lower-to-ir.cpp` has this `emitOuterGenerics()` function that is supposed to take a declaration that might be nested inside of zero or more levels of AST generics, and emit corresponding IR generics for all those levels.
This is needed because two different AST functions nested inside a single generic `struct` declaration should turn into distinct `IRFunc`s nested in distinct `IRGeneric`s.
The tricky bit to making that all work is that the same AST-level generic type parameter will then map to *different* IR-level instructions (the parameters of distinct `IRGeneric`s) when lowering each function.
The existing logic handled this in an idiomatic way by making "sub-builders" and "sub-contexts."
This change refactors some of the repeated logic into a `NestedContext` type to help simplify the pattern, and applies it consistently throughout the `lower-to-ir.cpp` file.

Besides that cleanup, the major change is `lowerFunctionStaticVarDecl` which, unsurprisingly, handles lower of function-`static` variables to IR globals.
The careful handling of nested contexts here is needed because if we are in the middle of lowering a generic function, then a `static` variable should turn into its *own* `IRGeneric` wrapping an `IRGlobalVar`. The body of the function should refer to the global variable by specializing the global variable's `IRGeneric` to the parameters of the *functions* `IRGeneric`. This tricky detail is handled by `defaultSpecializeOuterGenerics`.

An additional subtlety not actually required for this raytracing work (and thus not properly tested right now) is handling function-`static` variables with initializers.
These can't just be lowered to globals with initializers, because HLSL follows the C rule that function-`static` variables are initialized when the declaration statement is first executed (and this could be visible in the presence of side-effects).
The lowering strategy here translates any `static` variable with an initializer into *two* globals: one for the actual storage, plus a second `bool` variable to track whether it has been initialized yet.
There are some opportunities to optimize this case, especially for `static const` data, but that will need to wait for future changes.

We've slowly been shifting away from the model where a user thinks of a "profile" as including both a stage and a feature level.
Instead, the user should think about selecting a profile that only describes a feature level (e.g., `sm_6_1`, `glsl_450`, etc.), and then separately specifying a stage (`vertex`, `raygeneration, etc.) for each entry point.
The challenge here is that the command-line processing still only had a single `-profile` switch, and no way to specify the stage.

Adding the `-stage` option was relatively easy, but making it work with the existing validation logic for command-line arguments was tricky, because of the complex model that `slangc` supports for compiling multiple entry points in a single pass.

* In `slang.h` add new reflection parameter categories for ray payloads and hit attributes, as part of entry point input/output signatures.

* A previous change already updated our copy of glslang to one that supports the `GL_NVX_raytracing` extension, so in `slang-glslang.cpp` we just needed to map Slang's `enum` values for the raytracing stage names to their equivalents in the glslang code.

* Moved the logic for looking up a stage by name (`findStageByName()`) out of `check.cpp` and into `compiler.cpp`, with a declaration in `profile.h`

* Added a `$z` suffix to the GLSL translation of `Texture*.SampleLevel()`, to handle cases where the texture element type is not a 4-component vector. Note that this fix should actually be applied to *all* these texture-sampling operations, but I didn't want to add a bunch of changes that are (clearly) not being tested right now.

* The layout logic for entry points was updated to correctly skip producing a `TypeLayout` for an entry point result of type `void`, which meant that the related emit logic now needs to guard against a null value for the result layout.

* In `ir.cpp`, dump decorations on every instruction instead of just selected ones, so that our IR dump output is more complete.

* Added a command-line `-line-directive-mode` option so that we can easily turn off `#line` directives in the output when debugging. Not all cases where plumbed through because the `none` case is realistically the most important.

* Parser was fixed to properly initialize parent links for "scope" declarations used for statements, so that we can walk backwards from a function-scope variable (including a `static`) and see the outer function/generics/etc.

* Added GLSL 460 profile, since it is required for ray tracing. Also updated the logic for computing the "effective" profile to use to recognize that GLSL raytracing stages require GLSL 460.

* Added some conventional ray-tracing shader suffixes to the handling in `slang-test`. This code isn't actually used, but was relevant when I started by copy-pasting some existing VKRay shaders as the starting point for my testing.

* Fixup: typos

1 files changed, 375 insertions, 225 deletions

diff --git a/source/slang/ir.cpp b/source/slang/ir.cpp
index 72eea2e7c..0b4427f35 100644
--- a/source/slang/ir.cpp
+++ b/source/slang/ir.cpp
@@ -2849,6 +2849,30 @@ namespace Slang
                 }
                 break;
 
+            case kIRDecorationOp_TargetIntrinsic:
+                {
+                    auto decoration = (IRTargetIntrinsicDecoration*) dd;
+
+                    dump(context, "\n");
+                    dumpIndent(context);
+                    dump(context, "[targetIntrinsic(");
+                    dump(context, StringRepresentation::getData(decoration->targetName));
+                    dump(context, ", ");
+                    dump(context, StringRepresentation::getData(decoration->definition));
+                    dump(context, ")]");
+                }
+                break;
+
+            case kIRDecorationOp_VulkanRayPayload:
+                {
+                    dump(context, "\n[__vulkanRayPayload]");
+                }
+                break;
+            case kIRDecorationOp_VulkanHitAttributes:
+                {
+                    dump(context, "\n[__vulkanHitAttributes]");
+                }
+                break;
             }
         }
     }
@@ -2857,9 +2881,6 @@ namespace Slang
         IRDumpContext*          context,
         IRGlobalValueWithCode*  code)
     {
-        // TODO: should apply this to all instructions
-        dumpIRDecorations(context, code);
-
         auto opInfo = getIROpInfo(code->op);
 
         dump(context, "\n");
@@ -2920,9 +2941,6 @@ namespace Slang
         IRDumpContext*  context,
         IRParentInst*   inst)
     {
-        // TODO: should apply this to all instructions
-        dumpIRDecorations(context, inst);
-
         auto opInfo = getIROpInfo(inst->op);
 
         dump(context, "\n");
@@ -2988,6 +3006,8 @@ namespace Slang
 
         auto op = inst->op;
 
+        dumpIRDecorations(context, inst);
+
         // There are several ops we want to special-case here,
         // so that they will be more pleasant to look at.
         //
@@ -4188,11 +4208,17 @@ namespace Slang
                     fieldKey));
 
         case ScalarizedVal::Flavor::address:
-            return ScalarizedVal::address(
-                builder->emitFieldAddress(
-                    getFieldType(val.irValue->getDataType(), fieldKey),
-                    val.irValue,
-                    fieldKey));
+            {
+                auto ptrType = as<IRPtrTypeBase>(val.irValue->getDataType());
+                auto valType = ptrType->getValueType();
+                auto fieldType = getFieldType(valType, fieldKey);
+                auto fieldPtrType = builder->getPtrType(ptrType->op, fieldType);
+                return ScalarizedVal::address(
+                    builder->emitFieldAddress(
+                        fieldPtrType,
+                        val.irValue,
+                        fieldKey));
+            }
 
         case ScalarizedVal::Flavor::tuple:
             {
@@ -4536,6 +4562,308 @@ namespace Slang
         return nullptr;
     }
 
+    void legalizeRayTracingEntryPointParameterForGLSL(
+        GLSLLegalizationContext*    context,
+        IRParam*                    pp,
+        VarLayout*                  paramLayout)
+    {
+        auto builder = context->getBuilder();
+        auto paramType = pp->getDataType();
+
+        if(auto paramPtrType = as<IROutTypeBase>(paramType) )
+        {
+            // This is either an `out` or `in out` parameter.
+            // We want to treat `out` parameters the same
+            // as `in out` for our purposes, since there are
+            // no pure `out` parameters defined for the ray
+            // tracing stages.
+
+            // Unlike the default legalization strategy for
+            // `out` and `in out` entry point parameters,
+            // we will not introduce an intermediate temporary.
+            //
+            // Instead we will simply create a global variable
+            // and replace uses of the parameter with uses
+            // of that global variable.
+
+            auto valueType = paramPtrType->getValueType();
+
+            auto globalVariable = addGlobalVariable(builder->getModule(), valueType);
+            builder->addLayoutDecoration(globalVariable, paramLayout);
+            moveValueBefore(globalVariable, builder->getFunc());
+
+            pp->replaceUsesWith(globalVariable);
+        }
+        else
+        {
+            // This is the `in` parameter case, so that the parameter
+            // was not a pointer. We will allocate a global variable
+            // to represent the parameter, and then perform a load
+            // form it at the start of the function.
+            //
+            auto valueType = paramType;
+            auto globalVariable = addGlobalVariable(builder->getModule(), valueType);
+            builder->addLayoutDecoration(globalVariable, paramLayout);
+            moveValueBefore(globalVariable, builder->getFunc());
+
+            auto irLoad = builder->emitLoad(globalVariable);
+            pp->replaceUsesWith(irLoad);
+        }
+
+    }
+
+    void legalizeEntryPointParameterForGLSL(
+        GLSLLegalizationContext*    context,
+        IRFunc*                     func,
+        IRParam*                    pp,
+        VarLayout*                  paramLayout)
+    {
+        auto builder = context->getBuilder();
+        auto stage = context->getStage();
+
+        // We need to create a global variable that will replace the parameter.
+        // It seems superficially obvious that the variable should have
+        // the same type as the parameter.
+        // However, if the parameter was a pointer, in order to
+        // support `out` or `in out` parameter passing, we need
+        // to be sure to allocate a variable of the pointed-to
+        // type instead.
+        //
+        // We also need to replace uses of the parameter with
+        // uses of the variable, and the exact logic there
+        // will differ a bit between the pointer and non-pointer
+        // cases.
+        auto paramType = pp->getDataType();
+
+        // First we will special-case stage input/outputs that
+        // don't fit into the standard varying model.
+        // For right now we are only doing special-case handling
+        // of geometry shader output streams.
+        if( auto paramPtrType = as<IROutTypeBase>(paramType) )
+        {
+            auto valueType = paramPtrType->getValueType();
+            if( auto gsStreamType = as<IRHLSLStreamOutputType>(valueType) )
+            {
+                // An output stream type like `TriangleStream<Foo>` should
+                // more or less translate into `out Foo` (plus scalarization).
+
+                auto globalOutputVal = createGLSLGlobalVaryings(
+                    context,
+                    builder,
+                    valueType,
+                    paramLayout,
+                    LayoutResourceKind::VaryingOutput,
+                    stage);
+
+                // TODO: a GS output stream might be passed into other
+                // functions, so that we should really be modifying
+                // any function that has one of these in its parameter
+                // list (and in the limit we should be leagalizing any
+                // type that nests these...).
+                //
+                // For now we will just try to deal with `Append` calls
+                // directly in this function.
+
+
+
+                for( auto bb = func->getFirstBlock(); bb; bb = bb->getNextBlock() )
+                {
+                    for( auto ii = bb->getFirstInst(); ii; ii = ii->getNextInst() )
+                    {
+                        // Is it a call?
+                        if(ii->op != kIROp_Call)
+                            continue;
+
+                        // Is it calling the append operation?
+                        auto callee = ii->getOperand(0);
+                        for(;;)
+                        {
+                            // If the instruction is `specialize(X,...)` then
+                            // we want to look at `X`, and if it is `generic { ... return R; }`
+                            // then we want to look at `R`. We handle this
+                            // iteratively here.
+                            //
+                            // TODO: This idiom seems to come up enough that we
+                            // should probably have a dedicated convenience routine
+                            // for this.
+                            //
+                            // Alternatively, we could switch the IR encoding so
+                            // that decorations are added to the generic instead of the
+                            // value it returns.
+                            //
+                            switch(callee->op)
+                            {
+                            case kIROp_Specialize:
+                                {
+                                    callee = cast<IRSpecialize>(callee)->getOperand(0);
+                                    continue;
+                                }
+
+                            case kIROp_Generic:
+                                {
+                                    auto genericResult = findGenericReturnVal(cast<IRGeneric>(callee));
+                                    if(genericResult)
+                                    {
+                                        callee = genericResult;
+                                        continue;
+                                    }
+                                }
+
+                            default:
+                                break;
+                            }
+                            break;
+                        }
+                        if(callee->op != kIROp_Func)
+                            continue;
+
+                        // HACK: we will identify the operation based
+                        // on the target-intrinsic definition that was
+                        // given to it.
+                        auto decoration = findTargetIntrinsicDecoration(callee, "glsl");
+                        if(!decoration)
+                            continue;
+
+                        if(StringRepresentation::asSlice(decoration->definition) != UnownedStringSlice::fromLiteral("EmitVertex()"))
+                        {
+                            continue;
+                        }
+
+                        // Okay, we have a declaration, and we want to modify it!
+
+                        builder->setInsertBefore(ii);
+
+                        assign(builder, globalOutputVal, ScalarizedVal::value(ii->getOperand(2)));
+                    }
+                }
+
+                return;
+            }
+        }
+
+        // When we have an HLSL ray tracing shader entry point,
+        // we don't want to translate the inputs/outputs for GLSL/SPIR-V
+        // according to our default rules, for two reasons:
+        //
+        // 1. The input and output for these stages are expected to
+        // be packaged into `struct` types rather than be scalarized,
+        // so the usual scalarization approach we take here should
+        // not be applied.
+        //
+        // 2. An `in out` parameter isn't just sugar for a combination
+        // of an `in` and an `out` parameter, and instead represents the
+        // read/write "payload" that was passed in. It should legalize
+        // to a single variable, and we can lower reads/writes of it
+        // directly, rather than introduce an intermediate temporary.
+        //
+        switch( stage )
+        {
+        default:
+            break;
+
+        case Stage::AnyHit:
+        case Stage::Callable:
+        case Stage::ClosestHit:
+        case Stage::Intersection:
+        case Stage::Miss:
+        case Stage::RayGeneration:
+            legalizeRayTracingEntryPointParameterForGLSL(context, pp, paramLayout);
+            return;
+        }
+
+        // Is the parameter type a special pointer type
+        // that indicates the parameter is used for `out`
+        // or `inout` access?
+        if(auto paramPtrType = as<IROutTypeBase>(paramType) )
+        {
+            // Okay, we have the more interesting case here,
+            // where the parameter was being passed by reference.
+            // We are going to create a local variable of the appropriate
+            // type, which will replace the parameter, along with
+            // one or more global variables for the actual input/output.
+
+            auto valueType = paramPtrType->getValueType();
+
+            auto localVariable = builder->emitVar(valueType);
+            auto localVal = ScalarizedVal::address(localVariable);
+
+            if( auto inOutType = as<IRInOutType>(paramPtrType) )
+            {
+                // In the `in out` case we need to declare two
+                // sets of global variables: one for the `in`
+                // side and one for the `out` side.
+                auto globalInputVal = createGLSLGlobalVaryings(
+                    context,
+                    builder, valueType, paramLayout, LayoutResourceKind::VaryingInput, stage);
+
+                assign(builder, localVal, globalInputVal);
+            }
+
+            // Any places where the original parameter was used inside
+            // the function body should instead use the new local variable.
+            // Since the parameter was a pointer, we use the variable instruction
+            // itself (which is an `alloca`d pointer) directly:
+            pp->replaceUsesWith(localVariable);
+
+            // We also need one or more global variables to write the output to
+            // when the function is done. We create them here.
+            auto globalOutputVal = createGLSLGlobalVaryings(
+                    context,
+                    builder, valueType, paramLayout, LayoutResourceKind::VaryingOutput, stage);
+
+            // Now we need to iterate over all the blocks in the function looking
+            // for any `return*` instructions, so that we can write to the output variable
+            for( auto bb = func->getFirstBlock(); bb; bb = bb->getNextBlock() )
+            {
+                auto terminatorInst = bb->getLastInst();
+                if(!terminatorInst)
+                    continue;
+
+                switch( terminatorInst->op )
+                {
+                default:
+                    continue;
+
+                case kIROp_ReturnVal:
+                case kIROp_ReturnVoid:
+                    break;
+                }
+
+                // We dont' re-use `builder` here because we don't want to
+                // disrupt the source location it is using for inserting
+                // temporary variables at the top of the function.
+                //
+                IRBuilder terminatorBuilder;
+                terminatorBuilder.sharedBuilder = builder->sharedBuilder;
+                terminatorBuilder.setInsertBefore(terminatorInst);
+
+                // Assign from the local variabel to the global output
+                // variable before the actual `return` takes place.
+                assign(&terminatorBuilder, globalOutputVal, localVal);
+            }
+        }
+        else
+        {
+            // This is the "easy" case where the parameter wasn't
+            // being passed by reference. We start by just creating
+            // one or more global variables to represent the parameter,
+            // and attach the required layout information to it along
+            // the way.
+
+            auto globalValue = createGLSLGlobalVaryings(
+                context,
+                builder, paramType, paramLayout, LayoutResourceKind::VaryingInput, stage);
+
+            // Next we need to replace uses of the parameter with
+            // references to the variable(s). We are going to do that
+            // somewhat naively, by simply materializing the
+            // variables at the start.
+            IRInst* materialized = materializeValue(builder, globalValue);
+
+            pp->replaceUsesWith(materialized);
+        }
+    }
+
     void legalizeEntryPointForGLSL(
         Session*                session,
         IRModule*               module,
@@ -4672,218 +5000,11 @@ namespace Slang
                 SLANG_ASSERT(entryPointLayout->fields.Count() > paramIndex);
                 auto paramLayout = entryPointLayout->fields[paramIndex];
 
-                // We need to create a global variable that will replace the parameter.
-                // It seems superficially obvious that the variable should have
-                // the same type as the parameter.
-                // However, if the parameter was a pointer, in order to
-                // support `out` or `in out` parameter passing, we need
-                // to be sure to allocate a variable of the pointed-to
-                // type instead.
-                //
-                // We also need to replace uses of the parameter with
-                // uses of the variable, and the exact logic there
-                // will differ a bit between the pointer and non-pointer
-                // cases.
-                auto paramType = pp->getDataType();
-
-                // First we will special-case stage input/outputs that
-                // don't fit into the standard varying model.
-                // For right now we are only doing special-case handling
-                // of geometry shader output streams.
-                if( auto paramPtrType = as<IROutTypeBase>(paramType) )
-                {
-                    auto valueType = paramPtrType->getValueType();
-                    if( auto gsStreamType = as<IRHLSLStreamOutputType>(valueType) )
-                    {
-                        // An output stream type like `TriangleStream<Foo>` should
-                        // more or less translate into `out Foo` (plus scalarization).
-
-                        auto globalOutputVal = createGLSLGlobalVaryings(
-                            &context,
-                            &builder,
-                            valueType,
-                            paramLayout,
-                            LayoutResourceKind::VaryingOutput,
-                            stage);
-
-                        // TODO: a GS output stream might be passed into other
-                        // functions, so that we should really be modifying
-                        // any function that has one of these in its parameter
-                        // list (and in the limit we should be leagalizing any
-                        // type that nests these...).
-                        //
-                        // For now we will just try to deal with `Append` calls
-                        // directly in this function.
-
-
-
-                        for( auto bb = func->getFirstBlock(); bb; bb = bb->getNextBlock() )
-                        {
-                            for( auto ii = bb->getFirstInst(); ii; ii = ii->getNextInst() )
-                            {
-                                // Is it a call?
-                                if(ii->op != kIROp_Call)
-                                    continue;
-
-                                // Is it calling the append operation?
-                                auto callee = ii->getOperand(0);
-                                for(;;)
-                                {
-                                    // If the instruction is `specialize(X,...)` then
-                                    // we want to look at `X`, and if it is `generic { ... return R; }`
-                                    // then we want to look at `R`. We handle this
-                                    // iteratively here.
-                                    //
-                                    // TODO: This idiom seems to come up enough that we
-                                    // should probably have a dedicated convenience routine
-                                    // for this.
-                                    //
-                                    // Alternatively, we could switch the IR encoding so
-                                    // that decorations are added to the generic instead of the
-                                    // value it returns.
-                                    //
-                                    switch(callee->op)
-                                    {
-                                    case kIROp_Specialize:
-                                        {
-                                            callee = cast<IRSpecialize>(callee)->getOperand(0);
-                                            continue;
-                                        }
-
-                                    case kIROp_Generic:
-                                        {
-                                            auto genericResult = findGenericReturnVal(cast<IRGeneric>(callee));
-                                            if(genericResult)
-                                            {
-                                                callee = genericResult;
-                                                continue;
-                                            }
-                                        }
-
-                                    default:
-                                        break;
-                                    }
-                                    break;
-                                }
-                                if(callee->op != kIROp_Func)
-                                    continue;
-
-                                // HACK: we will identify the operation based
-                                // on the target-intrinsic definition that was
-                                // given to it.
-                                auto decoration = findTargetIntrinsicDecoration(callee, "glsl");
-                                if(!decoration)
-                                    continue;
-
-                                if(StringRepresentation::asSlice(decoration->definition) != UnownedStringSlice::fromLiteral("EmitVertex()"))
-                                {
-                                    continue;
-                                }
-
-                                // Okay, we have a declaration, and we want to modify it!
-
-                                builder.setInsertBefore(ii);
-
-                                assign(&builder, globalOutputVal, ScalarizedVal::value(ii->getOperand(2)));
-                            }
-                        }
-
-                        continue;
-                    }
-                }
-
-
-                // Is the parameter type a special pointer type
-                // that indicates the parameter is used for `out`
-                // or `inout` access?
-                if(auto paramPtrType = as<IROutTypeBase>(paramType) )
-                {
-                    // Okay, we have the more interesting case here,
-                    // where the parameter was being passed by reference.
-                    // We are going to create a local variable of the appropriate
-                    // type, which will replace the parameter, along with
-                    // one or more global variables for the actual input/output.
-
-                    auto valueType = paramPtrType->getValueType();
-
-                    auto localVariable = builder.emitVar(valueType);
-                    auto localVal = ScalarizedVal::address(localVariable);
-
-                    if( auto inOutType = as<IRInOutType>(paramPtrType) )
-                    {
-                        // In the `in out` case we need to declare two
-                        // sets of global variables: one for the `in`
-                        // side and one for the `out` side.
-                        auto globalInputVal = createGLSLGlobalVaryings(
-                            &context,
-                            &builder, valueType, paramLayout, LayoutResourceKind::VaryingInput, stage);
-
-                        assign(&builder, localVal, globalInputVal);
-                    }
-
-                    // Any places where the original parameter was used inside
-                    // the function body should instead use the new local variable.
-                    // Since the parameter was a pointer, we use the variable instruction
-                    // itself (which is an `alloca`d pointer) directly:
-                    pp->replaceUsesWith(localVariable);
-
-                    // We also need one or more global variabels to write the output to
-                    // when the function is done. We create them here.
-                    auto globalOutputVal = createGLSLGlobalVaryings(
-                            &context,
-                            &builder, valueType, paramLayout, LayoutResourceKind::VaryingOutput, stage);
-
-                    // Now we need to iterate over all the blocks in the function looking
-                    // for any `return*` instructions, so that we can write to the output variable
-                    for( auto bb = func->getFirstBlock(); bb; bb = bb->getNextBlock() )
-                    {
-                        auto terminatorInst = bb->getLastInst();
-                        if(!terminatorInst)
-                            continue;
-
-                        switch( terminatorInst->op )
-                        {
-                        default:
-                            continue;
-
-                        case kIROp_ReturnVal:
-                        case kIROp_ReturnVoid:
-                            break;
-                        }
-
-                        // We dont' re-use `builder` here because we don't want to
-                        // disrupt the source location it is using for inserting
-                        // temporary variables at the top of the function.
-                        //
-                        IRBuilder terminatorBuilder;
-                        terminatorBuilder.sharedBuilder = builder.sharedBuilder;
-                        terminatorBuilder.setInsertBefore(terminatorInst);
-
-                        // Assign from the local variabel to the global output
-                        // variable before the actual `return` takes place.
-                        assign(&terminatorBuilder, globalOutputVal, localVal);
-                    }
-                }
-                else
-                {
-                    // This is the "easy" case where the parameter wasn't
-                    // being passed by reference. We start by just creating
-                    // one or more global variables to represent the parameter,
-                    // and attach the required layout information to it along
-                    // the way.
-
-                    auto globalValue = createGLSLGlobalVaryings(
-                        &context,
-                        &builder, paramType, paramLayout, LayoutResourceKind::VaryingInput, stage);
-
-                    // Next we need to replace uses of the parameter with
-                    // references to the variable(s). We are going to do that
-                    // somewhat naively, by simply materializing the
-                    // variables at the start.
-                    IRInst* materialized = materializeValue(&builder, globalValue);
-
-                    pp->replaceUsesWith(materialized);
-                }
+                legalizeEntryPointParameterForGLSL(
+                    &context,
+                    func,
+                    pp,
+                    paramLayout);
             }
 
             // At this point we should have eliminated all uses of the
@@ -5137,6 +5258,18 @@ namespace Slang
                 }
                 break;
 
+            case kIRDecorationOp_VulkanRayPayload:
+                {
+                    context->builder->addDecoration<IRVulkanRayPayloadDecoration>(clonedValue);
+                }
+                break;
+
+            case kIRDecorationOp_VulkanHitAttributes:
+                {
+                    context->builder->addDecoration<IRVulkanHitAttributesDecoration>(clonedValue);
+                }
+                break;
+
             default:
                 // Don't clone any decorations we don't understand.
                 break;
@@ -5721,9 +5854,26 @@ namespace Slang
     };
 
     TargetSpecializationLevel getTargetSpecialiationLevel(
-        IRGlobalValue*  val,
+        IRGlobalValue*  inVal,
         String const&   targetName)
     {
+        // HACK: Currently the front-end is placing modifiers related
+        // to target specialization on nodes like functions, even when
+        // those functions are being returned by a generic. This
+        // means that we need to try and inspect the value being
+        // returned by the generic if we are looking at a generic.
+        IRInst* val = inVal;
+        while( auto genericVal = as<IRGeneric>(val) )
+        {
+            auto firstBlock = genericVal->getFirstBlock();
+            if(!firstBlock) break;
+
+            auto returnInst = as<IRReturnVal>(firstBlock->getLastInst());
+            if(!returnInst) break;
+
+            val = returnInst->getVal();
+        }
+
         TargetSpecializationLevel result = TargetSpecializationLevel::notSpecialized;
         for( auto dd = val->firstDecoration; dd; dd = dd->next )
         {
@@ -5776,13 +5926,13 @@ namespace Slang
         switch (val->op)
         {
         case kIROp_WitnessTable:
-        case kIROp_GlobalVar:
         case kIROp_GlobalConstant:
         case kIROp_Func:
         case kIROp_Generic:
             return ((IRParentInst*)val)->getFirstChild() != nullptr;
 
         case kIROp_StructType:
+        case kIROp_GlobalVar:
             return true;
 
         default: