Fix a regression in geometry shader cross-compilation (#794)

The underlying problem here was that legalization of entry point parameters for GLSL eliminates all the parameters to `main()`, but we still left a dangling reference to one of those parameters if it was a geometry shader output stream. The un-parented parameter would lead to an infinite loop in a later IR step, because it would never be reached by the transformation, and thus could never change its status to the one for "visited" instructions.

The fix here is to simply replace any refernces to the GS input stream parameter with an `undefined` instruction in the IR, and then rely on the fact that the downstream GLSL emit logic wouldn't actually reference that value anyway (hence why the danlging reference wasn't originally an issue).

I included a basic cross-compilation test case for geometry shaders to try to avoid subsequent regressions like this (Vulkan GS support is one of the most commonly recurring regressions we've had).

The comment I put into the IR legalization logic makes it clear that the strategy used there isn't 100% rock-solid anyway (it only works in all the `EmitVertex()` calls come from the shader entry point function, and not subroutines. Adding a better (more robust) translation strategy for geometry shaders would be a nice bit of future work.

1 files changed, 17 insertions, 0 deletions

diff --git a/source/slang/ir-glsl-legalize.cpp b/source/slang/ir-glsl-legalize.cpp
index b659cf293..9c4210be0 100644
--- a/source/slang/ir-glsl-legalize.cpp
+++ b/source/slang/ir-glsl-legalize.cpp
@@ -1238,6 +1238,23 @@ void legalizeEntryPointParameterForGLSL(
                 }
             }
 
+            // We will still have references to the parameter coming
+            // from the `EmitVertex` calls, so we need to replace it
+            // with something. There isn't anything reasonable to
+            // replace it with that would have the right type, so
+            // we will replace it with an undefined value, knowing
+            // that the emitted code will not actually reference it.
+            //
+            // TODO: This approach to generating geometry shader code
+            // is not ideal, and we should strive to find a better
+            // approach that involes coding the `EmitVertex` operation
+            // directly in the stdlib, similar to how ray-tracing
+            // operations like `TraceRay` are handled.
+            //
+            builder->setInsertBefore(func->getFirstBlock()->getFirstOrdinaryInst());
+            auto undefinedVal = builder->emitUndefined(pp->getFullType());
+            pp->replaceUsesWith(undefinedVal);
+
             return;
         }
     }