Fix GS cross-compilation after IR type system change (#507)

The cross-compilation logic for geometry shaders would look through the user's entry point for calls like `someStream.Emit<X>(val)` and turn that into `outputGlobals = val; EmitVertex();`.

It was recognizing the `Emit()` calls by looking at the callee in all `call` instructions and seeing if it was registered to lower to GLSL as `EmitVertex()`. The logic was try to look "through" `specialize` instructions (to deal with the `<X>` bit in the call above), but this wasn't updated for the new IR encoding where the first operand to a `specialize` is the generic being specialized, and not the function nested inside it.

The fix here is to properly look through both `specialize` instructions and generics. This is kind of a gross operation and we've done things like it in a few places, so it might be something we try to extract into a utility function in the future.

1 files changed, 36 insertions, 2 deletions

diff --git a/source/slang/ir.cpp b/source/slang/ir.cpp
index 397726b0a..a75a02ca7 100644
--- a/source/slang/ir.cpp
+++ b/source/slang/ir.cpp
@@ -4279,9 +4279,43 @@ namespace Slang
 
                                 // Is it calling the append operation?
                                 auto callee = ii->getOperand(0);
-                                while( callee->op == kIROp_Specialize )
+                                for(;;)
                                 {
-                                    callee = ((IRSpecialize*) callee)->getOperand(0);
+                                    // If the instruction is `specialize(X,...)` then
+                                    // we want to look at `X`, and if it is `generic { ... return R; }`
+                                    // then we want to look at `R`. We handle this
+                                    // iteratively here.
+                                    //
+                                    // TODO: This idiom seems to come up enough that we
+                                    // should probably have a dedicated convenience routine
+                                    // for this.
+                                    //
+                                    // Alternatively, we could switch the IR encoding so
+                                    // that decorations are added to the generic instead of the
+                                    // value it returns.
+                                    //
+                                    switch(callee->op)
+                                    {
+                                    case kIROp_Specialize:
+                                        {
+                                            callee = cast<IRSpecialize>(callee)->getOperand(0);
+                                            continue;
+                                        }
+
+                                    case kIROp_Generic:
+                                        {
+                                            auto genericResult = findGenericReturnVal(cast<IRGeneric>(callee));
+                                            if(genericResult)
+                                            {
+                                                callee = genericResult;
+                                                continue;
+                                            }
+                                        }
+
+                                    default:
+                                        break;
+                                    }
+                                    break;
                                 }
                                 if(callee->op != kIROp_Func)
                                     continue;