Revise new COM-lite API (#1007)

* Revise new COM-lite API

This change revises the "COM-lite" API that was recently introduced to try to streamline it and introduce some missing central/base concepts.

The central new abstraction in the API is the notion of a "component type," which is a unit of shader code composition. A component type can have:

* IR code for some number of functions/types/etc.
* Zero or more global shader parameters
* Zero or more "entry point" functions at which execution can start
* Zero or more "specialization" parameters (types or values that must be filled in before kernel code can be generated)
* Zero or more "requirements" (dependencies on other component types that must be satisfied before kernel code can be generated)

Both individual compiled modules, and validated entry points are then examples of component types, and we additionally define a few services that apply to all component types:

* We can take N component types and compose them to create a new component type that combines their code, shader parameters, entry points, and specialization parameters. A composed component type may also include requirements from the sub-component types, but it is also possible that by composing thing we satisfy requirements (if `A` requires `B`, and we compose `A` and `B`, then the requirement is now satisfied, and doesn't appear on the composite).

* We can take a component type with N specialization parameters, and specialize it by giving N compatible specialization arguments. The result of specialization is a new component type with zero specialization parameters. Under the right circumstances the specialzed component type will be layout compatible with the unspecialized one.

* One more example that isn't exposed in the public API today is that we can take a component with requirements and "complete" it by automatically composing it with component types that satisfy those requirements. This can be seen as a kind of linking step that pulls together the transitive closure of dependencies.

* We can query the layout for the shader parameters and entry points of a component type, for a specific target.

* We can query compiled kernel code for an entry point in a component type (for a specific target). This only works for component types with zero specialization parameters and zero requirements.

The idea is that by giving users a fairly general algebra of operations on component types, they can compose final programs in ways that meet their requirements. For example, it becomes possible to incrementally "grow" a component type to represent the global root signature for ray tracing shaders as new entry points are added, in such a way that it always stays layout-compatible with kernels that have already been compiled.

Much of the implementation work here is in implementing the unifying component type abstraction, and in particular re-writing code that used to assume a program consisted of a flat list of modules and entry points to work with a hierarchical representation that reflects the underlying algebra (e.g., with types to represent composite and specialized component types).

There's also a hidden "legacy" case of a component type to deal with some legacy compiler behaviors that can't be directly modeled on top of the simple algebra with modules and entry points.

This API is by no means feature-complete or fully developed. It is expected that we will flesh it out more when bringing up application code (e.g., Falcor) on top of the revamped API.

One notable thing that went away in this change is explicit support for "entry point groups" and notions of local root signatures (especially the Falcor-specific handling of the `shared` keyword, which a previous change turned into an explicitly supported feature). With the new "building blocks" approach, it should be possible for a DXR application to deal with local root signatures as a matter of policy (on top of the API we provide). If/when we need to provide some kind of emulation of local root signatures for Vulkan (and/or if Vulkan is extended with an explicit notion of local root signatures), we might need to revisit this choice.

* Fix debug build

There was invalid code inside an `assert()`, so the release build didn't catch it.

* fixup: warnings

* fixup: more warnings-as-errors

* fixup: review notes

* fixup: use component type visitors in place of dynamic casting

1 files changed, 48 insertions, 44 deletions

diff --git a/source/slang/slang-ir-bind-existentials.cpp b/source/slang/slang-ir-bind-existentials.cpp
index e426e6e92..a99da3410 100644
--- a/source/slang/slang-ir-bind-existentials.cpp
+++ b/source/slang/slang-ir-bind-existentials.cpp
@@ -69,24 +69,8 @@ struct BindExistentialSlots
 
     void processGlobalExistentialSlots()
     {
-        // If there are any global existential slots, we will expect
-        // to find a `bindGlobalExistentialSlots` instruction at module scope.
-        //
-        // We will start out by finding that instruction, if it exists.
-        //
-        IRInst* bindGlobalExistentialSlotsInst = nullptr;
-        for( auto inst : module->getGlobalInsts() )
-        {
-            if( inst->op == kIROp_BindGlobalExistentialSlots )
-            {
-                bindGlobalExistentialSlotsInst = inst;
-                break;
-            }
-        }
-
-        // Now we will start looking for global shader parameters that make
-        // use of existential slots (we can determine this from their
-        // layout).
+        // We will search for global shader parameters that make
+        // use of existential specialization parameters.
         //
         for( auto inst : module->getGlobalInsts() )
         {
@@ -96,23 +80,27 @@ struct BindExistentialSlots
             if(!globalParam)
                 continue;
 
-            // We will delegate to a subroutine for the meat
-            // of the work, since much of it can be shared
-            // with the case for entry-point existential
-            // parameters.
+            // We only care about global shader parameters
+            // that have existential specialization parameters,
+            // and we expect all such parameters to have a
+            // `[bindExistentialSlots(...)]` decoration that
+            // was added during IR linking.
             //
-            processParameter(globalParam, bindGlobalExistentialSlotsInst);
-        }
+            auto bindSlotsInst = globalParam->findDecorationImpl(kIROp_BindExistentialSlotsDecoration);
+            if(!bindSlotsInst)
+                continue;
 
-        // Once we are done looping over global shader parameters,
-        // all of the relevant information from the
-        // `bindGlobalExistentialSlots` instruction will have
-        // been moved to the parameters themselves, so we
-        // can eliminate the binding instruction.
-        //
-        if( bindGlobalExistentialSlotsInst )
-        {
-            bindGlobalExistentialSlotsInst->removeAndDeallocate();
+            replaceTypeUsingExistentialSlots(
+                globalParam,
+                bindSlotsInst->getOperandCount(),
+                bindSlotsInst->getOperands());
+
+            // Once we have propagated the information from
+            // the `[bindExistentialSlots(...)]` decoration
+            // down into the parameter's type, we no longer
+            // need the decoration.
+            //
+            bindSlotsInst->removeAndDeallocate();
         }
     }
 
@@ -150,9 +138,25 @@ struct BindExistentialSlots
         // We then need to process each of the entry-point
         // parameters just like we did for global parameters.
         //
+        // Because the existential slot arguments for *all* of the parameters
+        // are attached in a single `[bindExistentialSlots(...)]` decoration,
+        // we need to carve them up appropriately across the parameters.
+        // The way we do this is a bit of a kludge, in that we track a
+        // single `slotOffset` and increment it for each parameter by the
+        // number of arguments it consumed.
+        //
+        // Note: a better approach here might rely on the layout information
+        // for the parameters, which should directly encode an offset for
+        // the existential specialization parameters it uses. The challenge
+        // with this is that we'd need to correctly interpret the offset
+        // relative to any global-scope specialization parameters or
+        // generic specialization parameters of the entry point.
+        // Ultimately the simplistic counter approach is less complicated.
+        //
+        Index slotOffset = 0;
         for( auto param : func->getParams() )
         {
-            processParameter(param, bindEntryPointExistentialSlotsInst);
+            processEntryPointParameter(param, bindEntryPointExistentialSlotsInst, slotOffset);
         }
 
         // TODO: We would need to consider what to do if
@@ -181,9 +185,10 @@ struct BindExistentialSlots
     // function `param` and a `[bindExistentialSlots(...)]`
     // decoration; both use the same subroutine.
     //
-    void processParameter(
+    void processEntryPointParameter(
         IRInst*     param,
-        IRInst*     bindSlotsInst)
+        IRInst*     bindSlotsInst,
+        Index&      ioSlotOperandOffset)
     {
         // We expect all shader parameters to have layout information,
         // but to be defensive we will skip any that don't.
@@ -206,7 +211,6 @@ struct BindExistentialSlots
         // find out the stating slot, and the information on
         // the type to find out the number of slots.
         //
-        UInt firstSlot = resInfo->index;
         UInt slotCount = 0;
         if(auto typeResInfo = varLayout->getTypeLayout()->FindResourceInfo(LayoutResourceKind::ExistentialTypeParam))
             slotCount = UInt(typeResInfo->count.getFiniteValue());
@@ -233,7 +237,8 @@ struct BindExistentialSlots
         // this parameter.
         //
         UInt bindOperandCount = bindSlotsInst->getOperandCount();
-        if( 2*(firstSlot + slotCount) > bindOperandCount )
+        UInt slotOperandCount = 2*slotCount;
+        if( (ioSlotOperandOffset + slotOperandCount) > bindOperandCount )
         {
             sink->diagnose(param->sourceLoc, Diagnostics::missingExistentialBindingsForParameter);
             return;
@@ -244,7 +249,7 @@ struct BindExistentialSlots
         // keeping in mind that each slot accounts for two
         // operands.
         //
-        auto operandsForInst = bindSlotsInst->getOperands() + firstSlot;
+        auto operandsForInst = bindSlotsInst->getOperands() + ioSlotOperandOffset;
 
         // Once we've found the operands that are relevent to
         // the slots used by `param`, we will defer to a routine
@@ -253,17 +258,17 @@ struct BindExistentialSlots
         //
         replaceTypeUsingExistentialSlots(
             param,
-            slotCount,
+            slotOperandCount,
             operandsForInst);
+
+        ioSlotOperandOffset += slotOperandCount;
     }
 
     void replaceTypeUsingExistentialSlots(
         IRInst*         inst,
-        UInt            slotCount,
+        UInt            slotOperandCount,
         IRUse const*    slotArgs)
     {
-        SLANG_UNUSED(slotCount);
-
         // We are going to alter the type of the
         // given `inst` based on information in
         // the `slotArgs`.
@@ -282,7 +287,6 @@ struct BindExistentialSlots
         // a witness table, so the total number of operands
         // is twice the number of slots we are filling.
         //
-        UInt slotOperandCount = slotCount*2;
         List<IRInst*> slotOperands;
         for(UInt ii = 0; ii < slotOperandCount; ++ii)
             slotOperands.add(slotArgs[ii].get());