Add a basc inlining facility for use in the stdlib (#1271)

The main feature visible to the stdlib here is the `[__unsafeForceInlineEarly]` attribute, which can be attached to a function definition and forces calls to that function to be inlined immediately after initial IR lowering.

The pass is implemented in `slang-ir-inline.{h,cpp}` and currently only handles the completely trivial case of a function with no control flow that ends with a single `return`. The lack of support for any other cases motivates the `__unsafe` prefix on the attribute.

In order to test that the pass works, I modified the "comma operator" in the standard library to be defined directly (rather than relying on special-case handling in IR lowering), and then added a test that uses that operator to make sure it generates code as expected. The compute version of the test confirms that we generate semantically correct code for the operator, while the SPIR-V cross-compilation test confirms that our output matches GLSL where the comma operator has been inlined, rather than turned into a subroutine.

Notes for the future:

* With this change it should be possible (in principle) to redefine all the compound operators in the stdlib to instead be ordinary functions with the new attribute, removing the need for `slang-compound-intrinsics.h`.

* Once the compound intrinsics are defined in the stdlib, it should be easier/possible to start making built-in operators like `+` be ordinary functions from the standpoint of the IR

* The attribute and pass here could be extended to include an alternative inlining attribute that happens later in compilation (after linking) but otherwise works the same. This could in theory be used for functions where we don't want to inline the definition into generated IR, but still want to inline things berfore generating final HlSL/GLSL/whatever.

* The inlining pass itself could be generalized to work for less trivial functions pretty easily; for the most part it would just mean "splitting" the block with the call site and then inserting clones of the blocks in the callee. Any `return` instructions in the clone would become unconditional branches (with arguments) to the block after the call (which would get a parameter to represent the returned value).

* The "hard" part for such an inlining pass would be handling cases where the control flow that results from inlining can't be handled by our later restructuring passes. The long-term fix there is to implement something like the "relooper" algorithm to restructure control flow as required for specific targets.

16 files changed, 611 insertions, 14 deletions

diff --git a/source/slang/core.meta.slang b/source/slang/core.meta.slang
index 2df6cb36b..f73178322 100644
--- a/source/slang/core.meta.slang
+++ b/source/slang/core.meta.slang
@@ -113,7 +113,12 @@ interface __FlagsEnumType : __EnumType
 // argument. The left-to-right evaluation order guaranteed by Slang then ensures that
 // `left` is evaluated before `right`.
 //
-__generic<T,U> __intrinsic_op($(kCompoundIntrinsicOp_Sequence)) U operator,(T left, U right);
+__generic<T,U>
+[__unsafeForceInlineEarly]
+U operator,(T left, U right)
+{
+    return right;
+}
 
 // The ternary `?:` operator does not short-circuit in HLSL, and Slang continues to
 // follow that definition, so that this operator is effectively just an ordinary
@@ -1773,3 +1778,5 @@ attribute_syntax [allow(diagnostic: String)] : AllowAttribute;
 __attributeTarget(Decl)
 attribute_syntax [__extern] : ExternAttribute;
 
+__attributeTarget(FunctionDeclBase)
+attribute_syntax [__unsafeForceInlineEarly] : UnsafeForceInlineEarlyAttribute;
\ No newline at end of file
diff --git a/source/slang/slang-compound-intrinsics.h b/source/slang/slang-compound-intrinsics.h
index 4a0b1a109..682e7bc4a 100644
--- a/source/slang/slang-compound-intrinsics.h
+++ b/source/slang/slang-compound-intrinsics.h
@@ -29,7 +29,6 @@
     M(PreDec)                \
     M(PostInc)               \
     M(PostDec)               \
-    M(Sequence)              \
     M(AddAssign)             \
     M(SubAssign)             \
     M(MulAssign)             \
diff --git a/source/slang/slang-ir-clone.cpp b/source/slang/slang-ir-clone.cpp
index df0555f9b..9f9349f99 100644
--- a/source/slang/slang-ir-clone.cpp
+++ b/source/slang/slang-ir-clone.cpp
@@ -228,7 +228,26 @@ IRInst* cloneInst(
     SLANG_ASSERT(builder);
     SLANG_ASSERT(oldInst);
 
-    auto newInst = cloneInstAndOperands(
+    IRInst* newInst = nullptr;
+    if( env->mapOldValToNew.TryGetValue(oldInst, newInst) )
+    {
+        // In this case, somebody is trying to clone an
+        // instruction that already had been cloned
+        // (e.g., trying to clone a `param` in a function
+        // body that had already been mapped to a specialization)
+        // so we will make the operation safer and more
+        // convenient by just returning the registered value.
+        //
+        // TODO: There might be cases where the client doesn't
+        // want this convenience feature (because it could
+        // accidentally mask a bug), so we should consider
+        // having two versions of `cloneInst()` with one
+        // explicitly not including this feature.
+        //
+        return newInst;
+    }
+
+    newInst = cloneInstAndOperands(
         env, builder, oldInst);
 
     env->mapOldValToNew.Add(oldInst, newInst);
diff --git a/source/slang/slang-ir-inline.cpp b/source/slang/slang-ir-inline.cpp
new file mode 100644
index 000000000..19083c09b
--- /dev/null
+++ b/source/slang/slang-ir-inline.cpp
@@ -0,0 +1,451 @@
+// slang-ir-inline.cpp
+#include "slang-ir-inline.h"
+
+// This file provides general facilities for inlining function calls.
+
+//
+// A  *call site* is an individual `call` instruction (`IRCall`), and the *callee*
+// for a given call site is whatever is being called. When the callee is a `func`
+// (`IRFunc`) or a specialization of a `generic` that yields a `func`, *and* the
+// function has a body, then inlinling is possible.
+//
+// Different inlining passes may apply different heuristics or rules to decide
+// which call sites should be inlined (if possible). The rules may be based
+// on user-supplied hints, or on optimization criteria like performance and
+// code size.
+
+#include "slang-ir.h"
+#include "slang-ir-clone.h"
+#include "slang-ir-insts.h"
+
+namespace Slang
+{
+    /// Base type for inlining passes, providing shared/common functionality
+struct InliningPassBase
+{
+        /// The module that we are optimizing/transforming
+    IRModule* m_module = nullptr;
+
+        /// Initialize an inlining pass to operate on the given `module`
+    InliningPassBase(IRModule* module)
+        : m_module(module)
+    {
+    }
+
+        /// Consider all the call sites in the module for inliing
+    void considerAllCallSites()
+    {
+        considerAllCallSitesRec(m_module->getModuleInst());
+    }
+
+        /// Consider all call sites at or under `inst` for inlining
+    void considerAllCallSitesRec(IRInst* inst)
+    {
+        if( auto call = as<IRCall>(inst) )
+        {
+            considerCallSite(call);
+        }
+
+        // Note: we defensively iterate through the child instructions
+        // so that even if `child` gets removed (because of inlining)
+        // we automatically start at the next instruction after it.
+        //
+        IRInst* next = nullptr;
+        for( auto child = inst->getFirstChild(); child; child = next )
+        {
+            next = child->getNextInst();
+            considerAllCallSitesRec(child);
+        }
+    }
+
+    // In order to inline a call site, we need certain information
+    // to be present/available. Most notable is that the callee must
+    // be known, and it must be in the form of an `IRFunc`.
+    //
+    // Since checking whether we *can* inline a call site involves
+    // finding all of this information, we will use that opportunity
+    // to package it all up in a `struct` that can be re-used when
+    // we actually get around to inlining a call site.
+
+        /// Information about a call site to be inlined
+    struct CallSiteInfo
+    {
+            /// The call instruction.
+        IRCall* call = nullptr;
+
+            /// The function being called.
+            ///
+            /// For an inlinable call, this must be non-null and a valid function *definition* (with a body) for inlining to proceed.
+        IRFunc* callee = nullptr;
+
+            /// The specialization of the function, if any.
+            ///
+            /// For an inlineable call, this must be non-null if the function is generic, but may be null otherwise.
+        IRSpecialize* specialize = nullptr;
+
+            /// The generic being specialized.
+            ///
+            /// For an inlineable call, this must be be non-null if `specialize` is non-null.
+        IRGeneric* generic = nullptr;
+    };
+
+    // With `CallSiteInfo` defined, we can now understand the
+    // basic proces of considering a call site for inlining.
+
+        /// Consider the given `call` site, and possibly inline it.
+    void considerCallSite(IRCall* call)
+    {
+        // We start by checking if inlining would even be possible,
+        // since doing so collects information about the call site
+        // that can simplify the following steps.
+        //
+        // If the call can't be inlined, there is nothing else
+        // to consider and we bail out.
+        //
+        CallSiteInfo callSite;
+        if(!canInline(call, callSite))
+            return;
+
+        // If we've decided that we *can* inline the given call
+        // site, we next need to check if we *should*. The rules
+        // for when we should inline may vary by subclass,
+        // so `shouldInline` is a virtual method.
+        //
+        if(!shouldInline(callSite))
+            return;
+
+        // Finally, if we both *can* and *should* inline the
+        // given call site, we hand off the a worker routine
+        // that does the meat of the work.
+        //
+        inlineCallSite(callSite);
+    }
+
+    // Every subclas of `InliningPassBase` should provide its own
+    // definition of `shouldInline`. We define a default implementation
+    // here for the benefit of passes that might implement their
+    // own logic for deciding what to inline, bypassing `considerCallSite`.
+
+        /// Determine whether `callSite` should be inlined.
+    virtual bool shouldInline(CallSiteInfo const& callSite)
+    {
+        SLANG_UNUSED(callSite);
+        return false;
+    }
+
+        /// Determine whether `call` can be inlined, and if so write information about it to `outCallSite`
+    bool canInline(IRCall* call, CallSiteInfo& outCallSite)
+    {
+        // We can start by writing the `call` instruction into our `CallSiteInfo`.
+        //
+        outCallSite.call = call;
+
+        // Next we consider the callee.
+        //
+        IRInst* callee = call->getCallee();
+
+        // If the callee is a `specialize` instruction, then we
+        // want to look at what is being specialized instead.
+        //
+        if( auto specialize = as<IRSpecialize>(callee) )
+        {
+            // If the `specialize` is applied to something other
+            // than a `generic` instruction, then we can't
+            // inline the call site. This can happen for a
+            // call to a generic method in an interface.
+            //
+            IRGeneric* generic = findSpecializedGeneric(specialize);
+            if(!generic)
+                return false;
+
+            // If we have a `generic` instruction, then we
+            // will look to see if we can determine what
+            // it returns. If a result is found, that
+            // will be used as the new callee for this
+            // call site.
+            //
+            // If we can't identify the value that the generic
+            // yields, then inlining isn't possible.
+            //
+            callee = findGenericReturnVal(generic);
+            if(!callee)
+                return false;
+
+            // If we decide to inline this call, then the information
+            // we've just extracted about generic specialization
+            // will be relevant, so we write it to the `CallSiteInfo` now.
+            //
+            outCallSite.specialize = specialize;
+            outCallSite.generic = generic;
+        }
+
+        // Once we've dispensed with any possible generic specialization
+        // we will check if the callee is a `func` instruction (`IRFunc`).
+        //
+        // If it is not, then inlining isn't possible.
+        //
+        auto calleeFunc = as<IRFunc>(callee);
+        if(!calleeFunc)
+            return false;
+        //
+        // If the callee *is* a function, then we can update
+        // the `CalleSiteInfo` with what we've found.
+        //
+        outCallSite.callee = calleeFunc;
+
+        // At this point the `CallSiteInfo` is complete and
+        // could be used for inlining, but we have additional
+        // checks to make.
+        //
+        // In particular, we should only go about inlining
+        // a call site if the callee function is a full definition
+        // in the IR (not just a declaration).
+        //
+        if(!isDefinition(calleeFunc))
+            return false;
+
+        return true;
+    }
+
+        /// Inline the given `callSite`, which is assumed to have been validated
+    void inlineCallSite(CallSiteInfo const& callSite)
+    {
+        // Information about the call site, including
+        // the `call` instruction and the callee `func`
+        // should already have been computed and stored
+        // in the `CallSiteInfo`.
+        //
+        IRCall* call = callSite.call;
+        IRFunc* callee = callSite.callee;
+
+        // We will use the existing IR cloning infrastructure to clone
+        // the body of the callee, but we need to establish an
+        // environment for cloning in which any parameters of
+        // the callee are replaced with the matching arguments
+        // at the call site.
+        //
+        IRCloneEnv env;
+
+        // We also need an `IRBuilder` to construct the cloned IR,
+        // and will set it up to insert before the `call` that
+        // is going to be replaced.
+        //
+        SharedIRBuilder sharedBuilder;
+        sharedBuilder.session = m_module->getSession();
+        sharedBuilder.module = m_module;
+        IRBuilder builder;
+        builder.sharedBuilder = &sharedBuilder;
+        builder.setInsertBefore(call);
+
+        // If the callee is a generic function, then we will
+        // need to include the substitution of generic parameters
+        // with their argument values in our cloning.
+        //
+        if( auto specialize = callSite.specialize )
+        {
+            auto generic = callSite.generic;
+
+            // We start by establishing a mapping from the
+            // generic parameters to the matching arguments.
+            //
+            Int argCounter = 0;
+            for( auto param : generic->getParams() )
+            {
+                SLANG_ASSERT(argCounter < (Int)specialize->getArgCount());
+                auto arg = specialize->getArg(argCounter++);
+
+                env.mapOldValToNew.Add(param, arg);
+            }
+            SLANG_ASSERT(argCounter == (Int)specialize->getArgCount());
+
+            // We also need to clone any instructions in the
+            // body of the `generic` being specialized, since
+            // these might construct types or constants that
+            // reference the generic parameters.
+            //
+            auto body = generic->getFirstBlock();
+            SLANG_ASSERT(!body->getNextBlock()); // All IR generics should have a single block.
+
+            for( auto inst : body->getChildren() )
+            {
+                if( inst == callee )
+                {
+                    // We don't want to create a clone of the callee
+                    // function at the call site, since it would
+                    // immediately become dead code when we inline
+                    // its body.
+                }
+                else if(as<IRReturn>(inst))
+                {
+                    // We also don't want to clone any `return`
+                    // instruction in the generic, since that is
+                    // how they yield their result (which we
+                    // already know is `callee`.
+                }
+                else
+                {
+                    // In the default case, we just clone the instruction
+                    // from the body of the generic into the call site.
+                    //
+                    // TODO: This assumes that deduplication will work
+                    // as intended, so in practice we might run into
+                    // problems if we create new instances of IR types
+                    // or constants that already exist.
+                    //
+                    cloneInst(&env, &builder, inst);
+                }
+            }
+        }
+
+        // Compared to dealing with generic parameters, the process
+        // for dealing with value parameters is much simpler.
+        //
+        {
+            // For each parameter of the callee function, we
+            // insert a mapping into `env` from that parameter to the
+            // matching argument at the call site.
+            //
+            Int argCounter = 0;
+            for(auto param : callee->getParams())
+            {
+                SLANG_ASSERT(argCounter < (Int)call->getArgCount());
+                auto arg = call->getArg(argCounter++);
+                env.mapOldValToNew.Add(param, arg);
+            }
+            SLANG_ASSERT(argCounter == (Int)call->getArgCount());
+        }
+
+        // For now, our inlining pass only handles the case where
+        // the callee is a "trivial" function, which can support
+        // a very simple approach to inlining.
+        //
+        if( isTrivialFunc(callee) )
+        {
+            inlineTrivialFuncBody(callSite, &env, &builder);
+        }
+        else
+        {
+            // Running into any non-trivial function to be inlined
+            // is currently an internal compiler error.
+            //
+            SLANG_UNIMPLEMENTED_X("general case of inlining");
+        }
+    }
+
+        /// Check if `func` represents a trivial single-block callee that can be inlined simply
+    bool isTrivialFunc(IRFunc* func)
+    {
+        // The function must have a single bocy block to be trivial.
+        //
+        auto firstBlock = func->getFirstBlock();
+        if( firstBlock->getNextBlock() )
+            return false;
+
+        // If the body block is decorated (for some reason), then the function is non-trivial.
+        //
+        if( firstBlock->getFirstDecoration() )
+            return false;
+
+        // If the body block terminates in something other than a `return` then the function is non-trivial.
+        //
+        auto terminator = firstBlock->getTerminator();
+        if( !as<IRReturn>(terminator) )
+            return false;
+
+        return true;
+    }
+
+        /// Inline the body of the callee for `callSite`, where the callee is trivial as tested by `isTrivialFunc`
+    void inlineTrivialFuncBody(CallSiteInfo const& callSite, IRCloneEnv* env, IRBuilder* builder)
+    {
+        auto call = callSite.call;
+        auto callee = callSite.callee;
+        auto firstBlock = callee->getFirstBlock();
+
+        // We know that the callee has a single block, so if we encounter
+        // a `returnVal` instruction then it must be the one and only
+        // return point for the block, and its operand will be the value
+        // the calee returns.
+        //
+        IRInst* returnedValue = nullptr;
+
+        // We will loop over the instructions of the one and only block,
+        // and clone each of them appropriately.
+        //
+        for( auto inst : firstBlock->getChildren() )
+        {
+            switch( inst->op )
+            {
+            default:
+                // The default value is to clone the instruction using
+                // the existing cloning infrastructure and the `env`
+                // we have already set up.
+                //
+                cloneInst(env, builder, inst);
+                break;
+
+            case kIROp_Param:
+                // Parameters can be completely ignored in the single-block
+                // case, because they have all been replaced via `env`.
+                break;
+
+            case kIROp_ReturnVoid:
+                // A return with no operand can be ignored, since a return
+                // from the inlined call should just continue after the
+                // call site.
+                //
+                break;
+
+            case kIROp_ReturnVal:
+                // A return with a value is similar to `returnVoid` except
+                // that we need to note the (clone of the) value being
+                // returned, so that we can use it to replace the value
+                // of the original call.
+                //
+                returnedValue = findCloneForOperand(env, inst->getOperand(0));
+                break;
+            }
+        }
+
+        // If there was a `returnVal` instruction that established
+        // the return value of the inlined function, then that value
+        // should be used to replace any uses of the original call.
+        //
+        if( returnedValue )
+        {
+            call->replaceUsesWith(returnedValue);
+        }
+
+        // Once we've cloned the body of the callee in at the call site,
+        // there is no reason to keep around the original `call` instruction,
+        // so we remove it.
+        //
+        call->removeAndDeallocate();
+    }
+};
+
+    /// An inlining pass that inlines calls to `[unsafeForceInlineEarly]` functions
+struct MandatoryEarlyInliningPass : InliningPassBase
+{
+    typedef InliningPassBase Super;
+
+    MandatoryEarlyInliningPass(IRModule* module)
+        : Super(module)
+    {}
+
+    bool shouldInline(CallSiteInfo const& info)
+    {
+        if(info.callee->findDecoration<IRUnsafeForceInlineEarlyDecoration>())
+            return true;
+        return false;
+    }
+};
+
+
+void performMandatoryEarlyInlining(IRModule* module)
+{
+    MandatoryEarlyInliningPass pass(module);
+    pass.considerAllCallSites();
+}
+
+} // namespace Slang
diff --git a/source/slang/slang-ir-inline.h b/source/slang/slang-ir-inline.h
new file mode 100644
index 000000000..322b4c855
--- /dev/null
+++ b/source/slang/slang-ir-inline.h
@@ -0,0 +1,10 @@
+// slang-ir-inline.h
+#pragma once
+
+namespace Slang
+{
+    struct IRModule;
+
+        /// Inline any call sites to functions marked `[unsafeForceInlineEarly]`
+    void performMandatoryEarlyInlining(IRModule* module);
+}
diff --git a/source/slang/slang-ir-inst-defs.h b/source/slang/slang-ir-inst-defs.h
index 3fdf9f113..c6aaf57ca 100644
--- a/source/slang/slang-ir-inst-defs.h
+++ b/source/slang/slang-ir-inst-defs.h
@@ -448,6 +448,9 @@ INST(HighLevelDeclDecoration,               highLevelDecl,          1, 0)
         /// A `[format(f)]` decoration specifies that the format of an image should be `f`
     INST(FormatDecoration, format, 1, 0)
 
+        /// An `[unsafeForceInlineEarly]` decoration specifies that calls to this function should be inline after initial codegen
+    INST(UnsafeForceInlineEarlyDecoration, unsafeForceInlineEarly, 0, 0)
+
     /* LinkageDecoration */
         INST(ImportDecoration, import, 1, 0)
         INST(ExportDecoration, export, 1, 0)
diff --git a/source/slang/slang-ir-insts.h b/source/slang/slang-ir-insts.h
index 76d2a8940..448bf9f0e 100644
--- a/source/slang/slang-ir-insts.h
+++ b/source/slang/slang-ir-insts.h
@@ -345,6 +345,9 @@ struct IRFormatDecoration : IRDecoration
     }
 };
 
+IR_SIMPLE_DECORATION(UnsafeForceInlineEarlyDecoration)
+
+
 // An instruction that specializes another IR value
 // (representing a generic) to a particular set of generic arguments 
 // (instructions representing types, witness tables, etc.)
diff --git a/source/slang/slang-ir.cpp b/source/slang/slang-ir.cpp
index f84300327..45cbd65ff 100644
--- a/source/slang/slang-ir.cpp
+++ b/source/slang/slang-ir.cpp
@@ -5027,6 +5027,21 @@ namespace Slang
         return val;
     }
 
+    IRGeneric* findSpecializedGeneric(IRSpecialize* specialize)
+    {
+        return as<IRGeneric>(specialize->getBase());
+    }
+
+
+    IRInst* findSpecializeReturnVal(IRSpecialize* specialize)
+    {
+        auto generic = findSpecializedGeneric(specialize);
+        if(!generic)
+            return nullptr;
+
+        return findGenericReturnVal(generic);
+    }
+
     IRInst* getResolvedInstForDecorations(IRInst* inst)
     {
         IRInst* candidate = inst;
diff --git a/source/slang/slang-ir.h b/source/slang/slang-ir.h
index 06dd0f94c..0ca0093f2 100644
--- a/source/slang/slang-ir.h
+++ b/source/slang/slang-ir.h
@@ -1249,6 +1249,10 @@ struct IRGeneric : IRGlobalValueWithParams
 // a pass can glean information from it.
 IRInst* findGenericReturnVal(IRGeneric* generic);
 
+struct IRSpecialize;
+IRGeneric* findSpecializedGeneric(IRSpecialize* specialize);
+IRInst* findSpecializeReturnVal(IRSpecialize* specialize);
+
 // Resolve an instruction that might reference a static definition
 // to the most specific IR node possible, so that we can read
 // decorations from it (e.g., if this is a `specialize` instruction,
diff --git a/source/slang/slang-lower-to-ir.cpp b/source/slang/slang-lower-to-ir.cpp
index 12d7245b8..f57e15bc4 100644
--- a/source/slang/slang-lower-to-ir.cpp
+++ b/source/slang/slang-lower-to-ir.cpp
@@ -8,6 +8,7 @@
 #include "slang-compound-intrinsics.h"
 #include "slang-ir-constexpr.h"
 #include "slang-ir-dce.h"
+#include "slang-ir-inline.h"
 #include "slang-ir-insts.h"
 #include "slang-ir-missing-return.h"
 #include "slang-ir-sccp.h"
@@ -474,6 +475,14 @@ bool isImportedDecl(IRGenContext* context, Decl* decl)
     return false;
 }
 
+static bool isInline(Decl* decl)
+{
+    if(decl->HasModifier<UnsafeForceInlineEarlyAttribute>())
+        return true;
+
+    return false;
+}
+
     /// Should the given `decl` nested in `parentDecl` be treated as a static rather than instance declaration?
 bool isEffectivelyStatic(
     Decl*           decl,
@@ -746,15 +755,6 @@ LoweredValInfo emitCallToDeclRef(
             case kCompoundIntrinsicOp_Pos:
                 return LoweredValInfo::simple(args[0]);
 
-            case kCompoundIntrinsicOp_Sequence:
-                // The main effect of "operator comma" is to enforce
-                // sequencing of its operands, but Slang already
-                // implements a strictly left-to-right evaluation
-                // order for function arguments, so in practice we
-                // just need to compile `a, b` to the value of `b`
-                // (because argument evaluation already happened).
-                return LoweredValInfo::simple(args[1]);
-
 #define CASE(COMPOUND, OP)  \
             case COMPOUND: return emitCompoundAssignOp(context, type, OP, argCount, args)
 
@@ -6042,7 +6042,7 @@ struct DeclLoweringVisitor : DeclVisitor<DeclLoweringVisitor, LoweredValInfo>
 
         subBuilder->setInsertInto(irFunc);
 
-        if (isImportedDecl(decl))
+        if (isImportedDecl(decl) && !isInline(decl))
         {
             // Always emit imported declarations as declarations,
             // and not definitions.
@@ -6358,6 +6358,11 @@ struct DeclLoweringVisitor : DeclVisitor<DeclLoweringVisitor, LoweredValInfo>
             getBuilder()->addDecoration(irFunc, kIROp_OutputControlPointsDecoration, intLit);
         }
 
+        if(decl->FindModifier<UnsafeForceInlineEarlyAttribute>())
+        {
+            getBuilder()->addDecoration(irFunc, kIROp_UnsafeForceInlineEarlyDecoration);
+        }
+
         // For convenience, ensure that any additional global
         // values that were emitted while outputting the function
         // body appear before the function itself in the list
@@ -6913,7 +6918,12 @@ IRModule* generateIRForTranslationUnit(
 
     //      dumpIR(module);
 
-    // First, attempt to promote local variables to SSA
+    // First, inline calls to any functions that have been
+    // marked for mandatory "early" inlining.
+    //
+    performMandatoryEarlyInlining(module);
+
+    // Next, attempt to promote local variables to SSA
     // temporaries whenever possible.
     constructSSA(module);
 
diff --git a/source/slang/slang-modifier-defs.h b/source/slang/slang-modifier-defs.h
index aa9b25d00..58f2c5af9 100644
--- a/source/slang/slang-modifier-defs.h
+++ b/source/slang/slang-modifier-defs.h
@@ -490,3 +490,8 @@ END_SYNTAX_CLASS()
 // A `[__extern]` attribute, which indicates that a function/type is defined externally
 //
 SIMPLE_SYNTAX_CLASS(ExternAttribute, Attribute)
+
+// An `[__unsafeForceInlineExternal]` attribute indicates that the callee should be inlined
+// into call sites after initial IR generation (that is, as early as possible).
+//
+SIMPLE_SYNTAX_CLASS(UnsafeForceInlineEarlyAttribute, Attribute)
\ No newline at end of file
diff --git a/source/slang/slang.vcxproj b/source/slang/slang.vcxproj
index 2f55fffdc..0d702d9f5 100644
--- a/source/slang/slang.vcxproj
+++ b/source/slang/slang.vcxproj
@@ -217,6 +217,7 @@
     <ClInclude Include="slang-ir-dominators.h" />
     <ClInclude Include="slang-ir-entry-point-uniforms.h" />
     <ClInclude Include="slang-ir-glsl-legalize.h" />
+    <ClInclude Include="slang-ir-inline.h" />
     <ClInclude Include="slang-ir-inst-defs.h" />
     <ClInclude Include="slang-ir-insts.h" />
     <ClInclude Include="slang-ir-link.h" />
@@ -302,6 +303,7 @@
     <ClCompile Include="slang-ir-dominators.cpp" />
     <ClCompile Include="slang-ir-entry-point-uniforms.cpp" />
     <ClCompile Include="slang-ir-glsl-legalize.cpp" />
+    <ClCompile Include="slang-ir-inline.cpp" />
     <ClCompile Include="slang-ir-legalize-types.cpp" />
     <ClCompile Include="slang-ir-link.cpp" />
     <ClCompile Include="slang-ir-missing-return.cpp" />
diff --git a/source/slang/slang.vcxproj.filters b/source/slang/slang.vcxproj.filters
index 7a0f465fe..a73a74756 100644
--- a/source/slang/slang.vcxproj.filters
+++ b/source/slang/slang.vcxproj.filters
@@ -252,6 +252,9 @@
     <ClInclude Include="slang-ir-strip-witness-tables.h">
       <Filter>Header Files</Filter>
     </ClInclude>
+    <ClInclude Include="slang-ir-inline.h">
+      <Filter>Header Files</Filter>
+    </ClInclude>
   </ItemGroup>
   <ItemGroup>
     <ClCompile Include="slang-check-conformance.cpp">
@@ -473,6 +476,9 @@
     <ClCompile Include="slang-ir-strip-witness-tables.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
+    <ClCompile Include="slang-ir-inline.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
   </ItemGroup>
   <ItemGroup>
     <None Include="..\core\core.natvis">
diff --git a/tests/compute/comma-operator.slang b/tests/compute/comma-operator.slang
new file mode 100644
index 000000000..1036482ef
--- /dev/null
+++ b/tests/compute/comma-operator.slang
@@ -0,0 +1,35 @@
+// comma-operator.slang
+
+//TEST(compute):COMPARE_COMPUTE:-cpu
+//TEST(compute):COMPARE_COMPUTE:
+
+// Test that the "comma operator" behaves as expected
+
+// We will also include a cross-compilation test just to
+// confirm that the generated SPIR-V (and by extension DXIL/DXBC)
+// doesn't include a subroutine defintion/call for `operator,`
+
+//TEST:CROSS_COMPILE:-target spirv-assembly -entry computeMain -stage compute
+
+int test(int inVal)
+{
+	int a = inVal;
+
+	// We expect the left operand to `operator,` to be
+	// evaluated before the right operand, and for the
+	// result to be the value of the right operand
+	//
+	return a*=2, a+1;
+}
+
+//TEST_INPUT:ubuffer(data=[0 1 2 3], stride=4):out,name=outputBuffer
+RWStructuredBuffer<int> outputBuffer;
+
+[numthreads(4, 1, 1)]
+void computeMain(uint3 dispatchThreadID : SV_DispatchThreadID)
+{
+	uint tid = dispatchThreadID.x;
+	int inVal = outputBuffer[tid];
+	int outVal = test(inVal);
+	outputBuffer[tid] = outVal;
+}
\ No newline at end of file
diff --git a/tests/compute/comma-operator.slang.expected.txt b/tests/compute/comma-operator.slang.expected.txt
new file mode 100644
index 000000000..9b4237ab1
--- /dev/null
+++ b/tests/compute/comma-operator.slang.expected.txt
@@ -0,0 +1,4 @@
+1
+3
+5
+7
diff --git a/tests/compute/comma-operator.slang.glsl b/tests/compute/comma-operator.slang.glsl
new file mode 100644
index 000000000..3107f5773
--- /dev/null
+++ b/tests/compute/comma-operator.slang.glsl
@@ -0,0 +1,24 @@
+// comma-operator.slang.glsl
+#version 450
+
+//TEST_IGNORE_FILE:
+
+layout(std430, binding = 0)
+buffer _S1 {
+    int _data[];
+} outputBuffer_0;
+
+int test_0(int inVal_0)
+{
+    return inVal_0 * 2 + 1;
+}
+
+layout(local_size_x = 4, local_size_y = 1, local_size_z = 1) in;void main()
+{
+    uint tid_0 = gl_GlobalInvocationID.x;
+
+    int outVal_0 = test_0(outputBuffer_0._data[tid_0]);
+
+    outputBuffer_0._data[tid_0] = outVal_0;
+    return;
+}