Refactor: eliminate BackEndCompileRequest (#2178)

An earlier refactoring pass over the compiler codebase split the
type that had been called `CompileRequest` into three distinct
pieces:

* `FrontEndCompileRequest` which was supposed to own state and
  options related to running the compiler front end and producing
  IR + reflection (e.g., what translation units and source
  files/strings are included).

* `BackEndCompileRequest` which was supposed to own state and options
  related to running the compiler back end to translate the IR
  for a `ComponentType` (program) into output code. (Note that the
  `BackEndCompileRequest` was conceived of as orthogonal to the
  `TargetRequest`s, which store per-target and target-specific
  options.)

* `EndToEndCompileRequest` which was an umbrella object that owns
  separate front-end and back-end requests, plus any state that is
  only relevant when doing a true end-to-end compile (such as the
  kinds of compiles initiated with `slangc`). As originally conceived,
  the only state that this type was supposed to own was stuff related
  to "pass-through" compilation, as well as state related to writing
  of generated code to output files.

That refactoring work was very useful at the time, because it allowed
us to "scrub" the back end compilation steps to remove all
dependencies on front-end and AST state (this was important for our
goals of enabling linking and codegen from serialized Slang IR).

At this point, however, it is clear that the hierarchy that was built
up serves very little purpose:

* The `BackEndCompileRequest` type is only used in two places:

    * As part of an `EndToEndCompileRequest`, where the settings on
      the `BackEndCompileRequest` can be configured, but only through
      the `EndToEndCompileRequest`

    * As part of on-demand code generation through the `IComponentType`
      APIs. In this case, the settings stored on the
      `BackEndCompileRequest` are not accessible to the application
      at all, and will always use their default values, so that
      instantiating a "request" object doesn't really make any sense.

* The `FrontEndCompileRequest` type has a similar situation:

    * Front-end compilation as part of an `EndToEndCompileRequest`
      supports user configuration of `FrontEndCompileRequest` settings,
      but only through the `EndToEndCompileRequest`

    * Front-end compilation triggered by an `import` or a `loadModule()`
      call does not support user configuration of settings at all. It
      will always derive all relevant settings from thsoe on the
      session ("linkage").

In addition, subsequent changes have been made to the compiler that
show a bit of a "code smell" and/or forward-looking worries for this
decomposition:

* In some cases we've had to add the same setting to multiple types
  in the breakdown (front-end, back-end, end-to-end, linkage, target,
  etc.) which makes it harder for us to validate that all the possible
  mixtures of state work correctly.

* Related to the above, in some cases we have manual logic that copies
  state from one of the objects in the breakdown to another, in order
  to ensure that the user's intention is actually followed.

* As a forward-looking concern, it seems that developers have sometimes
  added new configuration options and state to places that don't really
  make sense according to the rationale of the original decomposition
  (e.g., we probably don't want to have a lot of state that is only
  available via end-to-end requests, given that the API structure is
  meant to push users *away* from end-to-end compiles).

As a result of all of the above, I've been planning a large refactor
with the following big-picture goals:

* Eliminate `BackEndCompileRequest`

    * Move all relevant state/options from the back-end request to
      the end-to-end request, since that is the only place they could
      be set anyway.

    * Introduce a transient "context" type to be used for the duration
      of code generation that serves the main functions that back-end
      requests really served in the codebase

* Make `EndToEndCompileRequest` be a subclass of
  `FrontEndCompileRequest`

    * Consider addding a transient "context" type for front-end
      compiles that can be used in `import`-like cases rather than
      needing a full front-end request object. If this works, then
      eliminate `FrontEndCompileRequest` and be back to world with
      just a single `CompileRequest` type

* Move *all* compiler configuration options to a distinct type (named
  something like `CompilerConfig` or `CompilerOptions` or whatever)
  which stores setting as key-value pairs, and has a notion of
  "inheritance" such that one configuration can extend or build on top
  of another. Make all the relevant types use this catch-all structure
  instead of redundantly storing flags in many places.

This change deals with the first of those bullets: removeal of
`BackEndCompileRequest`. The addition of the `CodeGenContext` type is
perhaps an unncessary additional step, but making that change helps
clean up a bunch of the code related to per-target code generation,
so I think it is the right choice.

Co-authored-by: Yong He <yonghe@outlook.com>

1 files changed, 283 insertions, 87 deletions

diff --git a/source/slang/slang-compiler.h b/source/slang/slang-compiler.h
index 7f8257777..bf715d5d4 100755
--- a/source/slang/slang-compiler.h
+++ b/source/slang/slang-compiler.h
@@ -133,11 +133,11 @@ namespace Slang
         Maximal = SLANG_OPTIMIZATION_LEVEL_MAXIMAL,
     };
 
+    struct CodeGenContext;
+    class EndToEndCompileRequest;
+    class FrontEndCompileRequest;
     class Linkage;
     class Module;
-    class FrontEndCompileRequest;
-    class BackEndCompileRequest;
-    class EndToEndCompileRequest;
     class TranslationUnitRequest;
 
     // Result of compiling an entry point.
@@ -2143,8 +2143,9 @@ namespace Slang
         }
 
         CompileResult& _createWholeProgramResult(
-            BackEndCompileRequest*  backEndRequest,
-            EndToEndCompileRequest* endToEndRequest);
+            DiagnosticSink*         sink,
+            EndToEndCompileRequest* endToEndReq = nullptr);
+
             /// Internal helper for `getOrCreateEntryPointResult`.
             ///
             /// This is used so that command-line and API-based
@@ -2154,8 +2155,8 @@ namespace Slang
             ///
         CompileResult& _createEntryPointResult(
             Int                     entryPointIndex,
-            BackEndCompileRequest*  backEndRequest,
-            EndToEndCompileRequest* endToEndRequest);
+            DiagnosticSink*         sink,
+            EndToEndCompileRequest* endToEndReq = nullptr);
 
         RefPtr<IRModule> getOrCreateIRModuleForLayout(DiagnosticSink* sink);
 
@@ -2185,40 +2186,257 @@ namespace Slang
         RefPtr<IRModule> m_irModuleForLayout;
     };
 
-        /// A request to generate code for a program
-    class BackEndCompileRequest : public CompileRequestBase
+        /// A back-end-specific object to track optional feaures/capabilities/extensions
+        /// that are discovered to be used by a program/kernel as part of code generation.
+    class ExtensionTracker : public RefObject
     {
+        // TODO: The existence of this type is evidence of a design/architecture problem.
+        //
+        // A better formulation of things requires a few key changes:
+        //
+        // 1. All optional capabilities need to be enumerated as part of the `CapabilitySet`
+        //  system, so that they can be reasoned about uniformly across different targets
+        //  and different layers of the compiler.
+        //
+        // 2. The front-end should be responsible for either or both of:
+        //
+        //   * Checking that `public` or otherwise externally-visible items (declarations/definitions)
+        //     explicitly declare the capabilities they require, and that they only ever
+        //     make use of items that are comatible with those required capabilities.
+        //
+        //   * Inferring the capabilities required by items that are not externally visible,
+        //     and attaching those capabilities explicit as a modifier or other synthesized AST node.
+        //
+        // 3. The capabilities required by a given `ComponentType` and its entry points should be
+        // explicitly know-able, and they should be something we can compare to the capabilities
+        // of a code generation target *before* back-end code generation is started. We should be
+        // able to issue error messages around lacking capabilities in a way the user can understand,
+        // in terms of the high-level-language entities.
+
     public:
-        BackEndCompileRequest(
-            Linkage*        linkage,
-            DiagnosticSink* sink,
-            ComponentType*  program = nullptr);
+    };
 
-        // Should we dump intermediate results along the way, for debugging?
-        bool shouldDumpIntermediates = false;
+        /// A context for code generation in the compiler back-end
+    struct CodeGenContext
+    {
+    public:
+        typedef List<Index> EntryPointIndices;
 
-        ComponentType* getProgram() { return m_program; }
-        void setProgram(ComponentType* program) { m_program = program; }
+        struct Shared
+        {
+        public:
+            Shared(
+                TargetProgram*              targetProgram,
+                EntryPointIndices const&    entryPointIndices,
+                DiagnosticSink*             sink,
+                EndToEndCompileRequest*     endToEndReq)
+                : targetProgram(targetProgram)
+                , entryPointIndices(entryPointIndices)
+                , sink(sink)
+                , endToEndReq(endToEndReq)
+            {}
+
+//            Shared(
+//                TargetProgram*              targetProgram,
+//                EndToEndCompileRequest*     endToEndReq);
+
+            TargetProgram*          targetProgram = nullptr;
+            EntryPointIndices       entryPointIndices;
+            DiagnosticSink*         sink = nullptr;
+            EndToEndCompileRequest* endToEndReq = nullptr;
+        };
+
+        CodeGenContext(
+            Shared* shared)
+            : m_shared(shared)
+            , m_targetFormat(shared->targetProgram->getTargetReq()->getTarget())
+        {}
+
+        CodeGenContext(
+            CodeGenContext*     base,
+            CodeGenTarget       targetFormat,
+            ExtensionTracker*   extensionTracker = nullptr)
+            : m_shared(base->m_shared)
+            , m_targetFormat(targetFormat)
+            , m_extensionTracker(extensionTracker)
+        {}
+
+            /// Get the diagnostic sink
+        DiagnosticSink* getSink()
+        {
+            return m_shared->sink;
+        }
+
+        TargetProgram* getTargetProgram()
+        {
+            return m_shared->targetProgram;
+        }
+
+        EntryPointIndices const& getEntryPointIndices()
+        {
+            return m_shared->entryPointIndices;
+        }
+
+        CodeGenTarget getTargetFormat()
+        {
+            return m_targetFormat;
+        }
+
+        ExtensionTracker* getExtensionTracker()
+        {
+            return m_extensionTracker;
+        }
+
+        TargetRequest* getTargetReq()
+        {
+            return getTargetProgram()->getTargetReq();
+        }
+
+        CapabilitySet getTargetCaps()
+        {
+            return getTargetReq()->getTargetCaps();
+        }
+
+        CodeGenTarget getFinalTargetFormat()
+        {
+            return getTargetReq()->getTarget();
+        }
+
+        ComponentType* getProgram()
+        {
+            return getTargetProgram()->getProgram();
+        }
+
+        Linkage* getLinkage()
+        {
+            return getProgram()->getLinkage();
+        }
+
+        Session* getSession()
+        {
+            return getLinkage()->getSessionImpl();
+        }
+
+            /// Get the source manager
+        SourceManager* getSourceManager()
+        {
+            return getLinkage()->getSourceManager();
+        }
+
+        ISlangFileSystemExt* getFileSystemExt()
+        {
+            return getLinkage()->getFileSystemExt();
+        }
+
+        EndToEndCompileRequest* isEndToEndCompile()
+        {
+            return m_shared->endToEndReq;
+        }
+
+        EndToEndCompileRequest* isPassThroughEnabled();
+
+        Count getEntryPointCount()
+        {
+            return getEntryPointIndices().getCount();
+        }
+
+        EntryPoint* getEntryPoint(Index index)
+        {
+            return getProgram()->getEntryPoint(index);
+        }
+
+        Index getSingleEntryPointIndex()
+        {
+            SLANG_ASSERT(getEntryPointCount() == 1);
+            return getEntryPointIndices()[0];
+        }
 
-        // Should R/W images without explicit formats be assumed to have "unknown" format?
         //
-        // The default behavior is to make a best-effort guess as to what format is intended.
+
+        IRDumpOptions getIRDumpOptions();
+
+        bool shouldValidateIR();
+        bool shouldDumpIR();
+
+        bool shouldDumpIntermediates();
+        String getIntermediateDumpPrefix();
+
+        bool getUseUnknownImageFormatAsDefault();
+
+        bool isSpecializationDisabled();
+
         //
-        bool useUnknownImageFormatAsDefault = false;
 
-        // If true will disable generics/existential value specialization pass.
-        bool disableSpecialization = false;
+        CompileResult emitEntryPoints();
 
-        // If true will disable generating dynamic dispatch code.
-        bool disableDynamicDispatch = false;
+        SlangResult dissassembleWithDownstream(
+            const void* data,
+            size_t          dataSizeInBytes,
+            ISlangBlob** outBlob);
 
-        // The default IR dumping options
-        IRDumpOptions m_irDumpOptions;
+        SlangResult dissassembleWithDownstream(
+            DownstreamCompileResult* downstreamResult,
+            ISlangBlob** outBlob);
 
-        String m_dumpIntermediatePrefix;
+    protected:
+        CodeGenTarget m_targetFormat = CodeGenTarget::Unknown;
+        ExtensionTracker* m_extensionTracker = nullptr;
+
+        // Helper to dump intermediate output when debugging
+        void maybeDumpIntermediate(
+            void const* data,
+            size_t      size);
+        void maybeDumpIntermediate(
+            char const* text);
+
+        void maybeDumpIntermediate(
+            DownstreamCompileResult* compileResult);
+
+        void dumpIntermediate(
+            void const* data,
+            size_t      size,
+            char const* ext,
+            bool        isBinary);
+
+        void dumpIntermediateText(
+            void const* data,
+            size_t      size,
+            char const* ext);
+
+        void dumpIntermediateBinary(
+            void const* data,
+            size_t      size,
+            char const* ext);
+
+        /* Emits entry point source taking into account if a pass-through or not. Uses 'targetFormat' to determine
+        the target (not targetReq) */
+        SlangResult emitEntryPointsSource(
+            String& outSource);
+
+        SlangResult emitEntryPointsSourceFromIR(
+            String& outSource);
+
+        SlangResult emitWithDownstreamForEntryPoints(
+            RefPtr<DownstreamCompileResult>& outResult);
+
+        /* Determines a suitable filename to identify the input for a given entry point being compiled.
+        If the end-to-end compile is a pass-through case, will attempt to find the (unique) source file
+        pathname for the translation unit containing the entry point at `entryPointIndex.
+        If the compilation is not in a pass-through case, then always returns `"slang-generated"`.
+        @param endToEndReq The end-to-end compile request which might be using pass-through compilation
+        @param entryPointIndex The index of the entry point to compute a filename for.
+        @return the appropriate source filename */
+        String calcSourcePathForEntryPoints();
+
+        TranslationUnitRequest* findPassThroughTranslationUnit(
+            Int                     entryPointIndex);
+
+
+        SlangResult _emitEntryPoints(
+            RefPtr<DownstreamCompileResult>& outDownstreamResult);
 
     private:
-        RefPtr<ComponentType> m_program;
+        Shared* m_shared = nullptr;
     };
 
         /// A compile request that spans the front and back ends of the compiler
@@ -2402,7 +2620,6 @@ namespace Slang
         NamePool* getNamePool() { return getLinkage()->getNamePool(); }
 
         FrontEndCompileRequest* getFrontEndReq() { return m_frontEndReq; }
-        BackEndCompileRequest* getBackEndReq() { return m_backEndReq; }
 
         ComponentType* getUnspecializedGlobalComponentType() { return getFrontEndReq()->getGlobalComponentType(); }
         ComponentType* getUnspecializedGlobalAndEntryPointsComponentType()
@@ -2422,10 +2639,47 @@ namespace Slang
             m_linkage = nullptr;
             m_frontEndReq = nullptr;
         }
+
+        void generateOutput();
+
+
+        // Note: The following settings used to be considered part of the "back-end" compile
+        // request, but were only being used as part of end-to-end compilation anyway,
+        // so they were moved here.
+        //
+
+        // Should we dump intermediate results along the way, for debugging?
+        bool shouldDumpIntermediates = false;
+
+        // Should R/W images without explicit formats be assumed to have "unknown" format?
+        //
+        // The default behavior is to make a best-effort guess as to what format is intended.
+        //
+        bool useUnknownImageFormatAsDefault = false;
+
+        // If true will disable generics/existential value specialization pass.
+        bool disableSpecialization = false;
+
+        // If true will disable generating dynamic dispatch code.
+        bool disableDynamicDispatch = false;
+
+        // The default IR dumping options
+//        IRDumpOptions m_irDumpOptions;
+
+        String m_dumpIntermediatePrefix;
+
     private:
+        void writeWholeProgramResult(
+            TargetRequest* targetReq);
+        void writeEntryPointResult(
+            TargetRequest* targetReq,
+            Int             entryPointIndex);
 
         ISlangUnknown* getInterface(const Guid& guid);
 
+        void generateOutput(ComponentType* program);
+        void generateOutput(TargetProgram* targetProgram);
+
         void init();
 
         Session*                        m_session = nullptr;
@@ -2435,35 +2689,12 @@ namespace Slang
         RefPtr<ComponentType>           m_specializedGlobalComponentType;
         RefPtr<ComponentType>           m_specializedGlobalAndEntryPointsComponentType;
         List<RefPtr<ComponentType>>     m_specializedEntryPoints;
-        RefPtr<BackEndCompileRequest>   m_backEndReq;
 
         // For output
 
         RefPtr<StdWriters> m_writers;
     };
 
-    void generateOutput(
-        BackEndCompileRequest* compileRequest);
-
-    void generateOutput(
-        EndToEndCompileRequest* compileRequest);
-
-    // Helper to dump intermediate output when debugging
-    void maybeDumpIntermediate(
-        BackEndCompileRequest* compileRequest,
-        void const*     data,
-        size_t          size,
-        CodeGenTarget   target);
-    void maybeDumpIntermediate(
-        BackEndCompileRequest* compileRequest,
-        char const*     text,
-        CodeGenTarget   target);
-
-    void maybeDumpIntermediate(
-        BackEndCompileRequest* compileRequest,
-        DownstreamCompileResult* compileResult,
-        CodeGenTarget   target);
-
     /* Returns SLANG_OK if pass through support is available */
     SlangResult checkExternalCompilerSupport(Session* session, PassThroughMode passThrough);
     /* Report an error appearing from external compiler to the diagnostic sink error to the diagnostic sink.
@@ -2473,41 +2704,6 @@ namespace Slang
     @param sink The diagnostic sink to report to */
     void reportExternalCompileError(const char* compilerName, SlangResult res, const UnownedStringSlice& diagnostic, DiagnosticSink* sink);
 
-    /* Determines a suitable filename to identify the input for a given entry point being compiled.
-    If the end-to-end compile is a pass-through case, will attempt to find the (unique) source file
-    pathname for the translation unit containing the entry point at `entryPointIndex.
-    If the compilation is not in a pass-through case, then always returns `"slang-generated"`.
-    @param endToEndReq The end-to-end compile request which might be using pass-through compilation
-    @param entryPointIndex The index of the entry point to compute a filename for.
-    @return the appropriate source filename */
-    String calcSourcePathForEntryPoint(EndToEndCompileRequest* endToEndReq, Int entryPointIndex);
-    String calcSourcePathForEntryPoints(EndToEndCompileRequest* endToEndReq, const List<Int>& entryPointIndices);
-
-    class ExtensionTracker : public RefObject
-    {
-    public:
-    };
-
-    /* Emits entry point source taking into account if a pass-through or not. Uses 'target' to determine
-    the target (not targetReq) */
-    SlangResult emitEntryPointsSource(
-        BackEndCompileRequest*  compileRequest,
-        const List<Int>&        entryPointIndices,
-        TargetRequest*          targetReq,
-        CodeGenTarget           target,
-        EndToEndCompileRequest* endToEndReq,
-        ExtensionTracker*       extensionTracker,
-        String&                 outSource);
-
-    SlangResult emitEntryPointSource(
-        BackEndCompileRequest*  compileRequest,
-        Int                     entryPointIndex,
-        TargetRequest*          targetReq,
-        CodeGenTarget           target,
-        EndToEndCompileRequest* endToEndReq,
-        ExtensionTracker*       extensionTracker,
-        String&                 outSource);
-
     //
 
     // Information about BaseType that's useful for checking literals