First pass at support for cross-compilation

This is a large change that contains many pieces:

- Update the `cross-compile0` test to actually make use of cross compilation.

  Now the `cross-compile0.hlsl` file contains both HLSL and GLSL source code, and then imports code from `cross-compile0.slang`, which provides a "library" (one function) that can be shared between both the HLSL and GLSL version of things.

- Fixed a bug in the support for backslash-escaped newlines.

- Added a new `__import` declaration type (replaces the `using` directive that was still around in a vestigial form)

  An `__import` causes the compiler to look for a Slang source file (currently using the ordinary `#include` lookup logic), and then parse/check the found file as an additional module ("translation unit"), before making its declarations visible in the current scope.

- Refactored the main compilation flow to be simpler. There were the `ShaderCompiler` and `ShaderCompilerImpl` classes that weren't relaly doing anything, but added complexity to the whole workflow.

- The `render-test` application has been heavily modified to better support testing cross-compilation workflows. At the most basic level we are starting to distinguish pass-through vs. rewriter workflows, and are passing various `#define`s down to the compiler(s) to let the source code be customized as needed for each case.

  Several annoying corner cases are caused here by having to support the GLSL compilation model, which really wants each entry point in its own specific translation unit, whereas we really want to keep things nicely contained in single files.

- Added support for `__intrinsic` operations to have target-specific behavior.

  This allows a function to be given a different name for some specific target (so a call gets emitted as a call to that other operation).
More generally, the library writer can put together an arbitrary format string that will be used in place of expressions that call the given function, e.g.:

      __intrinsic(hlsl, "$1 - $0") __intrinsic int foo(int a, int b);

  Given this declaration, a call like `foo(x,y)` will code generate as `x - y` for HLSL, and as `foo(x,y)` for all other targets.

Annoying things still to be dealt with:

- The way that I'm filtering the user-provided options when passing things down to the compilation of dynamically loaded modules is a bit ad hoc. It would be good to have a systematic notion of which options will be inherited and which won't. There is also more code duplication than I'd like, so we risk having the compiler behave differently when compiling a file at the top level, vs. because of `__import`.

- Adding target-specific behavior to intrinsics is all well and good, but the current approach means we can only add this to the original declaration, which limits the ability to easily extend the set of targets.

  A better approach long-term would be to add a more robust notion of target-based overload resolution (which would happen after semantic checking). Then one mechanism would be used to find the right target-specific overload to use for an operation, and then each (target-specific) definition could use a simpler attribute to intercept code-generation behavior.

  Note that we might eventually need a similar notion to deal with stage- or profile-specific functions and the overloading behavior around them, so using this for intrinsics doesn't seem like a bad idea.

6 files changed, 158 insertions, 25 deletions

diff --git a/tools/render-test/main.cpp b/tools/render-test/main.cpp
index 2746c505d..aeff565c6 100644
--- a/tools/render-test/main.cpp
+++ b/tools/render-test/main.cpp
@@ -298,10 +298,12 @@ int main(
     {
     case Mode::Slang:
     case Mode::HLSL:
+    case Mode::HLSLRewrite:
         renderer = createD3D11Renderer();
         break;
 
     case Mode::GLSL:
+    case Mode::GLSLRewrite:
     case Mode::GLSLCrossCompile:
         renderer = createGLRenderer();
         break;
@@ -318,11 +320,19 @@ int main(
     switch( gOptions.mode )
     {
     case Mode::Slang:
-        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_HLSL);
+        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_SOURCE_LANGUAGE_SLANG, SLANG_HLSL);
+        break;
+
+    case Mode::HLSLRewrite:
+        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_SOURCE_LANGUAGE_HLSL, SLANG_HLSL);
+        break;
+
+    case Mode::GLSLRewrite:
+        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_SOURCE_LANGUAGE_GLSL, SLANG_GLSL);
         break;
 
     case Mode::GLSLCrossCompile:
-        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_GLSL);
+        shaderCompiler = createSlangShaderCompiler(shaderCompiler, SLANG_SOURCE_LANGUAGE_SLANG, SLANG_GLSL);
         break;
 
     default:
diff --git a/tools/render-test/options.cpp b/tools/render-test/options.cpp
index 260e6f46c..a486c8b15 100644
--- a/tools/render-test/options.cpp
+++ b/tools/render-test/options.cpp
@@ -60,6 +60,14 @@ void parseOptions(int* argc, char** argv)
         {
             gOptions.mode = Mode::GLSL;
         }
+        else if( strcmp(arg, "-hlsl-rewrite") == 0 )
+        {
+            gOptions.mode = Mode::HLSLRewrite;
+        }
+        else if( strcmp(arg, "-glsl-rewrite") == 0 )
+        {
+            gOptions.mode = Mode::GLSLRewrite;
+        }
         else if( strcmp(arg, "-slang") == 0 )
         {
             gOptions.mode = Mode::Slang;
diff --git a/tools/render-test/options.h b/tools/render-test/options.h
index 81617630f..0731a5dc9 100644
--- a/tools/render-test/options.h
+++ b/tools/render-test/options.h
@@ -10,9 +10,18 @@ typedef uintptr_t UInt;
 
 enum class Mode
 {
+    // Slang being used as an HLSL-ish compiler
     Slang,
+
+    // Raw HLSL or GLSL input, bypassing Slang
     HLSL,
     GLSL,
+
+    // Raw HLSL or GLSL input, passed through the Slang rewriter
+    HLSLRewrite,
+    GLSLRewrite,
+
+    // Slang/HLSL input -> GLSL output
     GLSLCrossCompile,
 };
 
diff --git a/tools/render-test/render-gl.cpp b/tools/render-test/render-gl.cpp
index ea56a10b4..d6166fc3c 100644
--- a/tools/render-test/render-gl.cpp
+++ b/tools/render-test/render-gl.cpp
@@ -135,7 +135,6 @@ public:
         switch (type)
         {
         case GL_DEBUG_TYPE_ERROR:
-            assert(!"unexpected");
             break;
 
         default:
@@ -471,7 +470,48 @@ public:
 
     GLuint loadShader(GLenum stage, char const* source)
     {
-        auto shaderID = glCreateShader(stage);
+
+        // GLSL in monumentally stupid. It officially requires the `#version` directive
+        // to be the first thing in the file, which wouldn't be so bad but the API
+        // doesn't provide a way to pass a `#define` into your shader other than by
+        // prepending it to the whole thing.
+        //
+        // We are going to solve this problem by doing some surgery on the source
+        // that was passed in.
+
+        char const* sourceBegin = source;
+        char const* sourceEnd = source + strlen(source);
+
+        // Look for a version directive in the user-provided source.
+        char const* versionBegin = strstr(source, "#version");
+        char const* versionEnd = nullptr;
+        if( versionBegin )
+        {
+            // If we found a directive, then scan for the end-of-line
+            // after it, and use that to specify the slice.
+            versionEnd = strchr(versionBegin, '\n');
+            if( !versionEnd )
+            {
+                versionEnd = sourceEnd;
+            }
+            else
+            {
+                versionEnd = versionEnd + 1;
+            }
+        }
+        else
+        {
+            // If we didn't find a directive, then treat it as being
+            // a zero-byte slice at the start of the string
+            versionBegin = sourceBegin;
+            versionEnd = sourceBegin;
+        }
+
+        enum { kMaxSourceStringCount = 16 };
+        GLchar const* sourceStrings[kMaxSourceStringCount];
+        GLint sourceStringLengths[kMaxSourceStringCount];
+
+        int sourceStringCount = 0;
 
         char const* stagePrelude = "\n";
         switch (stage)
@@ -492,18 +532,28 @@ public:
             "#define __GLSL__ 1\n"
             ;
 
-        char const* sourceStrings[] =
-        {
-            stagePrelude,
-            prelude,
-            source,
-        };
+#define ADD_SOURCE_STRING_SPAN(BEGIN, END)                              \
+        sourceStrings[sourceStringCount] = BEGIN;                       \
+        sourceStringLengths[sourceStringCount++] = GLint(END - BEGIN)   \
+        /* end */
+
+#define ADD_SOURCE_STRING(BEGIN)                                        \
+        sourceStrings[sourceStringCount] = BEGIN;                       \
+        sourceStringLengths[sourceStringCount++] = GLint(strlen(BEGIN)) \
+        /* end */
 
+        ADD_SOURCE_STRING_SPAN(versionBegin, versionEnd);
+        ADD_SOURCE_STRING(stagePrelude);
+        ADD_SOURCE_STRING(prelude);
+        ADD_SOURCE_STRING_SPAN(sourceBegin, versionBegin);
+        ADD_SOURCE_STRING_SPAN(versionEnd, sourceEnd);
+
+        auto shaderID = glCreateShader(stage);
         glShaderSource(
             shaderID,
-            sizeof(sourceStrings) / sizeof(sourceStrings[0]),
+            sourceStringCount,
             &sourceStrings[0],
-            nullptr);
+            &sourceStringLengths[0]);
         glCompileShader(shaderID);
 
         GLint success = GL_FALSE;
diff --git a/tools/render-test/slang-support.cpp b/tools/render-test/slang-support.cpp
index 06a11ad4c..bec651e87 100644
--- a/tools/render-test/slang-support.cpp
+++ b/tools/render-test/slang-support.cpp
@@ -12,6 +12,7 @@ struct SlangShaderCompilerWrapper : public ShaderCompiler
 {
     ShaderCompiler*     innerCompiler;
     SlangCompileTarget  target;
+    SlangSourceLanguage sourceLanguage;
 
     virtual ShaderProgram* compileProgram(ShaderCompileRequest const& request) override
     {
@@ -20,16 +21,58 @@ struct SlangShaderCompilerWrapper : public ShaderCompiler
 
         spSetCodeGenTarget(slangRequest, target);
 
-        // Define a macro so that shader code in a test can detect when it is being
-        // compiled as Slang source code.
-        spAddPreprocessorDefine(slangRequest, "__SLANG__", "1");
+        // Define a macro so that shader code in a test can detect what language we
+        // are nominally working with.
+        char const* langDefine = nullptr;
+        switch (sourceLanguage)
+        {
+        case SLANG_SOURCE_LANGUAGE_GLSL:    langDefine = "__GLSL__";    break;
+        case SLANG_SOURCE_LANGUAGE_HLSL:    langDefine = "__HLSL__";    break;
+        case SLANG_SOURCE_LANGUAGE_SLANG:   langDefine = "__SLANG__";   break;
+        default:
+            assert(!"unexpected");
+            break;
+        }
+        spAddPreprocessorDefine(slangRequest, langDefine, "1");
+
+        int vertexTranslationUnit = 0;
+        int fragmentTranslationUnit = 0;
+        if( sourceLanguage == SLANG_SOURCE_LANGUAGE_GLSL )
+        {
+            // GLSL presents unique challenges because, frankly, it got the whole
+            // compilation model wrong. One aspect of working around this is that
+            // we will compile the same source file multiple times: once per
+            // entry point, and we will have different preprocessor definitions
+            // active in each case.
+
+            vertexTranslationUnit = spAddTranslationUnit(slangRequest, sourceLanguage, nullptr);
+            spAddTranslationUnitSourceString(slangRequest, vertexTranslationUnit, request.source.path, request.source.text);
 
-        int translationUnitIndex = spAddTranslationUnit(slangRequest, SLANG_SOURCE_LANGUAGE_SLANG, nullptr);
+            spTranslationUnit_addPreprocessorDefine(slangRequest, vertexTranslationUnit, "__GLSL_VERTEX__", "1");
 
-        spAddTranslationUnitSourceString(slangRequest, translationUnitIndex, request.source.path, request.source.text);
+            fragmentTranslationUnit = spAddTranslationUnit(slangRequest, sourceLanguage, nullptr);
+            spAddTranslationUnitSourceString(slangRequest, fragmentTranslationUnit, request.source.path, request.source.text);
+
+            spTranslationUnit_addPreprocessorDefine(slangRequest, fragmentTranslationUnit, "__GLSL_FRAGMENT__", "1");
+        }
+        else
+        {
+            int translationUnit = spAddTranslationUnit(slangRequest, sourceLanguage, nullptr);
+            spAddTranslationUnitSourceString(slangRequest, translationUnit, request.source.path, request.source.text);
+
+            vertexTranslationUnit = translationUnit;
+            fragmentTranslationUnit = translationUnit;
+        }
 
-        int vertexEntryPoint = spAddTranslationUnitEntryPoint(slangRequest, translationUnitIndex, request.vertexShader.name,   spFindProfile(slangSession, request.vertexShader.profile));
-        int fragmentEntryPoint = spAddTranslationUnitEntryPoint(slangRequest, translationUnitIndex, request.fragmentShader.name, spFindProfile(slangSession, request.fragmentShader.profile));
+
+        // If we aren't dealing with true Slang input, then don't enable checking.
+        if (sourceLanguage != SLANG_SOURCE_LANGUAGE_SLANG)
+        {
+            spSetCompileFlags(slangRequest, SLANG_COMPILE_FLAG_NO_CHECKING);
+        }
+
+        int vertexEntryPoint = spAddTranslationUnitEntryPoint(slangRequest, vertexTranslationUnit, request.vertexShader.name,   spFindProfile(slangSession, request.vertexShader.profile));
+        int fragmentEntryPoint = spAddTranslationUnitEntryPoint(slangRequest, fragmentTranslationUnit, request.fragmentShader.name, spFindProfile(slangSession, request.fragmentShader.profile));
 
         int compileErr = spCompile(slangRequest);
         if(auto diagnostics = spGetDiagnosticOutput(slangRequest))
@@ -43,12 +86,18 @@ struct SlangShaderCompilerWrapper : public ShaderCompiler
             return nullptr;
         }
 
-        char const* translatedCode = spGetTranslationUnitSource(slangRequest, translationUnitIndex);
-        char const* vertexCode = spGetEntryPointSource(slangRequest, translationUnitIndex, vertexEntryPoint);
-        char const* fragmentCode = spGetEntryPointSource(slangRequest, translationUnitIndex, fragmentEntryPoint);
 
         ShaderCompileRequest innerRequest = request;
-        innerRequest.source.text = translatedCode;
+
+        if( sourceLanguage != SLANG_SOURCE_LANGUAGE_GLSL )
+        {
+            char const* translatedCode = spGetTranslationUnitSource(slangRequest, 0);
+            innerRequest.source.text = translatedCode;
+        }
+
+        char const* vertexCode = spGetEntryPointSource(slangRequest, vertexTranslationUnit, vertexEntryPoint);
+        char const* fragmentCode = spGetEntryPointSource(slangRequest, fragmentTranslationUnit, fragmentEntryPoint);
+
         innerRequest.vertexShader.source.text = vertexCode;
         innerRequest.fragmentShader.source.text = fragmentCode;
 
@@ -66,10 +115,14 @@ struct SlangShaderCompilerWrapper : public ShaderCompiler
     }
 };
 
-ShaderCompiler* createSlangShaderCompiler(ShaderCompiler* innerCompiler, SlangCompileTarget target)
+ShaderCompiler* createSlangShaderCompiler(
+    ShaderCompiler*     innerCompiler,
+    SlangSourceLanguage sourceLanguage,
+    SlangCompileTarget  target)
 {
     auto result = new SlangShaderCompilerWrapper();
     result->innerCompiler = innerCompiler;
+    result->sourceLanguage = sourceLanguage;
     result->target = target;
 
     return result;
diff --git a/tools/render-test/slang-support.h b/tools/render-test/slang-support.h
index a191fbfef..d2e32c52a 100644
--- a/tools/render-test/slang-support.h
+++ b/tools/render-test/slang-support.h
@@ -7,6 +7,9 @@
 
 namespace renderer_test {
 
-ShaderCompiler* createSlangShaderCompiler(ShaderCompiler* innerCompiler, SlangCompileTarget target);
+ShaderCompiler* createSlangShaderCompiler(
+    ShaderCompiler*     innerCompiler,
+    SlangSourceLanguage sourceLanguage,
+    SlangCompileTarget  target);
 
 } // renderer_test