Option to build all slang-repro in a directory (#1911)

* #include an absolute path didn't work - because paths were taken to always be relative.

* Add repro-directory feature.

* Added writer support to repro-directory.

* Upgrade glslang to 11.5.0

* Add -load-repro-directory option

* Improve repro doc.

2 files changed, 99 insertions, 3 deletions

diff --git a/docs/repro.md b/docs/repro.md
index f339f429b..7a032df07 100644
--- a/docs/repro.md
+++ b/docs/repro.md
@@ -13,7 +13,8 @@ There are a few command line options
 * `-extract-repro [filename]` extracts the contents of the repro file. The contained files are placed in a directory with a name, the same as the repro file minus the extension. Also contains a 'manifest'.
 * `-load-repro [filename]` loads the repro and compiles using it's options. Note this must be the last arg on the command line.
 * `-dump-repro-on-error` if a compilation fails will attempt to save a repro (using a filename generated from first source filename)
-* `-repro-file-system` [filename] makes the repros file contents appear as the file system during a compilation. Does not set any compilation options.
+* `-repro-file-system [filename]` makes the repros file contents appear as the file system during a compilation. Does not set any compilation options.
+* `-load-repro-directory [directory]` compiles all of the .slang-repro files found in `directory`
 
 The `manifest` made available via `-extract-repro` provides some very useful information
 
@@ -23,9 +24,9 @@ The `manifest` made available via `-extract-repro` provides some very useful inf
 
 First it is worth just describing what is required to reproduce a compilation. Most straightforwardly the options setup for the compilation need to be stored. This would include any flags, and defines, include paths, entry points, input filenames and so forth. Also needed will be the contents of any files that were specified. This might be files on the file system, but could also be 'files' specified as strings through the slang API. Lastly we need any files that were referenced as part of the compilation - this could be include files, or module source files and so forth. All of this information is bundled up together into a file that can then later be loaded and compiled. This is broadly speaking all of the data that is stored within a repro file. 
 
-In order to capture a complete repro file a typically a compilation has to be attempted. The state before compilation can be recorded (through the API for example), but it may not be enough to repeat a compilation, as files referenced by the compilation would not yet have been accessed. The repro feature records all of these accesses and contents of such files such that compilation can either be completed or at least to the same point as was reached on the host machine. 
+In order to capture a complete repro file typically a compilation has to be attempted. The state before compilation can be recorded (through the API for example), but it may not be enough to repeat a compilation, as files referenced by the compilation would not yet have been accessed. The repro feature records all of these accesses and contents of such files such that compilation can either be completed or at least to the same point as was reached on the host machine. 
 
-One of the more subtle issues around reproducing a compilation is around filenames. That using the API a client can specify source files without names, or multiple files with the same name. If files are loaded via `ISlangFileSystem`, they are typically part of a hiearchical file system. This could mean they are referenced relatively. That there can be distinct files with the same name, differenciated by directory. That the files may not easily be reconstructed back into a similar hieararchical file system - as depending on the include paths (or perhaps other mechanisms) the 'files' and their contents could be arranged in a manner very hard to replicate. To work around this the repro feature does not attempt to replicate a hierarchical file system. Instead it gives every file a unique name, based on their original name. If there are multiple files with the same name it will 'uniquify' them by appending an index. Doing so means that the contents of the file system can just be held as a flat collection of files. This is not enough to enable repeating the compilation though, as we now need Slang to know which files to reference when they are requested, as they are now no longer part of a hierarchical file system and their names may have been altered. To achieve this the repro functionality stores off a map of all path requests to their contents (or lack there of). Doing so means that the file system still appears to Slang as it did in the original compilation, even with all the files being actually stored using the simpler 'flat' arrangement. 
+One of the more subtle issues around reproducing a compilation is around filenames. Using the API, a client can specify source files without names, or multiple files with the same name. If files are loaded via `ISlangFileSystem`, they are typically part of a hiearchical file system. This could mean they are referenced relatively. This means there can be distinct files with the same name but differenciated by directory. The files may not easily be reconstructed back into a similar hieararchical file system - as depending on the include paths (or perhaps other mechanisms) the 'files' and their contents could be arranged in a manner very hard to replicate. To work around this the repro feature does not attempt to replicate a hierarchical file system. Instead it gives every file a unique name based on their original name. If there are multiple files with the same name it will 'uniquify' them by appending an index. Doing so means that the contents of the file system can just be held as a flat collection of files. This is not enough to enable repeating the compilation though, as we now need Slang to know which files to reference when they are requested, as they are now no longer part of a hierarchical file system and their names may have been altered. To achieve this the repro functionality stores off a map of all path requests to their contents (or lack there of). Doing so means that the file system still appears to Slang as it did in the original compilation, even with all the files being actually stored using the simpler 'flat' arrangement. 
 
 This means that when a repro is 'extracted' it does so to a directory which holds the files with their unique 'flat' names. The name of the directory is the name of the repro file without it's extension, or if it has no extension, with the postfix '-files'. This directory will be referered to from now on as the `repro directory`.
 
diff --git a/source/slang/slang-options.cpp b/source/slang/slang-options.cpp
index 00372ca3c..73ccaab55 100644
--- a/source/slang/slang-options.cpp
+++ b/source/slang/slang-options.cpp
@@ -399,6 +399,94 @@ struct OptionsParser
         }
     }
 
+    class ReproPathVisitor : public Slang::Path::Visitor
+    {
+    public:
+        virtual void accept(Slang::Path::Type type, const Slang::UnownedStringSlice& filename) SLANG_OVERRIDE
+        {
+            if (type == Path::Type::File && Path::getPathExt(filename) == "slang-repro")
+            {
+                m_filenames.add(filename);
+            }
+        }
+
+        Slang::List<String> m_filenames;
+    };
+
+    static SlangResult _compileReproDirectory(SlangSession* session, EndToEndCompileRequest* originalRequest, const String& dir)
+    {
+        auto stdOut = originalRequest->getWriter(WriterChannel::StdOutput);
+
+        ReproPathVisitor visitor;
+        Path::find(dir, nullptr, &visitor);
+
+        for (auto filename : visitor.m_filenames)
+        {
+            auto path = Path::combine(dir, filename);
+
+            ComPtr<slang::ICompileRequest> request;
+            SLANG_RETURN_ON_FAIL(session->createCompileRequest(request.writeRef()));
+
+            auto requestImpl = asInternal(request);
+
+            List<uint8_t> buffer;
+            SLANG_RETURN_ON_FAIL(ReproUtil::loadState(path, buffer));
+
+            auto requestState = ReproUtil::getRequest(buffer);
+            MemoryOffsetBase base;
+            base.set(buffer.getBuffer(), buffer.getCount());
+
+            // If we can find a directory, that exists, we will set up a file system to load from that directory
+            ComPtr<ISlangFileSystem> fileSystem;
+            String dirPath;
+            if (SLANG_SUCCEEDED(ReproUtil::calcDirectoryPathFromFilename(path, dirPath)))
+            {
+                SlangPathType pathType;
+                if (SLANG_SUCCEEDED(Path::getPathType(dirPath, &pathType)) && pathType == SLANG_PATH_TYPE_DIRECTORY)
+                {
+                    fileSystem = new RelativeFileSystem(OSFileSystem::getExtSingleton(), dirPath);
+                }
+            }
+
+            SLANG_RETURN_ON_FAIL(ReproUtil::load(base, requestState, fileSystem, requestImpl));
+
+            if (stdOut)
+            {
+                StringBuilder buf;
+                buf << filename << "\n";
+                stdOut->write(buf.getBuffer(), buf.getLength());
+            }
+
+            StringBuilder bufs[Index(WriterChannel::CountOf)];
+            ComPtr<ISlangWriter> writers[Index(WriterChannel::CountOf)];
+            for (Index i = 0; i < Index(WriterChannel::CountOf); ++i)
+            {
+                writers[i] = new StringWriter(&bufs[0], 0);
+                requestImpl->setWriter(WriterChannel(i), writers[i]);
+            }
+
+            if (SLANG_FAILED(requestImpl->compile()))
+            {
+                const char failed[] = "FAILED!\n";
+                stdOut->write(failed, SLANG_COUNT_OF(failed) - 1);
+
+                const auto& diagnostics = bufs[Index(WriterChannel::Diagnostic)];
+
+                stdOut->write(diagnostics.getBuffer(), diagnostics.getLength());
+
+                return SLANG_FAIL;
+            }
+        }
+
+        if (stdOut)
+        {
+            const char end[] = "(END)\n";
+            stdOut->write(end, SLANG_COUNT_OF(end) - 1);
+        }
+
+        return SLANG_OK;
+    }
+
     SlangResult parse(
         int             argc,
         char const* const*  argv)
@@ -620,6 +708,13 @@ struct OptionsParser
 
                     hasLoadedRepro = true;
                 }
+                else if (argValue == "-load-repro-directory")
+                {
+                    CommandLineArg reproDirectory;
+                    SLANG_RETURN_ON_FAIL(reader.expectArg(reproDirectory));
+
+                    SLANG_RETURN_ON_FAIL(_compileReproDirectory(session, requestImpl, reproDirectory.value));
+                }
                 else if (argValue == "-repro-file-system")
                 {
                     CommandLineArg reproName;