Using repro as a file system (#1104)

* Added RiffReadHelper

* Move type to fourCC in Chunk simplifies some code.

* Make MemoryArena able to track external blocks.
Allow ownership of Data to vary.
Changed IR serialization to use moved allocations to avoid copies.

As it turns out all of the array writes could use unowned data, but doing so requires the IRData to stay in scope longer than IRSerialData, which it does at the moment - but perhaps needs better naming or a control for the feature.

* Write out slang-module container.

* WIP on -r option.
Loading modules - with -r.

* Making the serialized-module run (without using imported module).

* Split compiling module from the test.

* Separate module compilation with a function working.

* Remove serialization test as not used.

* Fix warning on gcc.

* Updated test to have types across module boundary.

* Allow entry point declaration.
A test that tries to build with just an entry point declaration and a module.

* Try to make link work with multiple modules.

* Multi module linking first pass working.

* Multi module test working with -module-name option

* Added feature to repro manifest of approximation of command line that was used.

* Use isDefinition - for determining to add decorations to entry point lowering.

* Added support for repo-file-system.h
More precise control of CacheFileSystem.
Allow RelativeFileSystem to strip paths optionally.
Use canonical paths in PathInfo cache.
Fix bug in -D options for command line output of StateSerailizeUtil

* Add missing slang-options.h

* Fix bug in bit slang-state-serialize.cpp with bit removal.

* Added documentation around -repro-file-system
Added spLoadReproAsFileSystem function.

* Fix warning.

8 files changed, 302 insertions, 87 deletions

diff --git a/docs/repro.md b/docs/repro.md
index 5e9070640..f339f429b 100644
--- a/docs/repro.md
+++ b/docs/repro.md
@@ -13,6 +13,13 @@ There are a few command line options
 * `-extract-repro [filename]` extracts the contents of the repro file. The contained files are placed in a directory with a name, the same as the repro file minus the extension. Also contains a 'manifest'.
 * `-load-repro [filename]` loads the repro and compiles using it's options. Note this must be the last arg on the command line.
 * `-dump-repro-on-error` if a compilation fails will attempt to save a repro (using a filename generated from first source filename)
+* `-repro-file-system` [filename] makes the repros file contents appear as the file system during a compilation. Does not set any compilation options.
+
+The `manifest` made available via `-extract-repro` provides some very useful information
+
+* Provides an approximation of the command line that will produce the same compilation under [compile-line]
+* A list of all the unique files held in the repro [files]. It specified their 'unique name' (as used to identify in the repro) and their unique identifier as used by the file system.
+* A list of how paths map to unique files. Listed as the path used to access, followed by the unique name used in the repro
 
 First it is worth just describing what is required to reproduce a compilation. Most straightforwardly the options setup for the compilation need to be stored. This would include any flags, and defines, include paths, entry points, input filenames and so forth. Also needed will be the contents of any files that were specified. This might be files on the file system, but could also be 'files' specified as strings through the slang API. Lastly we need any files that were referenced as part of the compilation - this could be include files, or module source files and so forth. All of this information is bundled up together into a file that can then later be loaded and compiled. This is broadly speaking all of the data that is stored within a repro file. 
 
@@ -28,6 +35,8 @@ When a repro is loaded, before files are loaded from the repro itself, they will
 2) Go to the `repro directory` and edit files that you wish to change. You can also just delete files that do not need changing, as they will be loaded from the repro.
 3) Load the repro - it will now load any files requested from the `repro directory`
 
+Now you might want to change the compilation options. Using -load-repro it will compile with the options as given. It is not possible to change those options as part of -load-repro. If you want to change the compilation options (and files), you can use -extract-repro, and look at the manifest which will list a command line that will typically repeat the compilation. Now you can just attach the repro as a file system, and set the command line options as appropriate, based on the command line listed in the manifest. Note! If there is a fairly complex directory hierarchy, it may be necessary to specify the input sources paths *as if* they are held on the original files system. You can see how these map in the manifest. 
+
 Note that currently is is disabled to access any new source files - they will be determined as `not found`. This behaviour could be changed such that the regular file system was used, or the ISlangFilesystem set on the API is used as a fallback. 
 
 There currently isn't a mechanism to alter the options of a repro from the command line (other than altering the contents of the source). The reason for this is because of how command lines are processed currently in Slang. A future update could enable specifying a repro and then altering the command line options used. It can be achieved through the API though. Once the repro is loaded via the `spLoadRepro` function, options can be changed as normal. The two major places where option alteration may have surprising behavior are... 
@@ -57,7 +66,14 @@ SLANG_API SlangResult spExtractRepro(
     const void* reproData, 
     size_t reproDataSize, 
     ISlangFileSystemExt* fileSystem);    
-    
+
+SLANG_API SlangResult spLoadReproAsFileSystem(
+    SlangSession* session,
+    const void* reproData,
+    size_t reproDataSize,
+    ISlangFileSystem* replaceFileSystem,
+    ISlangFileSystemExt** outFileSystem);
+   
 ```
 
 The fileSystem parameter passed to `spLoadRepro` provides the mechanism for client code to replace the files that are held within the repro. NOTE! That the files will be loaded from this file system with their `unique names` as if they are part of the flat file system. If an attempt to load a file fails, the file within the repro is used. That `spLoadRepro` is typically performed on a new 'unused' SlangCompileRequest. After a call to `spLoadRepro` normal functions to alter the state of the SlangCompileRequest are available. 
@@ -66,6 +82,8 @@ The function `spEnableReproCapture` should be set after any ISlangFileSystem has
     
 The function `spExtractRepro` allows for extracting the files used in a request (along with the associated manifest). They files and manifest are stored under the 'unique names' in the root of the user provided ISlangFileSystemExt.     
     
+The function `spLoadReproAsFileSystem` creates a file system that can access the contents of the repro with the same paths that were used on the originating system. The ISlangFileSystemExt produced can be set on a request and used for compilation.    
+    
 Repro files are currently stored in a binary format. This format is sensitive to changes in the API, as well as internal state within a SlangCompileRequest. This means that the functionality can only be guarenteed to work with exactly the same version of Slang on the same version of compiler. In practice things are typically not so draconian, and future versions will aim to provide a more clear slang repro versioning system, and work will be performed to make more generally usable. 
 
 Finally this version of the repo system does not take into account endianess at all. The system the repro is saved from must have the same endianess as the system loaded on.
diff --git a/slang.h b/slang.h
index 85e8b7449..83811ffd1 100644
--- a/slang.h
+++ b/slang.h
@@ -1685,6 +1685,25 @@ extern "C"
         size_t reproDataSize,
         ISlangFileSystemExt* fileSystem);
 
+    /* Turns a repro into a file system.
+
+    Makes the contents of the repro available as a file system - that is able to access the files with the same
+    paths as were used on the original repro file system. 
+
+    @param session          The slang session
+    @param reproData        The repro data
+    @param reproDataSize    The size of the repro data
+    @param replaceFileSystem  Will attempt to load by unique names from this file system before using contents of the repro. Optional.
+    @param outFileSystem    The file system that can be used to access contents
+    @returns                A `SlangResult` to indicate success or failure.
+    */
+    SLANG_API SlangResult spLoadReproAsFileSystem(
+        SlangSession* session,
+        const void* reproData,
+        size_t reproDataSize,
+        ISlangFileSystem* replaceFileSystem,
+        ISlangFileSystemExt** outFileSystem);
+    
     /*
     Forward declarations of types used in the reflection interface;
     */
diff --git a/source/slang/slang-file-system.cpp b/source/slang/slang-file-system.cpp
index 7d6d6df66..c6bee28e1 100644
--- a/source/slang/slang-file-system.cpp
+++ b/source/slang/slang-file-system.cpp
@@ -203,18 +203,35 @@ SLANG_NO_THROW SlangResult SLANG_MCALL CacheFileSystem::queryInterface(SlangUUID
 }
 
 
-CacheFileSystem::CacheFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMode uniqueIdentityMode, PathStyle pathStyle) :
-    m_fileSystem(fileSystem),
-    m_uniqueIdentityMode(uniqueIdentityMode),
-    m_pathStyle(pathStyle)
+CacheFileSystem::CacheFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMode uniqueIdentityMode, PathStyle pathStyle) 
 {
+    setInnerFileSystem(fileSystem, uniqueIdentityMode, pathStyle);
+}
+
+CacheFileSystem::~CacheFileSystem()
+{
+    for (const auto& pair : m_uniqueIdentityMap)
+    {
+        delete pair.Value;
+    }
+}
+
+void CacheFileSystem::setInnerFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMode uniqueIdentityMode, PathStyle pathStyle)
+{
+    m_fileSystem = fileSystem;
+
+    m_uniqueIdentityMode = uniqueIdentityMode;
+    m_pathStyle = pathStyle;
+    
+    m_fileSystemExt.setNull();
+
     if (fileSystem)
     {
         // Try to get the more sophisticated interface
         fileSystem->queryInterface(IID_ISlangFileSystemExt, (void**)m_fileSystemExt.writeRef());
     }
 
-    switch (uniqueIdentityMode)
+    switch (m_uniqueIdentityMode)
     {
         case UniqueIdentityMode::Default:
         case UniqueIdentityMode::FileSystemExt:
@@ -226,29 +243,22 @@ CacheFileSystem::CacheFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMod
         default: break;
     }
 
-    if (m_fileSystemExt)
-    {
-        // We just defer to the m_fileSystem, so we mark as unknown 
-        m_pathStyle = PathStyle::Unknown;
-    }
-    else if (m_pathStyle == PathStyle::Default)
+    if (pathStyle == PathStyle::Default)
     {
         // We'll assume it's simplify-able 
         m_pathStyle = PathStyle::Simplifiable;
+        // If we have fileSystemExt, we defer to that
+        if (m_fileSystemExt)
+        {
+            // We just defer to the m_fileSystem
+            m_pathStyle = PathStyle::FileSystemExt;
+        }
     }
 
     // It can't be default
     SLANG_ASSERT(m_uniqueIdentityMode != UniqueIdentityMode::Default);
 }
 
-CacheFileSystem::~CacheFileSystem()
-{
-    for (const auto& pair : m_uniqueIdentityMap)
-    {
-        delete pair.Value;
-    }
-}
-
 void CacheFileSystem::clearCache()
 {
     for (const auto& pair : m_uniqueIdentityMap)
@@ -450,14 +460,17 @@ SlangResult CacheFileSystem::getFileUniqueIdentity(const char* path, ISlangBlob*
 SlangResult CacheFileSystem::calcCombinedPath(SlangPathType fromPathType, const char* fromPath, const char* path, ISlangBlob** pathOut)
 {
     // Just defer to contained implementation
-    if (m_fileSystemExt)
-    {
-        return m_fileSystemExt->calcCombinedPath(fromPathType, fromPath, path, pathOut);
-    }
-    else
+    switch (m_pathStyle)
     {
-        // Just use the default implementation
-        return _calcCombinedPath(fromPathType, fromPath, path, pathOut);
+        case PathStyle::FileSystemExt:
+        {
+            return m_fileSystemExt->calcCombinedPath(fromPathType, fromPath, path, pathOut);
+        }
+        default:
+        {
+            // Just use the default implementation
+            return _calcCombinedPath(fromPathType, fromPath, path, pathOut);
+        }
     }
 }
 
@@ -498,34 +511,24 @@ SlangResult CacheFileSystem::getPathType(const char* pathIn, SlangPathType* path
 SlangResult CacheFileSystem::getSimplifiedPath(const char* path, ISlangBlob** outSimplifiedPath)
 {
     // If we have a ISlangFileSystemExt we can just pass on the request to it
-    if (m_fileSystemExt)
+    switch (m_pathStyle)
     {
-        return m_fileSystemExt->getSimplifiedPath(path, outSimplifiedPath);
-    }
-    else
-    {
-        // Use the path style to see what we can do with it
-        switch (m_pathStyle)
+        case PathStyle::FileSystemExt:
         {
-            case PathStyle::Simplifiable:
-            {
-                String simplifiedPath = Path::simplify(_fixPathDelimiters(path));
-                *outSimplifiedPath = StringUtil::createStringBlob(simplifiedPath).detach();
-                return SLANG_OK;
-            }
-            default: return SLANG_E_NOT_IMPLEMENTED;
+            return m_fileSystemExt->getSimplifiedPath(path, outSimplifiedPath);
         }
+        case PathStyle::Simplifiable:
+        {
+            String simplifiedPath = Path::simplify(_fixPathDelimiters(path));
+            *outSimplifiedPath = StringUtil::createStringBlob(simplifiedPath).detach();
+            return SLANG_OK;
+        }
+        default: return SLANG_E_NOT_IMPLEMENTED;
     }
 }
 
 SlangResult CacheFileSystem::getCanonicalPath(const char* path, ISlangBlob** outCanonicalPath)
 {
-    // If we don't have a backing full file system, we can't produce a canonical path with just ISlangFileSystem::loadFile
-    if (!m_fileSystemExt)
-    {
-        return SLANG_E_NOT_IMPLEMENTED;
-    }
-
     // A file must exist to get a canonical path... 
     PathInfo* info = _resolvePathCacheInfo(path);
     if (!info)
@@ -536,6 +539,11 @@ SlangResult CacheFileSystem::getCanonicalPath(const char* path, ISlangBlob** out
     // We don't have this -> so read it ...
     if (info->m_getCanonicalPathResult == CompressedResult::Uninitialized)
     {
+        if (!m_fileSystemExt)
+        {
+            return SLANG_E_NOT_IMPLEMENTED;
+        }
+
         // Try getting the canonicalPath by asking underlying file system
         ComPtr<ISlangBlob> canonicalPathBlob;
         SlangResult res = m_fileSystemExt->getCanonicalPath(path, canonicalPathBlob.writeRef());
@@ -576,7 +584,15 @@ ISlangUnknown* RelativeFileSystem::getInterface(const Guid& guid)
 SlangResult RelativeFileSystem::_getFixedPath(const char* path, String& outPath)
 {
     ComPtr<ISlangBlob> blob;
-    SLANG_RETURN_ON_FAIL(m_fileSystem->calcCombinedPath(SLANG_PATH_TYPE_DIRECTORY, m_relativePath.getBuffer(), path,  blob.writeRef()));
+    if (m_stripPath)
+    {
+        String strippedPath = Path::getFileName(path);
+        SLANG_RETURN_ON_FAIL(m_fileSystem->calcCombinedPath(SLANG_PATH_TYPE_DIRECTORY, m_relativePath.getBuffer(), strippedPath.getBuffer(), blob.writeRef()));
+    }
+    else
+    {
+        SLANG_RETURN_ON_FAIL(m_fileSystem->calcCombinedPath(SLANG_PATH_TYPE_DIRECTORY, m_relativePath.getBuffer(), path, blob.writeRef()));
+    }
 
     outPath = StringUtil::getString(blob);
     return SLANG_OK;
diff --git a/source/slang/slang-file-system.h b/source/slang/slang-file-system.h
index 71596be5b..3d2bfe3c1 100644
--- a/source/slang/slang-file-system.h
+++ b/source/slang/slang-file-system.h
@@ -117,8 +117,8 @@ class CacheFileSystem: public ISlangFileSystemExt, public RefObject
     enum class PathStyle
     {
         Default,                    ///< Pass to say use the default 
-        Unknown,                    ///< It's an unknown type of path
         Simplifiable,               ///< It can be simplified by Path::Simplify
+        FileSystemExt,              ///< Use file system
     };
 
     enum UniqueIdentityMode
@@ -216,6 +216,14 @@ class CacheFileSystem: public ISlangFileSystemExt, public RefObject
         return SLANG_E_NOT_IMPLEMENTED;
     }
 
+        /// Get the unique identity mode
+    UniqueIdentityMode getUniqueIdentityMode() const { return m_uniqueIdentityMode; }
+        /// Get the path style
+    PathStyle getPathStyle() const { return m_pathStyle; }
+
+        /// Set the inner file system 
+    void setInnerFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMode uniqueIdentityMode = UniqueIdentityMode::Default, PathStyle pathStyle = PathStyle::Default);
+
         /// Ctor
     CacheFileSystem(ISlangFileSystem* fileSystem, UniqueIdentityMode uniqueIdentityMode = UniqueIdentityMode::Default, PathStyle pathStyle = PathStyle::Default);
         /// Dtor
@@ -291,9 +299,10 @@ public:
 
     virtual SLANG_NO_THROW SlangResult SLANG_MCALL saveFile(const char* path, const void* data, size_t size) SLANG_OVERRIDE;
 
-    RelativeFileSystem(ISlangFileSystemExt* fileSystem, const String& relativePath):
+    RelativeFileSystem(ISlangFileSystemExt* fileSystem, const String& relativePath, bool stripPath = false):
         m_fileSystem(fileSystem),
-        m_relativePath(relativePath)
+        m_relativePath(relativePath),
+        m_stripPath(stripPath)
     {
     }
 
@@ -303,6 +312,8 @@ protected:
 
     ISlangUnknown* getInterface(const Guid& guid);
 
+    bool m_stripPath;
+
     ComPtr<ISlangFileSystemExt> m_fileSystem;
     String m_relativePath;
 };
diff --git a/source/slang/slang-options.cpp b/source/slang/slang-options.cpp
index b17e35d69..ad1d27114 100644
--- a/source/slang/slang-options.cpp
+++ b/source/slang/slang-options.cpp
@@ -563,6 +563,39 @@ struct OptionsParser
 
                     return SLANG_OK;
                 }
+                else if (argStr == "-repro-file-system")
+                {
+                    String reproName;
+                    SLANG_RETURN_ON_FAIL(tryReadCommandLineArgument(sink, arg, &argCursor, argEnd, reproName));
+
+                    List<uint8_t> buffer;
+                    SLANG_RETURN_ON_FAIL(StateSerializeUtil::loadState(reproName, buffer));
+
+                    auto requestState = StateSerializeUtil::getRequest(buffer);
+                    MemoryOffsetBase base;
+                    base.set(buffer.getBuffer(), buffer.getCount());
+
+                    // If we can find a directory, that exists, we will set up a file system to load from that directory
+                    ComPtr<ISlangFileSystem> dirFileSystem;
+                    String dirPath;
+                    if (SLANG_SUCCEEDED(StateSerializeUtil::calcDirectoryPathFromFilename(reproName, dirPath)))
+                    {
+                        SlangPathType pathType;
+                        if (SLANG_SUCCEEDED(Path::getPathType(dirPath, &pathType)) && pathType == SLANG_PATH_TYPE_DIRECTORY)
+                        {
+                            dirFileSystem = new RelativeFileSystem(OSFileSystemExt::getSingleton(), dirPath, true);
+                        }
+                    }
+
+                    RefPtr<CacheFileSystem> cacheFileSystem;
+                    SLANG_RETURN_ON_FAIL(StateSerializeUtil::loadFileSystem(base, requestState, dirFileSystem, cacheFileSystem));
+
+                    // I might want to make the dir file system the fallback file system...
+                    cacheFileSystem->setInnerFileSystem(dirFileSystem, cacheFileSystem->getUniqueIdentityMode(), cacheFileSystem->getPathStyle());
+
+                    // Set as the file system
+                    spSetFileSystem(compileRequest, cacheFileSystem);
+                }
                 else if (argStr == "-serial-ir")
                 {
                     requestImpl->getFrontEndReq()->useSerialIRBottleneck = true;
diff --git a/source/slang/slang-state-serialize.cpp b/source/slang/slang-state-serialize.cpp
index c373c5fcc..369c433ca 100644
--- a/source/slang/slang-state-serialize.cpp
+++ b/source/slang/slang-state-serialize.cpp
@@ -631,16 +631,18 @@ struct LoadContext
     typedef StateSerializeUtil::FileState FileState;
     typedef StateSerializeUtil::PathInfoState PathInfoState;
 
-    ISlangBlob* getFileBlob(FileState* file)
+    CacheFileSystem::PathInfo* getPathInfoFromFile(FileState* file)
     {
         if (!file)
         {
             return nullptr;
         }
 
-        ComPtr<ISlangBlob> blob;
-        if (!m_fileToBlobMap.TryGetValue(file, blob))
+        CacheFileSystem::PathInfo* dstInfo = nullptr;
+        if (!m_fileToPathInfoMap.TryGetValue(file, dstInfo))
         {
+            ComPtr<ISlangBlob> blob;
+
             if (m_fileSystem && file->uniqueName)
             {
                 // Try loading from the file system
@@ -653,11 +655,39 @@ struct LoadContext
                 blob = new StringBlob(m_base->asRaw(file->contents)->getSlice());
             }
 
+            dstInfo = new CacheFileSystem::PathInfo(String());
+
+            if (file->uniqueIdentity)
+            {
+                String uniqueIdentity = m_base->asRaw(file->uniqueIdentity)->getSlice();
+                dstInfo->m_uniqueIdentity = new StringBlob(uniqueIdentity);
+            }
+
+            if (file->canonicalPath)
+            {
+                dstInfo->m_canonicalPath = new StringBlob(m_base->asRaw(file->canonicalPath)->getSlice());
+            }
+
+            if (blob)
+            {
+                dstInfo->m_loadFileResult = CacheFileSystem::CompressedResult::Ok;
+                dstInfo->m_getPathTypeResult = CacheFileSystem::CompressedResult::Ok;
+                dstInfo->m_pathType = SLANG_PATH_TYPE_FILE;
+            }
+
+            dstInfo->m_fileBlob = blob;
+
             // Add to map, even if the blob is nullptr (say from a failed read)
-            m_fileToBlobMap.Add(file, blob);
+            m_fileToPathInfoMap.Add(file, dstInfo);
         }
 
-        return blob;
+        return dstInfo;
+    }
+
+    ISlangBlob* getFileBlobFromFile(FileState* file)
+    {
+        CacheFileSystem::PathInfo* pathInfo = getPathInfoFromFile(file);
+        return pathInfo ? pathInfo->m_fileBlob.get() : nullptr;
     }
 
     SourceFile* getSourceFile(SourceFileState* sourceFile)
@@ -671,7 +701,7 @@ struct LoadContext
         if (!m_sourceFileMap.TryGetValue(sourceFile, dstFile))
         {
             FileState* file = m_base->asRaw(sourceFile->file);
-            ISlangBlob* blob = getFileBlob(file);
+            ISlangBlob* blob = getFileBlobFromFile(file);
 
             PathInfo pathInfo;
 
@@ -711,22 +741,18 @@ struct LoadContext
             return pathInfo;
         }
 
-        CacheFileSystem::PathInfo* dstInfo = new CacheFileSystem::PathInfo(String());
         FileState* file = m_base->asRaw(srcInfo->file);
+        CacheFileSystem::PathInfo* dstInfo;
+
         if (file)
         {
-            if (file->uniqueIdentity)
-            {
-                String uniqueIdentity = m_base->asRaw(file->uniqueIdentity)->getSlice();
-                dstInfo->m_uniqueIdentity = new StringBlob(uniqueIdentity);
-            }
-
-            if (file->canonicalPath)
-            {
-                dstInfo->m_canonicalPath = new StringBlob(m_base->asRaw(file->canonicalPath)->getSlice());
-            }
-
-            dstInfo->m_fileBlob = getFileBlob(file);
+            dstInfo = getPathInfoFromFile(file);
+        }
+        else
+        {
+            // TODO(JS): Hmmm... this could end up not being cleared up
+            // Because it is not added to the unique set (as unique set is for files and this isn't a file)
+            dstInfo = new CacheFileSystem::PathInfo(String());
         }
 
         dstInfo->m_getCanonicalPathResult = srcInfo->getCanonicalPathResult;
@@ -773,14 +799,67 @@ struct LoadContext
     OffsetBase* m_base;
 
     SourceManager* m_sourceManager;
+
     Dictionary<SourceFileState*, SourceFile*> m_sourceFileMap;
-    Dictionary<FileState*, ComPtr<ISlangBlob> > m_fileToBlobMap;
+    Dictionary<FileState*, CacheFileSystem::PathInfo*> m_fileToPathInfoMap;
     Dictionary<const PathInfoState*, CacheFileSystem::PathInfo*> m_pathInfoMap;
 };
 
 } // anonymous
 
 
+/* static */SlangResult StateSerializeUtil::loadFileSystem(OffsetBase& base, RequestState* requestState, ISlangFileSystem* fileSystem, RefPtr<CacheFileSystem>& outFileSystem)
+{
+    LoadContext context(nullptr, fileSystem, &base);
+
+    RefPtr<CacheFileSystem> cacheFileSystem = new CacheFileSystem(nullptr);
+    auto& dstUniqueMap = cacheFileSystem->getUniqueMap();
+    auto& dstPathMap = cacheFileSystem->getPathMap();
+
+    for (auto fileOffset : requestState->files)
+    {
+        // add the file
+        FileState* fileState = base.asRaw(base.asRaw(fileOffset));
+        CacheFileSystem::PathInfo* pathInfo = context.getPathInfoFromFile(fileState);
+
+        if (fileState->foundPath)
+        {
+            String foundPath = base.asRaw(fileState->foundPath)->getSlice();
+            dstPathMap.AddIfNotExists(foundPath, pathInfo);
+        }
+    }
+
+    // Put all the paths to path info
+    {
+        for (const auto& pairOffset : requestState->pathInfoMap)
+        {
+            const auto& pair = base.asRaw(pairOffset);
+            CacheFileSystem::PathInfo* pathInfo = context.addPathInfo(base.asRaw(pair.pathInfo));
+            dstPathMap.AddIfNotExists(base.asRaw(pair.path)->getSlice(), pathInfo);
+        }
+    }
+
+    // Put all the path infos in the cache system
+    {
+        for (const auto& pair : context.m_fileToPathInfoMap)
+        {
+            CacheFileSystem::PathInfo* pathInfo = pair.Value;
+            SLANG_ASSERT(pathInfo->m_uniqueIdentity);
+            dstUniqueMap.Add(pathInfo->m_uniqueIdentity->getString(), pathInfo);
+
+            // Add canonical paths too..
+            if (pathInfo->m_canonicalPath)
+            {
+                String canonicalPath = pathInfo->m_canonicalPath->getString();
+
+                dstPathMap.AddIfNotExists(canonicalPath, pathInfo);
+            }
+        }
+    }
+
+    outFileSystem = cacheFileSystem;
+    return SLANG_OK;
+}
 
 /* static */SlangResult StateSerializeUtil::load(OffsetBase& base, RequestState* requestState, ISlangFileSystem* fileSystem, EndToEndCompileRequest* request)
 {
@@ -947,7 +1026,7 @@ struct LoadContext
         }
         // Put all the path infos in the cache system
         {
-            for (const auto& pair : context.m_pathInfoMap)
+            for (const auto& pair : context.m_fileToPathInfoMap)
             {
                 CacheFileSystem::PathInfo* pathInfo = pair.Value;
                 SLANG_ASSERT(pathInfo->m_uniqueIdentity);
@@ -1076,6 +1155,21 @@ struct LoadContext
     return extractFiles(base, requestState, &relFileSystem);
 }
 
+static void _calcPreprocessorDefines(OffsetBase& base, const Offset32Array<StateSerializeUtil::StringPair>& srcDefines, CommandLine& cmd)
+{
+    for (const auto& define : srcDefines)
+    {
+        StringBuilder builder;
+        builder << "-D" << base.asRaw(base.asRaw(define).first)->getSlice();
+        if (base.asRaw(define).second)
+        {
+            builder << "=" << base.asRaw(base.asRaw(define).second)->getSlice();
+        }
+
+        cmd.addArg(builder);
+    }
+}
+
 static SlangResult _calcCommandLine(OffsetBase& base, StateSerializeUtil::RequestState* requestState, CommandLine& cmd)
 {
     typedef StateSerializeUtil::TargetRequestState TargetRequestState;
@@ -1086,15 +1180,17 @@ static SlangResult _calcCommandLine(OffsetBase& base, StateSerializeUtil::Reques
         while (flags)
         {
             // Extract a bit
-            SlangCompileFlags newFlags = flags & (flags - 1);
-            SlangCompileFlags flag = newFlags ^ flags;
+            const SlangCompileFlags isolatedBit = flags & SlangCompileFlags(-int(flags)); 
 
-            switch (flag)
+            switch (isolatedBit)
             {
                 case SLANG_COMPILE_FLAG_NO_MANGLING:    cmd.addArg("-no-mangle"); break;
                 case SLANG_COMPILE_FLAG_NO_CODEGEN:     cmd.addArg("-no-codegen"); break;
                 default: break;
             }
+
+            // Remove the bit
+            flags &= ~isolatedBit;
         }
         //spSetDumpIntermediates(externalRequest, int(requestState->shouldDumpIntermediates));
 
@@ -1220,17 +1316,7 @@ static SlangResult _calcCommandLine(OffsetBase& base, StateSerializeUtil::Reques
         }
     }
 
-    {
-        for (const auto& define : requestState->preprocessorDefinitions)
-        {
-            StringBuilder builder;
-            builder << "-D" << base.asRaw(base.asRaw(define).first)->getSlice();
-            if (base.asRaw(define).second)
-            {
-                builder << "=" << base.asRaw(base.asRaw(define).second)->getSlice();
-            }
-        }
-    }
+    _calcPreprocessorDefines(base, requestState->preprocessorDefinitions, cmd);
 
     {
         const auto& srcTranslationUnits = requestState->translationUnits;
@@ -1239,8 +1325,9 @@ static SlangResult _calcCommandLine(OffsetBase& base, StateSerializeUtil::Reques
         {
             const auto& srcTranslationUnit = base.asRaw(srcTranslationUnits[i]);
 
-            //context.loadDefines(srcTranslationUnit.preprocessorDefinitions, dstTranslationUnit->preprocessorDefinitions);
+            _calcPreprocessorDefines(base, srcTranslationUnit.preprocessorDefinitions, cmd);
 
+            
 #if 0
             if (srcTranslationUnit.moduleName)
             {
diff --git a/source/slang/slang-state-serialize.h b/source/slang/slang-state-serialize.h
index 1a4c9b227..9a404138e 100644
--- a/source/slang/slang-state-serialize.h
+++ b/source/slang/slang-state-serialize.h
@@ -163,6 +163,10 @@ struct StateSerializeUtil
 
     static SlangResult saveState(EndToEndCompileRequest* request, Stream* stream);
 
+        /// Create a cache file system that uses contents of the request state.
+        /// The passed in fileSystem is used for accessing any file accesses not found in the cache
+    static SlangResult loadFileSystem(OffsetBase& base, RequestState* requestState, ISlangFileSystem* fileSystem, RefPtr<CacheFileSystem>& outFileSystem);
+
         /// Load the requestState into request
         /// The fileSystem is optional and can be passed as nullptr. If set, as each file is loaded
         /// it will attempt to load from fileSystem the *uniqueName*
diff --git a/source/slang/slang.cpp b/source/slang/slang.cpp
index c7171d835..946d7337b 100644
--- a/source/slang/slang.cpp
+++ b/source/slang/slang.cpp
@@ -3574,6 +3574,33 @@ SLANG_API SlangResult spExtractRepro(SlangSession* session, const void* reproDat
     return StateSerializeUtil::extractFiles(base, requestState, fileSystem);
 }
 
+SLANG_API SlangResult spLoadReproAsFileSystem(
+    SlangSession* session,
+    const void* reproData,
+    size_t reproDataSize,
+    ISlangFileSystem* replaceFileSystem,
+    ISlangFileSystemExt** outFileSystem)
+{
+    using namespace Slang;
+
+    SLANG_UNUSED(session);
+    
+    MemoryStreamBase stream(FileAccess::Read, reproData, reproDataSize);
+
+    List<uint8_t> buffer;
+    SLANG_RETURN_ON_FAIL(StateSerializeUtil::loadState(&stream, buffer));
+
+    auto requestState = StateSerializeUtil::getRequest(buffer);
+    MemoryOffsetBase base;
+    base.set(buffer.getBuffer(), buffer.getCount());
+
+    RefPtr<CacheFileSystem> cacheFileSystem;
+    SLANG_RETURN_ON_FAIL(StateSerializeUtil::loadFileSystem(base, requestState, replaceFileSystem, cacheFileSystem));
+
+    *outFileSystem = cacheFileSystem.detach();
+    return SLANG_OK;
+}
+
 // Reflection API
 
 SLANG_API SlangResult spCompileRequest_getProgram(