Using repro as a file system (#1104)

* Added RiffReadHelper

* Move type to fourCC in Chunk simplifies some code.

* Make MemoryArena able to track external blocks.
Allow ownership of Data to vary.
Changed IR serialization to use moved allocations to avoid copies.

As it turns out all of the array writes could use unowned data, but doing so requires the IRData to stay in scope longer than IRSerialData, which it does at the moment - but perhaps needs better naming or a control for the feature.

* Write out slang-module container.

* WIP on -r option.
Loading modules - with -r.

* Making the serialized-module run (without using imported module).

* Split compiling module from the test.

* Separate module compilation with a function working.

* Remove serialization test as not used.

* Fix warning on gcc.

* Updated test to have types across module boundary.

* Allow entry point declaration.
A test that tries to build with just an entry point declaration and a module.

* Try to make link work with multiple modules.

* Multi module linking first pass working.

* Multi module test working with -module-name option

* Added feature to repro manifest of approximation of command line that was used.

* Use isDefinition - for determining to add decorations to entry point lowering.

* Added support for repo-file-system.h
More precise control of CacheFileSystem.
Allow RelativeFileSystem to strip paths optionally.
Use canonical paths in PathInfo cache.
Fix bug in -D options for command line output of StateSerailizeUtil

* Add missing slang-options.h

* Fix bug in bit slang-state-serialize.cpp with bit removal.

* Added documentation around -repro-file-system
Added spLoadReproAsFileSystem function.

* Fix warning.

1 files changed, 19 insertions, 1 deletions

diff --git a/docs/repro.md b/docs/repro.md
index 5e9070640..f339f429b 100644
--- a/docs/repro.md
+++ b/docs/repro.md
@@ -13,6 +13,13 @@ There are a few command line options
 * `-extract-repro [filename]` extracts the contents of the repro file. The contained files are placed in a directory with a name, the same as the repro file minus the extension. Also contains a 'manifest'.
 * `-load-repro [filename]` loads the repro and compiles using it's options. Note this must be the last arg on the command line.
 * `-dump-repro-on-error` if a compilation fails will attempt to save a repro (using a filename generated from first source filename)
+* `-repro-file-system` [filename] makes the repros file contents appear as the file system during a compilation. Does not set any compilation options.
+
+The `manifest` made available via `-extract-repro` provides some very useful information
+
+* Provides an approximation of the command line that will produce the same compilation under [compile-line]
+* A list of all the unique files held in the repro [files]. It specified their 'unique name' (as used to identify in the repro) and their unique identifier as used by the file system.
+* A list of how paths map to unique files. Listed as the path used to access, followed by the unique name used in the repro
 
 First it is worth just describing what is required to reproduce a compilation. Most straightforwardly the options setup for the compilation need to be stored. This would include any flags, and defines, include paths, entry points, input filenames and so forth. Also needed will be the contents of any files that were specified. This might be files on the file system, but could also be 'files' specified as strings through the slang API. Lastly we need any files that were referenced as part of the compilation - this could be include files, or module source files and so forth. All of this information is bundled up together into a file that can then later be loaded and compiled. This is broadly speaking all of the data that is stored within a repro file. 
 
@@ -28,6 +35,8 @@ When a repro is loaded, before files are loaded from the repro itself, they will
 2) Go to the `repro directory` and edit files that you wish to change. You can also just delete files that do not need changing, as they will be loaded from the repro.
 3) Load the repro - it will now load any files requested from the `repro directory`
 
+Now you might want to change the compilation options. Using -load-repro it will compile with the options as given. It is not possible to change those options as part of -load-repro. If you want to change the compilation options (and files), you can use -extract-repro, and look at the manifest which will list a command line that will typically repeat the compilation. Now you can just attach the repro as a file system, and set the command line options as appropriate, based on the command line listed in the manifest. Note! If there is a fairly complex directory hierarchy, it may be necessary to specify the input sources paths *as if* they are held on the original files system. You can see how these map in the manifest. 
+
 Note that currently is is disabled to access any new source files - they will be determined as `not found`. This behaviour could be changed such that the regular file system was used, or the ISlangFilesystem set on the API is used as a fallback. 
 
 There currently isn't a mechanism to alter the options of a repro from the command line (other than altering the contents of the source). The reason for this is because of how command lines are processed currently in Slang. A future update could enable specifying a repro and then altering the command line options used. It can be achieved through the API though. Once the repro is loaded via the `spLoadRepro` function, options can be changed as normal. The two major places where option alteration may have surprising behavior are... 
@@ -57,7 +66,14 @@ SLANG_API SlangResult spExtractRepro(
     const void* reproData, 
     size_t reproDataSize, 
     ISlangFileSystemExt* fileSystem);    
-    
+
+SLANG_API SlangResult spLoadReproAsFileSystem(
+    SlangSession* session,
+    const void* reproData,
+    size_t reproDataSize,
+    ISlangFileSystem* replaceFileSystem,
+    ISlangFileSystemExt** outFileSystem);
+   
 ```
 
 The fileSystem parameter passed to `spLoadRepro` provides the mechanism for client code to replace the files that are held within the repro. NOTE! That the files will be loaded from this file system with their `unique names` as if they are part of the flat file system. If an attempt to load a file fails, the file within the repro is used. That `spLoadRepro` is typically performed on a new 'unused' SlangCompileRequest. After a call to `spLoadRepro` normal functions to alter the state of the SlangCompileRequest are available. 
@@ -66,6 +82,8 @@ The function `spEnableReproCapture` should be set after any ISlangFileSystem has
     
 The function `spExtractRepro` allows for extracting the files used in a request (along with the associated manifest). They files and manifest are stored under the 'unique names' in the root of the user provided ISlangFileSystemExt.     
     
+The function `spLoadReproAsFileSystem` creates a file system that can access the contents of the repro with the same paths that were used on the originating system. The ISlangFileSystemExt produced can be set on a request and used for compilation.    
+    
 Repro files are currently stored in a binary format. This format is sensitive to changes in the API, as well as internal state within a SlangCompileRequest. This means that the functionality can only be guarenteed to work with exactly the same version of Slang on the same version of compiler. In practice things are typically not so draconian, and future versions will aim to provide a more clear slang repro versioning system, and work will be performed to make more generally usable. 
 
 Finally this version of the repo system does not take into account endianess at all. The system the repro is saved from must have the same endianess as the system loaded on.