First pass Repro documentation (#1086)

* WIP on serialize/save state.

* Relative string encoding.

* Added RelativeContainer unit test.
Split out RelativeContainer into core.

* Fix bug in RelativeString encoding.

* More work around relative container.

* Fix checks.

* Use RelativeBase for safe access.
Use malloc/free/realloc instead of List.

* Add natvis support for relative types.

* Setting up of state (not includes) writing of repro state.

* Capture after spCompile.

* Writing SourceFile and file system files.
Added -dump-repo

* First pass at loading state.

* First pass at reading repro.

* Small optimization around Safe32Ptr

* Refactor how repro data is stored - to make saving off the files more simple, by having all all backed by 'files'.
Make file loading always set up PathInfo so we get uniqueIdentifier info.

* Generate unique file names.

* Added RelativeFileSystem
Added saveFile to ISlangFileSystemExt and implemented for interfaces
Added mechanism to save of files (and manifest)

* Added ability to replace files in repo with directory holding their contents.

* Add support for entry points.

* Fix problem compiling on linux.

* Added SIMPLE_EX option, where everything on command line must be specified.

* Fix typo in unit test for relative container.

* Fix another typo in unit test for RelativeContainer.

* Fix small bugs.

* Fix release unused variable issue in slang-state-serialize.cpp

* Fix checking for SIMPLE_EX in testing, else broke COMMAND_LINE_SIMPLE.

* Fix warnings on 32 bit debug build.

* First pass repro docs.

2 files changed, 63 insertions, 2 deletions

diff --git a/docs/repro.md b/docs/repro.md
new file mode 100644
index 000000000..c50fc2a70
--- /dev/null
+++ b/docs/repro.md
@@ -0,0 +1,58 @@
+Slang Compilation Reproduction
+==============================
+
+Slang has both API and command line support for reproducing compilations, so called 'repro' functionality.
+
+One use of the feature is if a compilation fails, or produces an unexpected or wrong result, it provides a simple to use mechanism where the compilation can be repeated or 'reproduced', most often on another machine. Instead of having to describe all the options, and make sure all of the files that are used are copied, and in such a way that it repeats the result, all that is required is for the compilation to be run on the host machine with repro capture enabled, and then that 'repro' used for a compilation on the test machine. There are also some mechanisms where the contents of the orginal compilation can be altered.
+
+The actual data saved is the contents of the SlangCompileReqest. Currently no state is saved from the SlangSession. Saving and loading a SlangCompileRequest into a new SlangCompileRequest should provide two SlangCompileRequests with the same state, and with the second compile request having access to all the files contents the original request had directly in memory. 
+
+There are 3 command line options
+
+* `-dump-repro [filename]` dumps the compilations state (ie post attempting to compile) to the file specified afterwards
+* `-extract-repro [filename]` extracts the contents of the repro file. The contained files are placed in a directory with a name, the same as the repro file minus the extension. Also contains a 'manifest'.
+* `-load-repro [filename]` loads the repro and compiles using it's options. Note this must be the last arg on the command line.
+
+First it is worth just describing what is required to reproduce a compilation. Most straightforwardly the options setup for the compilation need to be stored. This would include any flags, and defines, include paths, entry points, input filenames and so forth. Also needed will be the contents of any files that were specified. This might be files on the file system, but could also be 'files' specified as strings through the slang API. Lastly we need any files that were referenced as part of the compilation - this could be include files, or module source files and so forth. All of this information is bundled up together into a file that can then later be loaded and compiled. This is broadly speaking all of the data that is stored within a repro file. 
+
+In order to capture a complete repro file a typically a compilation has to be attempted. The state before compilation can be recorded (through the API for example), but it may not be enough to repeat a compilation, as files referenced by the compilation would not yet have been accessed. The repro feature records all of these accesses and contents of such files such that compilation can either be completed or at least to the same point as was reached on the host machine. 
+
+One of the more subtle issues around reproducing a compilation is around filenames. That using the API a client can specify source files without names, or multiple files with the same name. If files are loaded via `ISlangFileSystem`, they are typically part of a hiearchical file system. This could mean they are referenced relatively. That there can be distinct files with the same name, differenciated by directory. That the files may not easily be reconstructed back into a similar hieararchical file system - as depending on the include paths (or perhaps other mechanisms) the 'files' and their contents could be arranged in a manner very hard to replicate. To work around this the repro feature does not attempt to replicate a hierarchical file system. Instead it gives every file a unique name, based on their original name. If there are multiple files with the same name it will 'uniquify' them by appending an index. Doing so means that the contents of the file system can just be held as a flat collection of files. This is not enough to enable repeating the compilation though, as we now need Slang to know which files to reference when they are requested, as they are now no longer part of a hierarchical file system and their names may have been altered. To achieve this the repro functionality stores off a map of all path requests to their contents (or lack there of). Doing so means that the file system still appears to Slang as it did in the original compilation, even with all the files being actually stored using the simpler 'flat' arrangement. 
+
+This means that when a repro is 'extracted' it does so to a directory which holds the files with their unique 'flat' names. The name of the directory is the name of the repro file without it's extension, or if it has no extension, with the postfix '-files'. This directory will be referered to from now on as the `repro directory`.
+
+When a repro is loaded, before files are loaded from the repro itself, they will first be looked for via their unique names in the `repro directory`. If they are not there the contents of the repro file will be used. If they are there, their contents will be used instead of the contents in the repro. This provides a simple mechanism to be able to alter the source in a repro. The steps more concretely would be...
+
+1) First extract the repro (say with `-extract-repro`)
+2) Go to the `repro directory` and edit files that you wish to change. You can also just delete files that do not need changing, as they will be loaded from the repro.
+3) Load the repro - it will now load any files requested from the `repro directory`
+
+Note that currently is is disabled to access any new source files - they will be determined as `not found`. This behaviour could be changed such that the regular file system was used, or the ISlangFilesystem set on the API is used as a fallback. 
+
+There currently isn't a mechanism to alter the options of a repro from the command line (other than altering the contents of the source). The reason for this is because of how command lines are processed currently in Slang. A future update could enable specifying a repro and then altering the command line options used. It can be achieved through the API though. Once the repro is loaded via the `spLoadRepro` function, options can be changed as normal. The two major places where option alteration may have surprising behavior are... 
+
+1) Altering the include paths - unless this may break the mechanism used to map paths to files stored in the repro file
+2) Altering the ISlangFileSystem. That to make the contents of the file system appear to be that of the repro, slang uses a ISlangFileSystemExt that uses the contents of the repro file and/or the `repro directory`. If you replace the file system this mechanism will no longer work. 
+
+There are currently two API calls for using the repro functionality 
+
+```
+SLANG_API SlangResult spLoadRepro(
+        SlangCompileRequest* request,
+        ISlangFileSystem* fileSystem,
+        const void* data,
+        size_t size);
+
+SLANG_API SlangResult spSaveRepro(
+        SlangCompileRequest* request,
+        ISlangBlob** outBlob
+    );
+```
+
+The fileSystem parameter passed to `spLoadRepro` provides the mechanism for client code to replace the files that are held within the repro. NOTE! That the files will be loaded from this file system with their `unique names` as if they are part of the flat file system. If an attempt to load a file fails, the file within the repro is used. That `spLoadRepro` is typically performed on a new 'unused' SlangCompileRequest.
+
+That after `spLoadRepro`, normal functions to alter the state of the SlangCompileRequest are available. 
+
+Repro files are currently stored in a binary format. This format is sensitive to changes in the API, as well as internal state within a SlangCompileRequest. This means that the functionality can only be guarenteed to work with exactly the same version of Slang on the same version of compiler. In practice things are typically not so draconian, and future versions will aim to provide a more clear slang repro versioning system, and work will be performed to make more generally usable. 
+
+Finally this version of the repo system does not take into account endianess at all. The system the repro is saved from must have the same endianess as the system loaded on.
diff --git a/slang.h b/slang.h
index 970150edf..a7e4b8aa7 100644
--- a/slang.h
+++ b/slang.h
@@ -1618,10 +1618,13 @@ extern "C"
     
     /** Load repro from memory specified.
 
-    Should only be performed on a newly created request. 
+    Should only be performed on a newly created request.
+
+    NOTE! When using the fileSystem, files will be loaded via their `unique names` as if they are part of the flat file system. This
+    mechanism is described more fully in docs/repro.md.
 
     @param request          The request
-    @param fileSystem       An (optional) filesystem. If non null, files from repro will be loaded (by unique names) from the file system
+    @param fileSystem       An (optional) filesystem. Pass nullptr to just use contents of repro held in data.
     @param data             The data to load from.
     @param size             The size of the data to load from. 
     @returns                A `SlangResult` to indicate success or failure.