Shader caching (#2432)

* Changed all getEntryPointCode calls to use RendererBase::getEntryPointCodeFromShaderCache

* Hashing hooked up, tests pass but need to add more to fully test functionality

* checkpoint

* Checkpoint: File system creation seems functional, saving is broken

* checkpoint: Fixed filename generation from MD5 hash, shader blob might be going missing ahead of pipeline state creation

* Fixed a lot of bugs related to hash code generation, shader cache is likely working but needs further testing

* Added workaround for module loading by re-creating the test device, shader cache test functional

* Vulkan shader caching bug fixed, checkpoint commit before more refinement

* pre-ToT merge checkpoint

* checkpoint commit, improving cache keys

* Significantly expanded items included in the dependency hash for Module; Added dependency hash functions to SpecializedComponentType and RenamedEntryPointComponentType

* Temporarily disable shader cache test

* Mid cleanup changes, solution successfully builds

* Added several helper update functions to slang-md5 to help simplify usage; Added a function under ISession to compute a hash for all linkage-related items; Function renames and cleaned up some comments

* Ran premake.bat; Renamed getASTBasedHashCode to computeASTBasedHash

* Added slang unit tests for Checksum and MD5; Extended gfx shader cache test to test with multiple shader files and one shader file with multiple entry points

* Solution builds and shader cache tests pass, but at least a couple other tests now failing

* ran premake.bat

* More cleanup changes

* Added shaderCachePath field to IDevice desc in gfx.slang, gfx-smoke.slang should be functional

* ran premake

* cleanup changes; Adding test printf to getEntryPointCodeFromShaderCache to see if output can be seen in CI

* Removed debugging printfs; Added handling for getEntryPointCode() failing

* Cleanup changes; Jonathan's fixes to SerialWriter to zero initialize otherwise uninitialized memory; Change to SwizzleExpr creation to zero initialize elementCount

* Changed enable_if_t to enable_if

* Fixed enable_if

* Added test for import vs include and changes to included and imported files; Fixed build errors in CUDA; Renamed shader cache statistics fields

* cleanup changes

* Readd removed file

* Restructured computeDependencyBasedHash calls, added computeDependencyBasedHashImpl to all classes dervied from ComponentType

* Applied same restructuring to the AST hash functions

* Cleanup changes; Moved HashBuilder out to slang-digest.h and added some helper functions to streamline the process of adding items to a hash

* Cleanup; Fixed incorrect expected results for shader import and include test

1 files changed, 23 insertions, 2 deletions

diff --git a/source/slang/slang-serialize.h b/source/slang/slang-serialize.h
index fcd2daa1f..e08d26dd5 100644
--- a/source/slang/slang-serialize.h
+++ b/source/slang/slang-serialize.h
@@ -316,6 +316,18 @@ void SerialReader::getArray(SerialIndex index, List<T>& out)
 class SerialWriter : public RefObject
 {
 public:
+    typedef uint32_t Flags;
+    struct Flag
+    {
+        enum Enum : Flags
+        {
+                /// If set will zero initialize backing memory. This is slower but 
+                /// is desirable to make two serializations of the same thing produce the 
+                /// identical serialized result.
+            ZeroInitialize = 0x1
+        };
+    };
+
     SerialIndex addPointer(const NodeBase* ptr);
     SerialIndex addPointer(const RefObject* ptr);
 
@@ -367,8 +379,11 @@ public:
         /// Used for attaching extra objects necessary for serializing
     SerialExtraObjects& getExtraObjects() { return m_extraObjects; }
 
+        /// Get the flag
+    Flags getFlags() const { return m_flags; }
+
         /// Ctor
-    SerialWriter(SerialClasses* classes, SerialFilter* filter);
+    SerialWriter(SerialClasses* classes, SerialFilter* filter, Flags flags = Flag::ZeroInitialize);
 
 protected:
 
@@ -388,7 +403,6 @@ protected:
 
     Dictionary<const void*, Index> m_ptrMap;    // Maps a pointer to an entry index
 
-    
     // NOTE! Assumes the content stays in scope!
     SliceMap m_sliceMap;
     SliceMap m_importSymbolMap;
@@ -399,6 +413,8 @@ protected:
     MemoryArena m_arena;                    ///< Holds the payloads
     SerialClasses* m_classes;
     SerialFilter* m_filter;                 ///< Filter to control what is serialized
+
+    Flags m_flags;                          ///< Flags to control behavior
 };
 
 // ---------------------------------------------------------------------------
@@ -419,6 +435,11 @@ SerialIndex SerialWriter::addArray(const T* in, Index count)
         List<ElementSerialType> work;
         work.setCount(count);
 
+        if (getFlags() & Flag::ZeroInitialize)
+        {
+            ::memset(work.getBuffer(), 0, sizeof(ElementSerialType) * count);
+        }
+
         for (Index i = 0; i < count; ++i)
         {
             ElementTypeInfo::toSerial(this, &in[i], &work[i]);