Shader caching (#2432)

* Changed all getEntryPointCode calls to use RendererBase::getEntryPointCodeFromShaderCache

* Hashing hooked up, tests pass but need to add more to fully test functionality

* checkpoint

* Checkpoint: File system creation seems functional, saving is broken

* checkpoint: Fixed filename generation from MD5 hash, shader blob might be going missing ahead of pipeline state creation

* Fixed a lot of bugs related to hash code generation, shader cache is likely working but needs further testing

* Added workaround for module loading by re-creating the test device, shader cache test functional

* Vulkan shader caching bug fixed, checkpoint commit before more refinement

* pre-ToT merge checkpoint

* checkpoint commit, improving cache keys

* Significantly expanded items included in the dependency hash for Module; Added dependency hash functions to SpecializedComponentType and RenamedEntryPointComponentType

* Temporarily disable shader cache test

* Mid cleanup changes, solution successfully builds

* Added several helper update functions to slang-md5 to help simplify usage; Added a function under ISession to compute a hash for all linkage-related items; Function renames and cleaned up some comments

* Ran premake.bat; Renamed getASTBasedHashCode to computeASTBasedHash

* Added slang unit tests for Checksum and MD5; Extended gfx shader cache test to test with multiple shader files and one shader file with multiple entry points

* Solution builds and shader cache tests pass, but at least a couple other tests now failing

* ran premake.bat

* More cleanup changes

* Added shaderCachePath field to IDevice desc in gfx.slang, gfx-smoke.slang should be functional

* ran premake

* cleanup changes; Adding test printf to getEntryPointCodeFromShaderCache to see if output can be seen in CI

* Removed debugging printfs; Added handling for getEntryPointCode() failing

* Cleanup changes; Jonathan's fixes to SerialWriter to zero initialize otherwise uninitialized memory; Change to SwizzleExpr creation to zero initialize elementCount

* Changed enable_if_t to enable_if

* Fixed enable_if

* Added test for import vs include and changes to included and imported files; Fixed build errors in CUDA; Renamed shader cache statistics fields

* cleanup changes

* Readd removed file

* Restructured computeDependencyBasedHash calls, added computeDependencyBasedHashImpl to all classes dervied from ComponentType

* Applied same restructuring to the AST hash functions

* Cleanup changes; Moved HashBuilder out to slang-digest.h and added some helper functions to streamline the process of adding items to a hash

* Cleanup; Fixed incorrect expected results for shader import and include test

1 files changed, 82 insertions, 0 deletions

diff --git a/source/core/slang-md5.h b/source/core/slang-md5.h
new file mode 100644
index 000000000..8c51a03ec
--- /dev/null
+++ b/source/core/slang-md5.h
@@ -0,0 +1,82 @@
+/*
+ * This is an OpenSSL-compatible implementation of the RSA Data Security, Inc.
+ * MD5 Message-Digest Algorithm (RFC 1321).
+ *
+ * Homepage:
+ * http://openwall.info/wiki/people/solar/software/public-domain-source-code/md5
+ *
+ * Author:
+ * Alexander Peslyak, better known as Solar Designer <solar at openwall.com>
+ *
+ * This software was written by Alexander Peslyak in 2001.  No copyright is
+ * claimed, and the software is hereby placed in the public domain.
+ * In case this attempt to disclaim copyright and place the software in the
+ * public domain is deemed null and void, then the software is
+ * Copyright (c) 2001 Alexander Peslyak and it is hereby released to the
+ * general public under the following terms:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted.
+ *
+ * There's ABSOLUTELY NO WARRANTY, express or implied.
+ *
+ * See md5.c for more information.
+ */
+ 
+#ifdef HAVE_OPENSSL
+#include <openssl/md5.h>
+#elif !defined(_MD5_H)
+#define _MD5_H
+
+#pragma once
+#include "../../slang.h"
+#include "../core/slang-string.h"
+#include "../core/slang-list.h"
+
+namespace Slang
+{
+    /* Any 32-bit or wider unsigned integer data type will do */
+    typedef uint32_t MD5_u32plus;
+
+    struct MD5Context
+    {
+        MD5_u32plus lo, hi;
+        MD5_u32plus a, b, c, d;
+        unsigned char buffer[64];
+        MD5_u32plus block[16];
+    };
+
+    class MD5HashGen
+    {
+    public:
+        void init(MD5Context* ctx);
+
+        // Helper update function for raw values (e.g. ints, uints)
+        template<typename T,
+            typename = typename std::enable_if<std::is_enum<T>::value || std::is_arithmetic<T>::value>::type>
+        void update(MD5Context* ctx, const T& val)
+        {
+            update(ctx, &val, sizeof(T));
+        }
+        // Helper update function for Slang::List
+        template<typename T>
+        void update(MD5Context* ctx, const List<T>& list)
+        {
+            update(ctx, list.getBuffer(), list.getCount());
+        }
+        // Helper update function for UnownedStringSlice
+        void update(MD5Context* ctx, UnownedStringSlice string);
+        // Helper update function for Slang::String
+        void update(MD5Context* ctx, String str);
+        // Helper update function for Checksums
+        void update(MD5Context* ctx, const slang::Digest& checksum);
+
+        void finalize(MD5Context* ctx, slang::Digest* result);
+
+    private:
+        static const void* body(MD5Context* ctx, const void* data, SlangInt size);
+        void update(MD5Context* ctx, const void* data, SlangInt size);
+    };
+}
+ 
+#endif