Add support for "blobs" and a file-system callback (#596)

* Add support for "blobs" and a file-system callback

The most obvious change here is that the Slang header now includes a few COM-style interfaces that can be used for communication between the application and compiler. In order to support the declaration of COM-like interfaces, several platform-detection macros were lifted out of `slang-defines.h` and into the public `slang.h` header. As it exists right now, this change makes the Slang API C++-only, but a C-compatible version can be defined later with the help of lots of macros (and/or something like an IDL compiler).

The two big interfaces introduced are:

* The `ISlangBlob` interface, which is compatible with `ID3DBlob`, `IDxcBlob`, etc. This is used to pass ownership of source/compiled code across the API boundary without copies. New versions of various entry points have been added to allow passing blobs: e.g., `spAddTranslationUnitSourceBlob` and `spGetEntryPointCodeBlob`.

* The `ISlangFileSystem` interface, which is used to allow applications to intercept any attempt by the Slang compiler to load a file (input source files, include files, etc.). This is *not* the same as the `IDxcIncludeHandler` interface, because it assumes UTF-8 encoded path names, instead of the 16-bit encoding that dxc/Windows prefer. It is also not very similar to `ID3DInclude` as used by fxc, because this callback interface is *not* responsible for handling the search through include paths, etc. - it is just a file-system abstraction layer.

Internally, a few different parts of the compiler were changed to either store data in blob form all the time, or to be able to synthesize a blob on-demand. Because our internal `String` type is a reference-counted copy-on-write type, using a `SlangStringBlob` to hold string data should achieve transfer of ownership back to the application without extraneous copies. There is plenty of room to clean up the architecture of some of these internal pieces if they *know* that their data will end up in a blob.

The existing Slang testing doesn't touch any of the APIs introduced here, so they can only confirm that existing functionality hasn't been broken. The new ability to return code blobs has been tested by integration of that feature into Falcor, but there has been zero testing of the ability to pass *in* source code as blobs, and the ability to hook file loading. Future changes will need to add test coverage for the new features.

* fixup: define SLANG_NO_THROW for non-Windows builds

* fixup: header copy-paste error caught by clang/gcc

* Cleanup: return reference-counted objects via output parameters

Returning a reference-counted object through the API as a raw pointer creates challenges.
The "obvious" answer is that the returned pointer should have an added reference (it is returned at "+1"), and the caller is responsible for releasing that reference. This makes sense when using raw pointers on the calling side:

```c++
IFoo* foo = spGetFoo(...);
...
foo->Release();
```

However, as soon as smart pointers start getting involved (to handle releasing reference counts when we are done with things), the picture gets more complicated:

```c++
MySmartPtr<IFoo> foo = spGetFoo(...);
...
```

The intention of code like that is that `foo` gets released when the smart pointer goes out of scope, but this probably doesn't happen with most smart pointer implementations. If the `MySmartPtr` constructor that takes a raw pointer retains it, then the destructor will only release *that* reference, and so the object will leak.

It is possible that the user will have a smart pointer type where the constructor that takes a raw pointer doesn't retain it, but in general such types introduce the potential for errors of their own, and no matter what the Slang API shouldn't go in assuming any particular policy.

This change makes it so that any reference-counted objects that are logically returned from a call are returned through output pointers. This design makes the leak-free cases easy (enough) to implement with raw pointers or smart pointers:

```c++
// raw pointer
IFoo* foo = nullptr;
spGetFoo(..., &foo);
...
foo->Release();

// smart pointer
MySmartPtr<IFoo> foo;
spGetFoo(..., foo.writeableRef());
...
```

The only assumption here is that any COM smart-pointer type needs to provide an operation like `writableRef` that is suitable for using that pointer as an output parameter. Given that COM *loves* output parameters, this seems like a safe assumption (at the very least, anybody who interacts with COM would be used to this convention).

Future changes might introduce inline convenience methods for various operations that return results more directly, possibly by introducing a minimal smart-pointer type in the `slang.h` header (without prescribing that clients must use it...).

* fixup: another error caught by gcc/clang

3 files changed, 46 insertions, 194 deletions

diff --git a/source/core/slang-com-ptr.h b/source/core/slang-com-ptr.h
index 9f6651306..765edbee5 100644
--- a/source/core/slang-com-ptr.h
+++ b/source/core/slang-com-ptr.h
@@ -8,17 +8,17 @@
 
 namespace Slang {
 
-/*! \brief ComPtr is a simple smart pointer that manages types which implement COM based interfaces. 
+/*! \brief ComPtr is a simple smart pointer that manages types which implement COM based interfaces.
 \details A class that implements a COM, must derive from the IUnknown interface or a type that matches
 it's layout exactly (such as IForwardUnknown). Trying to use this template with a class that doesn't follow
-these rules, will lead to undefined behavior. 
-This is a 'strong' pointer type, and will AddRef when a non null pointer is set and Release when the pointer 
-leaves scope. 
+these rules, will lead to undefined behavior.
+This is a 'strong' pointer type, and will AddRef when a non null pointer is set and Release when the pointer
+leaves scope.
 Using 'detach' allows a pointer to be removed from the management of the ComPtr.
 To set the smart pointer to null, there is the method setNull, or alternatively just assign SLANG_NULL/nullptr.
 
 One edge case using the template is that sometimes you want access as a pointer to a pointer. Sometimes this
-is to write into the smart pointer, other times to pass as an array. To handle these different behaviors 
+is to write into the smart pointer, other times to pass as an array. To handle these different behaviors
 there are the methods readRef and writeRef, which are used instead of the & (ref) operator. For example
 
 \code
@@ -31,7 +31,7 @@ ComPtr<ID3D12Resource> resources[3];
 doSomething(resources[0].readRef(), SLANG_COUNT_OF(resource));
 \endcode
 
-A more common scenario writing to the pointer 
+A more common scenario writing to the pointer
 
 \code
 IUnknown* unk = ...;
@@ -40,35 +40,29 @@ ComPtr<ID3D12Resource> resource;
 Result res = unk->QueryInterface(resource.writeRef());
 \endcode
 */
- 
-struct Guid
-{
-    uint32_t data1;     ///< Low field of the timestamp
-    uint16_t data2;     ///< Middle field of the timestamp
-    uint16_t data3;     ///< High field of the timestamp with multiplexed version number
-    uint8_t  data4[8];  ///< 0, 1 = clock_seq_hi_and_reserved, clock_seq_low, followed by 'spatially unique node' (48 bits) 
-};
+
+typedef SlangUUID Guid;
 
 SLANG_FORCE_INLINE bool operator==(const Guid& aIn, const Guid& bIn)
 {
-	// Use the largest type the honors the alignment of Guid
-	typedef uint32_t CmpType;
-    struct GuidCompare
+    // Use the largest type the honors the alignment of Guid
+    typedef uint32_t CmpType;
+    union GuidCompare
     {
         Guid guid;
-		CmpType data[sizeof(Guid) / sizeof(CmpType)];
+        CmpType data[sizeof(Guid) / sizeof(CmpType)];
     };
-	// Type pun - so compiler can 'see' the pun and not break aliasing rules 
+    // Type pun - so compiler can 'see' the pun and not break aliasing rules
     const CmpType* a = reinterpret_cast<const GuidCompare&>(aIn).data;
     const CmpType* b = reinterpret_cast<const GuidCompare&>(bIn).data;
-	// Make the guid comparison a single branch, by not using short circuit
-	return ((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]) | (a[3] ^ b[3])) == 0;
-} 
+    // Make the guid comparison a single branch, by not using short circuit
+    return ((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]) | (a[3] ^ b[3])) == 0;
+}
 
 SLANG_FORCE_INLINE bool operator!=(const Guid& a, const Guid& b)
 {
     return !(a == b);
-} 
+}
 
 // Allows for defining of a GUID that works in C++ and C which defines in a format similar to microsofts INTERFACE style
 // MIDL_INTERFACE("00000000-0000-0000-C000-00 00 00 00 00 46")
@@ -84,16 +78,9 @@ SLANG_FORCE_INLINE bool operator!=(const Guid& a, const Guid& b)
 // Compatible with Microsoft IUnknown
 static const Guid IID_IComUnknown = SLANG_MAKE_GUID(00000000, 0000, 0000, C000, 000000000046);
 
-/// ! Must be kept in sync with IUnknown
-class IComUnknown
-{
-public:
-    virtual SLANG_NO_THROW Result SLANG_MCALL queryInterface(const Guid& iid, void* objOut) = 0;
-    virtual SLANG_NO_THROW uint32_t SLANG_MCALL addRef() = 0;
-    virtual SLANG_NO_THROW uint32_t SLANG_MCALL release() = 0;
-};
+typedef ISlangUnknown IComUnknown;
 
-// Enum to force initializing as an attach (without adding a reference) 
+// Enum to force initializing as an attach (without adding a reference)
 enum InitAttach
 {
     INIT_ATTACH
@@ -139,7 +126,7 @@ public:
 		/// For making method invocations through the smart pointer work through the dumb pointer
 	SLANG_FORCE_INLINE T* operator->() const { return m_ptr; }
 
-		/// Assign 
+		/// Assign
 	SLANG_FORCE_INLINE const ThisType &operator=(const ThisType& rhs);
 		/// Assign from dumb ptr
 	SLANG_FORCE_INLINE T* operator=(T* in);
@@ -202,7 +189,7 @@ T* ComPtr<T>::operator=(T* ptr)
 {
 	if (ptr) ((Ptr)ptr)->addRef();
 	if (m_ptr) ((Ptr)m_ptr)->release();
-	m_ptr = ptr;           
+	m_ptr = ptr;
 	return m_ptr;
 }
 //----------------------------------------------------------------------------
@@ -214,6 +201,6 @@ void ComPtr<T>::swap(ThisType& rhs)
 	rhs.m_ptr = tmp;
 }
 
-} // namespace Slang 
+} // namespace Slang
 
 #endif // SLANG_COM_PTR_H
diff --git a/source/core/slang-defines.h b/source/core/slang-defines.h
index 98993d662..580dd801c 100644
--- a/source/core/slang-defines.h
+++ b/source/core/slang-defines.h
@@ -2,140 +2,15 @@
 #define SLANG_DEFINES_H
 
 /*
-The following preprocessor identifiers specify compiler, OS, and architecture.
-All definitions have a value of 1 or 0, use '#if' instead of '#ifdef'.
+Some of our `#define`s are needed in the public API header as well, so
+we will go ahead and include that here so that we can share the definitions.
 */
-
-#ifndef SLANG_COMPILER
-#	define SLANG_COMPILER
+#include "../../slang.h"
 
 /*
-Compiler defines, see http://sourceforge.net/p/predef/wiki/Compilers/
-NOTE that SLANG_VC holds the compiler version - not just 1 or 0
-*/
-#	if defined(_MSC_VER)
-#		if _MSC_VER >= 1900
-#			define SLANG_VC 14
-#		elif _MSC_VER >= 1800
-#			define SLANG_VC 12
-#		elif _MSC_VER >= 1700
-#			define SLANG_VC 11
-#		elif _MSC_VER >= 1600
-#			define SLANG_VC 10
-#		elif _MSC_VER >= 1500
-#			define SLANG_VC 9
-#		else
-#			error "Unknown VC version"
-#		endif
-#	elif defined(__clang__)
-#		define SLANG_CLANG 1
-#	elif defined(__SNC__)
-#		define SLANG_SNC 1
-#	elif defined(__ghs__)
-#		define SLANG_GHS 1
-#	elif defined(__GNUC__) // note: __clang__, __SNC__, or __ghs__ imply __GNUC__
-#		define SLANG_GCC 1
-#	else
-#		error "Unknown compiler"
-#	endif
-
-// Zero unset
-#	ifndef SLANG_VC
-#		define SLANG_VC 0
-#	endif
-#	ifndef SLANG_CLANG
-#		define SLANG_CLANG 0
-#	endif
-#	ifndef SLANG_SNC
-#		define SLANG_SNC 0
-#	endif
-#	ifndef SLANG_GHS
-#		define SLANG_GHS 0
-#	endif
-#	ifndef SLANG_GCC
-#		define SLANG_GCC 0
-#	endif
-
-#endif // SLANG_COMPILER
-
-#ifndef SLANG_PLATFORM
-#	define SLANG_PLATFORM
-
-/**
-Operating system defines, see http://sourceforge.net/p/predef/wiki/OperatingSystems/
+The following preprocessor identifiers specify compiler, OS, and architecture.
+All definitions have a value of 1 or 0, use '#if' instead of '#ifdef'.
 */
-#	if defined(WINAPI_FAMILY) && WINAPI_FAMILY == WINAPI_PARTITION_APP
-#		define SLANG_WINRT 1 // Windows Runtime, either on Windows RT or Windows 8
-#	elif defined(XBOXONE)
-#		define SLANG_XBOXONE 1
-#	elif defined(_WIN64) // note: XBOXONE implies _WIN64
-#		define SLANG_WIN64 1
-#	elif defined(_M_PPC)
-#		define SLANG_X360 1
-#	elif defined(_WIN32) // note: _M_PPC implies _WIN32
-#		define SLANG_WIN32 1
-#	elif defined(__ANDROID__)
-#		define SLANG_ANDROID 1
-#	elif defined(__linux__) || defined(__CYGWIN__) // note: __ANDROID__ implies __linux__
-#		define SLANG_LINUX 1
-#	elif defined(__APPLE__) && (defined(__arm__) || defined(__arm64__))
-#		define SLANG_IOS 1
-#	elif defined(__APPLE__)
-#		define SLANG_OSX 1
-#	elif defined(__CELLOS_LV2__)
-#		define SLANG_PS3 1
-#	elif defined(__ORBIS__)
-#		define SLANG_PS4 1
-#	elif defined(__SNC__) && defined(__arm__)
-#		define SLANG_PSP2 1
-#	elif defined(__ghs__)
-#		define SLANG_WIIU 1
-#	else
-#		error "Unknown operating system"
-#	endif
-
-// zero unset
-#	ifndef SLANG_WINRT
-#		define SLANG_WINRT 0
-#	endif
-#	ifndef SLANG_XBOXONE
-#		define SLANG_XBOXONE 0
-#	endif
-#	ifndef SLANG_WIN64
-#		define SLANG_WIN64 0
-#	endif
-#	ifndef SLANG_X360
-#		define SLANG_X360 0
-#	endif
-#	ifndef SLANG_WIN32
-#		define SLANG_WIN32 0
-#	endif
-#	ifndef SLANG_ANDROID
-#		define SLANG_ANDROID 0
-#	endif
-#	ifndef SLANG_LINUX
-#		define SLANG_LINUX 0
-#	endif
-#	ifndef SLANG_IOS
-#		define SLANG_IOS 0
-#	endif
-#	ifndef SLANG_OSX
-#		define SLANG_OSX 0
-#	endif
-#	ifndef SLANG_PS3
-#		define SLANG_PS3 0
-#	endif
-#	ifndef SLANG_PS4
-#		define SLANG_PS4 0
-#	endif
-#	ifndef SLANG_PSP2
-#		define SLANG_PSP2 0
-#	endif
-#	ifndef SLANG_WIIU
-#		define SLANG_WIIU 0
-#	endif
-
-#endif // SLANG_PLATFORM
 
 #ifndef SLANG_PROCESSOR
 #	define SLANG_PROCESSOR
@@ -217,15 +92,6 @@ define anything not defined through the command line to 0
 /**
 family shortcuts
 */
-// compiler
-#define SLANG_GCC_FAMILY (SLANG_CLANG || SLANG_SNC || SLANG_GHS || SLANG_GCC)
-
-// os
-#define SLANG_WINDOWS_FAMILY (SLANG_WINRT || SLANG_WIN32 || SLANG_WIN64)
-#define SLANG_MICROSOFT_FAMILY (SLANG_XBOXONE || SLANG_X360 || SLANG_WINDOWS_FAMILY)
-#define SLANG_LINUX_FAMILY (SLANG_LINUX || SLANG_ANDROID)
-#define SLANG_APPLE_FAMILY (SLANG_IOS || SLANG_OSX)                  // equivalent to #if __APPLE__
-#define SLANG_UNIX_FAMILY (SLANG_LINUX_FAMILY || SLANG_APPLE_FAMILY) // shortcut for unix/posix platforms
 // architecture
 #define SLANG_INTEL_FAMILY (SLANG_X64 || SLANG_X86)					// Intel x86 family
 #define SLANG_ARM_FAMILY (SLANG_ARM || SLANG_A64)
@@ -233,9 +99,9 @@ family shortcuts
 
 #define SLANG_P64_FAMILY (SLANG_X64 || SLANG_A64) // shortcut for 64-bit architectures
 
-// Use for getting the amount of members of a standard C array. 
+// Use for getting the amount of members of a standard C array.
 #define SLANG_COUNT_OF(x) (sizeof(x)/sizeof(x[0]))
-/// SLANG_INLINE exists to have a way to inline consistent with SLANG_ALWAYS_INLINE  
+/// SLANG_INLINE exists to have a way to inline consistent with SLANG_ALWAYS_INLINE
 #define SLANG_INLINE inline
 
 // Other defines
@@ -253,7 +119,7 @@ family shortcuts
 General defines
 */
 
-// GCC Specific 
+// GCC Specific
 #if SLANG_GCC_FAMILY
 #	define SLANG_NO_INLINE __attribute__((noinline))
 
@@ -264,7 +130,7 @@ General defines
 
 #	if !SLANG_SNC && !SLANG_GHS
 #		define SLANG_OFFSET_OF(X, Y) __builtin_offsetof(X, Y)
-#	endif 
+#	endif
 
 //#	if !SLANG_LINUX // Workaround; Fedora Core 3 do not agree with force inline
 #	define SLANG_FORCE_INLINE inline __attribute__((always_inline))
@@ -314,7 +180,6 @@ General defines
 #	define SLANG_INT64(x) (x##i64)
 #	define SLANG_UINT64(x) (x##ui64)
 
-#	define SLANG_STDCALL __stdcall
 #	define SLANG_CALL_CONV __cdecl
 
 #endif // SLANG_MICROSOFT_FAMILY
@@ -340,7 +205,7 @@ General defines
 #ifndef SLANG_FORCE_INLINE
 #	define SLANG_FORCE_INLINE inline
 #endif
-#ifndef SLANG_NO_INLINE 
+#ifndef SLANG_NO_INLINE
 #	define SLANG_NO_INLINE
 #endif
 #ifndef SLANG_NO_ALIAS
@@ -364,15 +229,12 @@ General defines
 #	define SLANG_FUNCTION_SIG SLANG_FUNCTION_NAME
 #endif
 
-#ifndef SLANG_STDCALL
-#	define SLANG_STDCALL 
-#endif
 #ifndef SLANG_CALL_CONV
 #	define SLANG_CALL_CONV
 #endif
 
 //! casting the null ptr takes a special-case code path, which we don't want
-#define SLANG_OFFSETOF_BASE 0x100 
+#define SLANG_OFFSETOF_BASE 0x100
 #define SLANG_OFFSET_OF_RT(Class, Member)                                                                                 \
 	(reinterpret_cast<size_t>(&reinterpret_cast<Class*>(SLANG_OFFSETOF_BASE)->Member) - size_t(SLANG_OFFSETOF_BASE))
 
@@ -404,18 +266,13 @@ General defines
 #				define SLANG_HAS_ENUM_CLASS 1
 #			endif
 #			if (__GNUC__ * 100 + __GNUC_MINOR__) >= 407
-#				define SLANG_OVERRIDE override 
+#				define SLANG_OVERRIDE override
 #			endif
 #		endif
 #	endif // SLANG_GCC_FAMILY
 
 // Visual Studio
 
-// Macro for declaring if a method is no throw. Should be set before the return parameter. 
-#if SLANG_WINDOWS_FAMILY && !defined(SLANG_DISABLE_EXCEPTIONS)
-#	define SLANG_NO_THROW __declspec(nothrow) 
-#endif
-
 #	if SLANG_VC
 // C4481: nonstandard extension used: override specifier 'override'
 #		if _MSC_VER < 1700
@@ -434,7 +291,7 @@ General defines
 #	if SLANG_CLANG
 #	endif // SLANG_CLANG
 
-// Set non set  
+// Set non set
 
 #ifndef SLANG_NO_THROW
 #	define SLANG_NO_THROW
@@ -449,8 +306,6 @@ General defines
 #	define SLANG_HAS_MOVE_SEMANTICS 0
 #endif
 
-#define SLANG_MCALL SLANG_STDCALL
-
 #include <new>		// For placement new
 
 #endif // __cplusplus
diff --git a/source/core/slang-string.h b/source/core/slang-string.h
index 894776ca6..a67597360 100644
--- a/source/core/slang-string.h
+++ b/source/core/slang-string.h
@@ -172,7 +172,7 @@ namespace Slang
                 {
                     return i;
                 }
-            } 
+            }
             return -1;
         }
 
@@ -412,6 +412,11 @@ namespace Slang
             append(slice);
         }
 
+        String(UnownedStringSlice const& slice)
+        {
+            append(slice);
+        }
+
 		~String()
 		{
             buffer = 0;
@@ -741,6 +746,11 @@ namespace Slang
 			Append(str);
 			return *this;
 		}
+		StringBuilder & operator << (UnownedStringSlice const& str)
+		{
+			append(str);
+			return *this;
+		}
 		StringBuilder & operator << (const _EndLine)
 		{
 			Append('\n');