Add support for "blobs" and a file-system callback (#596)

* Add support for "blobs" and a file-system callback

The most obvious change here is that the Slang header now includes a few COM-style interfaces that can be used for communication between the application and compiler. In order to support the declaration of COM-like interfaces, several platform-detection macros were lifted out of `slang-defines.h` and into the public `slang.h` header. As it exists right now, this change makes the Slang API C++-only, but a C-compatible version can be defined later with the help of lots of macros (and/or something like an IDL compiler).

The two big interfaces introduced are:

* The `ISlangBlob` interface, which is compatible with `ID3DBlob`, `IDxcBlob`, etc. This is used to pass ownership of source/compiled code across the API boundary without copies. New versions of various entry points have been added to allow passing blobs: e.g., `spAddTranslationUnitSourceBlob` and `spGetEntryPointCodeBlob`.

* The `ISlangFileSystem` interface, which is used to allow applications to intercept any attempt by the Slang compiler to load a file (input source files, include files, etc.). This is *not* the same as the `IDxcIncludeHandler` interface, because it assumes UTF-8 encoded path names, instead of the 16-bit encoding that dxc/Windows prefer. It is also not very similar to `ID3DInclude` as used by fxc, because this callback interface is *not* responsible for handling the search through include paths, etc. - it is just a file-system abstraction layer.

Internally, a few different parts of the compiler were changed to either store data in blob form all the time, or to be able to synthesize a blob on-demand. Because our internal `String` type is a reference-counted copy-on-write type, using a `SlangStringBlob` to hold string data should achieve transfer of ownership back to the application without extraneous copies. There is plenty of room to clean up the architecture of some of these internal pieces if they *know* that their data will end up in a blob.

The existing Slang testing doesn't touch any of the APIs introduced here, so they can only confirm that existing functionality hasn't been broken. The new ability to return code blobs has been tested by integration of that feature into Falcor, but there has been zero testing of the ability to pass *in* source code as blobs, and the ability to hook file loading. Future changes will need to add test coverage for the new features.

* fixup: define SLANG_NO_THROW for non-Windows builds

* fixup: header copy-paste error caught by clang/gcc

* Cleanup: return reference-counted objects via output parameters

Returning a reference-counted object through the API as a raw pointer creates challenges.
The "obvious" answer is that the returned pointer should have an added reference (it is returned at "+1"), and the caller is responsible for releasing that reference. This makes sense when using raw pointers on the calling side:

```c++
IFoo* foo = spGetFoo(...);
...
foo->Release();
```

However, as soon as smart pointers start getting involved (to handle releasing reference counts when we are done with things), the picture gets more complicated:

```c++
MySmartPtr<IFoo> foo = spGetFoo(...);
...
```

The intention of code like that is that `foo` gets released when the smart pointer goes out of scope, but this probably doesn't happen with most smart pointer implementations. If the `MySmartPtr` constructor that takes a raw pointer retains it, then the destructor will only release *that* reference, and so the object will leak.

It is possible that the user will have a smart pointer type where the constructor that takes a raw pointer doesn't retain it, but in general such types introduce the potential for errors of their own, and no matter what the Slang API shouldn't go in assuming any particular policy.

This change makes it so that any reference-counted objects that are logically returned from a call are returned through output pointers. This design makes the leak-free cases easy (enough) to implement with raw pointers or smart pointers:

```c++
// raw pointer
IFoo* foo = nullptr;
spGetFoo(..., &foo);
...
foo->Release();

// smart pointer
MySmartPtr<IFoo> foo;
spGetFoo(..., foo.writeableRef());
...
```

The only assumption here is that any COM smart-pointer type needs to provide an operation like `writableRef` that is suitable for using that pointer as an output parameter. Given that COM *loves* output parameters, this seems like a safe assumption (at the very least, anybody who interacts with COM would be used to this convention).

Future changes might introduce inline convenience methods for various operations that return results more directly, possibly by introducing a minimal smart-pointer type in the `slang.h` header (without prescribing that clients must use it...).

* fixup: another error caught by gcc/clang

10 files changed, 434 insertions, 234 deletions

diff --git a/source/core/slang-com-ptr.h b/source/core/slang-com-ptr.h
index 9f6651306..765edbee5 100644
--- a/source/core/slang-com-ptr.h
+++ b/source/core/slang-com-ptr.h
@@ -8,17 +8,17 @@
 
 namespace Slang {
 
-/*! \brief ComPtr is a simple smart pointer that manages types which implement COM based interfaces. 
+/*! \brief ComPtr is a simple smart pointer that manages types which implement COM based interfaces.
 \details A class that implements a COM, must derive from the IUnknown interface or a type that matches
 it's layout exactly (such as IForwardUnknown). Trying to use this template with a class that doesn't follow
-these rules, will lead to undefined behavior. 
-This is a 'strong' pointer type, and will AddRef when a non null pointer is set and Release when the pointer 
-leaves scope. 
+these rules, will lead to undefined behavior.
+This is a 'strong' pointer type, and will AddRef when a non null pointer is set and Release when the pointer
+leaves scope.
 Using 'detach' allows a pointer to be removed from the management of the ComPtr.
 To set the smart pointer to null, there is the method setNull, or alternatively just assign SLANG_NULL/nullptr.
 
 One edge case using the template is that sometimes you want access as a pointer to a pointer. Sometimes this
-is to write into the smart pointer, other times to pass as an array. To handle these different behaviors 
+is to write into the smart pointer, other times to pass as an array. To handle these different behaviors
 there are the methods readRef and writeRef, which are used instead of the & (ref) operator. For example
 
 \code
@@ -31,7 +31,7 @@ ComPtr<ID3D12Resource> resources[3];
 doSomething(resources[0].readRef(), SLANG_COUNT_OF(resource));
 \endcode
 
-A more common scenario writing to the pointer 
+A more common scenario writing to the pointer
 
 \code
 IUnknown* unk = ...;
@@ -40,35 +40,29 @@ ComPtr<ID3D12Resource> resource;
 Result res = unk->QueryInterface(resource.writeRef());
 \endcode
 */
- 
-struct Guid
-{
-    uint32_t data1;     ///< Low field of the timestamp
-    uint16_t data2;     ///< Middle field of the timestamp
-    uint16_t data3;     ///< High field of the timestamp with multiplexed version number
-    uint8_t  data4[8];  ///< 0, 1 = clock_seq_hi_and_reserved, clock_seq_low, followed by 'spatially unique node' (48 bits) 
-};
+
+typedef SlangUUID Guid;
 
 SLANG_FORCE_INLINE bool operator==(const Guid& aIn, const Guid& bIn)
 {
-	// Use the largest type the honors the alignment of Guid
-	typedef uint32_t CmpType;
-    struct GuidCompare
+    // Use the largest type the honors the alignment of Guid
+    typedef uint32_t CmpType;
+    union GuidCompare
     {
         Guid guid;
-		CmpType data[sizeof(Guid) / sizeof(CmpType)];
+        CmpType data[sizeof(Guid) / sizeof(CmpType)];
     };
-	// Type pun - so compiler can 'see' the pun and not break aliasing rules 
+    // Type pun - so compiler can 'see' the pun and not break aliasing rules
     const CmpType* a = reinterpret_cast<const GuidCompare&>(aIn).data;
     const CmpType* b = reinterpret_cast<const GuidCompare&>(bIn).data;
-	// Make the guid comparison a single branch, by not using short circuit
-	return ((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]) | (a[3] ^ b[3])) == 0;
-} 
+    // Make the guid comparison a single branch, by not using short circuit
+    return ((a[0] ^ b[0]) | (a[1] ^ b[1]) | (a[2] ^ b[2]) | (a[3] ^ b[3])) == 0;
+}
 
 SLANG_FORCE_INLINE bool operator!=(const Guid& a, const Guid& b)
 {
     return !(a == b);
-} 
+}
 
 // Allows for defining of a GUID that works in C++ and C which defines in a format similar to microsofts INTERFACE style
 // MIDL_INTERFACE("00000000-0000-0000-C000-00 00 00 00 00 46")
@@ -84,16 +78,9 @@ SLANG_FORCE_INLINE bool operator!=(const Guid& a, const Guid& b)
 // Compatible with Microsoft IUnknown
 static const Guid IID_IComUnknown = SLANG_MAKE_GUID(00000000, 0000, 0000, C000, 000000000046);
 
-/// ! Must be kept in sync with IUnknown
-class IComUnknown
-{
-public:
-    virtual SLANG_NO_THROW Result SLANG_MCALL queryInterface(const Guid& iid, void* objOut) = 0;
-    virtual SLANG_NO_THROW uint32_t SLANG_MCALL addRef() = 0;
-    virtual SLANG_NO_THROW uint32_t SLANG_MCALL release() = 0;
-};
+typedef ISlangUnknown IComUnknown;
 
-// Enum to force initializing as an attach (without adding a reference) 
+// Enum to force initializing as an attach (without adding a reference)
 enum InitAttach
 {
     INIT_ATTACH
@@ -139,7 +126,7 @@ public:
 		/// For making method invocations through the smart pointer work through the dumb pointer
 	SLANG_FORCE_INLINE T* operator->() const { return m_ptr; }
 
-		/// Assign 
+		/// Assign
 	SLANG_FORCE_INLINE const ThisType &operator=(const ThisType& rhs);
 		/// Assign from dumb ptr
 	SLANG_FORCE_INLINE T* operator=(T* in);
@@ -202,7 +189,7 @@ T* ComPtr<T>::operator=(T* ptr)
 {
 	if (ptr) ((Ptr)ptr)->addRef();
 	if (m_ptr) ((Ptr)m_ptr)->release();
-	m_ptr = ptr;           
+	m_ptr = ptr;
 	return m_ptr;
 }
 //----------------------------------------------------------------------------
@@ -214,6 +201,6 @@ void ComPtr<T>::swap(ThisType& rhs)
 	rhs.m_ptr = tmp;
 }
 
-} // namespace Slang 
+} // namespace Slang
 
 #endif // SLANG_COM_PTR_H
diff --git a/source/core/slang-defines.h b/source/core/slang-defines.h
index 98993d662..580dd801c 100644
--- a/source/core/slang-defines.h
+++ b/source/core/slang-defines.h
@@ -2,140 +2,15 @@
 #define SLANG_DEFINES_H
 
 /*
-The following preprocessor identifiers specify compiler, OS, and architecture.
-All definitions have a value of 1 or 0, use '#if' instead of '#ifdef'.
+Some of our `#define`s are needed in the public API header as well, so
+we will go ahead and include that here so that we can share the definitions.
 */
-
-#ifndef SLANG_COMPILER
-#	define SLANG_COMPILER
+#include "../../slang.h"
 
 /*
-Compiler defines, see http://sourceforge.net/p/predef/wiki/Compilers/
-NOTE that SLANG_VC holds the compiler version - not just 1 or 0
-*/
-#	if defined(_MSC_VER)
-#		if _MSC_VER >= 1900
-#			define SLANG_VC 14
-#		elif _MSC_VER >= 1800
-#			define SLANG_VC 12
-#		elif _MSC_VER >= 1700
-#			define SLANG_VC 11
-#		elif _MSC_VER >= 1600
-#			define SLANG_VC 10
-#		elif _MSC_VER >= 1500
-#			define SLANG_VC 9
-#		else
-#			error "Unknown VC version"
-#		endif
-#	elif defined(__clang__)
-#		define SLANG_CLANG 1
-#	elif defined(__SNC__)
-#		define SLANG_SNC 1
-#	elif defined(__ghs__)
-#		define SLANG_GHS 1
-#	elif defined(__GNUC__) // note: __clang__, __SNC__, or __ghs__ imply __GNUC__
-#		define SLANG_GCC 1
-#	else
-#		error "Unknown compiler"
-#	endif
-
-// Zero unset
-#	ifndef SLANG_VC
-#		define SLANG_VC 0
-#	endif
-#	ifndef SLANG_CLANG
-#		define SLANG_CLANG 0
-#	endif
-#	ifndef SLANG_SNC
-#		define SLANG_SNC 0
-#	endif
-#	ifndef SLANG_GHS
-#		define SLANG_GHS 0
-#	endif
-#	ifndef SLANG_GCC
-#		define SLANG_GCC 0
-#	endif
-
-#endif // SLANG_COMPILER
-
-#ifndef SLANG_PLATFORM
-#	define SLANG_PLATFORM
-
-/**
-Operating system defines, see http://sourceforge.net/p/predef/wiki/OperatingSystems/
+The following preprocessor identifiers specify compiler, OS, and architecture.
+All definitions have a value of 1 or 0, use '#if' instead of '#ifdef'.
 */
-#	if defined(WINAPI_FAMILY) && WINAPI_FAMILY == WINAPI_PARTITION_APP
-#		define SLANG_WINRT 1 // Windows Runtime, either on Windows RT or Windows 8
-#	elif defined(XBOXONE)
-#		define SLANG_XBOXONE 1
-#	elif defined(_WIN64) // note: XBOXONE implies _WIN64
-#		define SLANG_WIN64 1
-#	elif defined(_M_PPC)
-#		define SLANG_X360 1
-#	elif defined(_WIN32) // note: _M_PPC implies _WIN32
-#		define SLANG_WIN32 1
-#	elif defined(__ANDROID__)
-#		define SLANG_ANDROID 1
-#	elif defined(__linux__) || defined(__CYGWIN__) // note: __ANDROID__ implies __linux__
-#		define SLANG_LINUX 1
-#	elif defined(__APPLE__) && (defined(__arm__) || defined(__arm64__))
-#		define SLANG_IOS 1
-#	elif defined(__APPLE__)
-#		define SLANG_OSX 1
-#	elif defined(__CELLOS_LV2__)
-#		define SLANG_PS3 1
-#	elif defined(__ORBIS__)
-#		define SLANG_PS4 1
-#	elif defined(__SNC__) && defined(__arm__)
-#		define SLANG_PSP2 1
-#	elif defined(__ghs__)
-#		define SLANG_WIIU 1
-#	else
-#		error "Unknown operating system"
-#	endif
-
-// zero unset
-#	ifndef SLANG_WINRT
-#		define SLANG_WINRT 0
-#	endif
-#	ifndef SLANG_XBOXONE
-#		define SLANG_XBOXONE 0
-#	endif
-#	ifndef SLANG_WIN64
-#		define SLANG_WIN64 0
-#	endif
-#	ifndef SLANG_X360
-#		define SLANG_X360 0
-#	endif
-#	ifndef SLANG_WIN32
-#		define SLANG_WIN32 0
-#	endif
-#	ifndef SLANG_ANDROID
-#		define SLANG_ANDROID 0
-#	endif
-#	ifndef SLANG_LINUX
-#		define SLANG_LINUX 0
-#	endif
-#	ifndef SLANG_IOS
-#		define SLANG_IOS 0
-#	endif
-#	ifndef SLANG_OSX
-#		define SLANG_OSX 0
-#	endif
-#	ifndef SLANG_PS3
-#		define SLANG_PS3 0
-#	endif
-#	ifndef SLANG_PS4
-#		define SLANG_PS4 0
-#	endif
-#	ifndef SLANG_PSP2
-#		define SLANG_PSP2 0
-#	endif
-#	ifndef SLANG_WIIU
-#		define SLANG_WIIU 0
-#	endif
-
-#endif // SLANG_PLATFORM
 
 #ifndef SLANG_PROCESSOR
 #	define SLANG_PROCESSOR
@@ -217,15 +92,6 @@ define anything not defined through the command line to 0
 /**
 family shortcuts
 */
-// compiler
-#define SLANG_GCC_FAMILY (SLANG_CLANG || SLANG_SNC || SLANG_GHS || SLANG_GCC)
-
-// os
-#define SLANG_WINDOWS_FAMILY (SLANG_WINRT || SLANG_WIN32 || SLANG_WIN64)
-#define SLANG_MICROSOFT_FAMILY (SLANG_XBOXONE || SLANG_X360 || SLANG_WINDOWS_FAMILY)
-#define SLANG_LINUX_FAMILY (SLANG_LINUX || SLANG_ANDROID)
-#define SLANG_APPLE_FAMILY (SLANG_IOS || SLANG_OSX)                  // equivalent to #if __APPLE__
-#define SLANG_UNIX_FAMILY (SLANG_LINUX_FAMILY || SLANG_APPLE_FAMILY) // shortcut for unix/posix platforms
 // architecture
 #define SLANG_INTEL_FAMILY (SLANG_X64 || SLANG_X86)					// Intel x86 family
 #define SLANG_ARM_FAMILY (SLANG_ARM || SLANG_A64)
@@ -233,9 +99,9 @@ family shortcuts
 
 #define SLANG_P64_FAMILY (SLANG_X64 || SLANG_A64) // shortcut for 64-bit architectures
 
-// Use for getting the amount of members of a standard C array. 
+// Use for getting the amount of members of a standard C array.
 #define SLANG_COUNT_OF(x) (sizeof(x)/sizeof(x[0]))
-/// SLANG_INLINE exists to have a way to inline consistent with SLANG_ALWAYS_INLINE  
+/// SLANG_INLINE exists to have a way to inline consistent with SLANG_ALWAYS_INLINE
 #define SLANG_INLINE inline
 
 // Other defines
@@ -253,7 +119,7 @@ family shortcuts
 General defines
 */
 
-// GCC Specific 
+// GCC Specific
 #if SLANG_GCC_FAMILY
 #	define SLANG_NO_INLINE __attribute__((noinline))
 
@@ -264,7 +130,7 @@ General defines
 
 #	if !SLANG_SNC && !SLANG_GHS
 #		define SLANG_OFFSET_OF(X, Y) __builtin_offsetof(X, Y)
-#	endif 
+#	endif
 
 //#	if !SLANG_LINUX // Workaround; Fedora Core 3 do not agree with force inline
 #	define SLANG_FORCE_INLINE inline __attribute__((always_inline))
@@ -314,7 +180,6 @@ General defines
 #	define SLANG_INT64(x) (x##i64)
 #	define SLANG_UINT64(x) (x##ui64)
 
-#	define SLANG_STDCALL __stdcall
 #	define SLANG_CALL_CONV __cdecl
 
 #endif // SLANG_MICROSOFT_FAMILY
@@ -340,7 +205,7 @@ General defines
 #ifndef SLANG_FORCE_INLINE
 #	define SLANG_FORCE_INLINE inline
 #endif
-#ifndef SLANG_NO_INLINE 
+#ifndef SLANG_NO_INLINE
 #	define SLANG_NO_INLINE
 #endif
 #ifndef SLANG_NO_ALIAS
@@ -364,15 +229,12 @@ General defines
 #	define SLANG_FUNCTION_SIG SLANG_FUNCTION_NAME
 #endif
 
-#ifndef SLANG_STDCALL
-#	define SLANG_STDCALL 
-#endif
 #ifndef SLANG_CALL_CONV
 #	define SLANG_CALL_CONV
 #endif
 
 //! casting the null ptr takes a special-case code path, which we don't want
-#define SLANG_OFFSETOF_BASE 0x100 
+#define SLANG_OFFSETOF_BASE 0x100
 #define SLANG_OFFSET_OF_RT(Class, Member)                                                                                 \
 	(reinterpret_cast<size_t>(&reinterpret_cast<Class*>(SLANG_OFFSETOF_BASE)->Member) - size_t(SLANG_OFFSETOF_BASE))
 
@@ -404,18 +266,13 @@ General defines
 #				define SLANG_HAS_ENUM_CLASS 1
 #			endif
 #			if (__GNUC__ * 100 + __GNUC_MINOR__) >= 407
-#				define SLANG_OVERRIDE override 
+#				define SLANG_OVERRIDE override
 #			endif
 #		endif
 #	endif // SLANG_GCC_FAMILY
 
 // Visual Studio
 
-// Macro for declaring if a method is no throw. Should be set before the return parameter. 
-#if SLANG_WINDOWS_FAMILY && !defined(SLANG_DISABLE_EXCEPTIONS)
-#	define SLANG_NO_THROW __declspec(nothrow) 
-#endif
-
 #	if SLANG_VC
 // C4481: nonstandard extension used: override specifier 'override'
 #		if _MSC_VER < 1700
@@ -434,7 +291,7 @@ General defines
 #	if SLANG_CLANG
 #	endif // SLANG_CLANG
 
-// Set non set  
+// Set non set
 
 #ifndef SLANG_NO_THROW
 #	define SLANG_NO_THROW
@@ -449,8 +306,6 @@ General defines
 #	define SLANG_HAS_MOVE_SEMANTICS 0
 #endif
 
-#define SLANG_MCALL SLANG_STDCALL
-
 #include <new>		// For placement new
 
 #endif // __cplusplus
diff --git a/source/core/slang-string.h b/source/core/slang-string.h
index 894776ca6..a67597360 100644
--- a/source/core/slang-string.h
+++ b/source/core/slang-string.h
@@ -172,7 +172,7 @@ namespace Slang
                 {
                     return i;
                 }
-            } 
+            }
             return -1;
         }
 
@@ -412,6 +412,11 @@ namespace Slang
             append(slice);
         }
 
+        String(UnownedStringSlice const& slice)
+        {
+            append(slice);
+        }
+
 		~String()
 		{
             buffer = 0;
@@ -741,6 +746,11 @@ namespace Slang
 			Append(str);
 			return *this;
 		}
+		StringBuilder & operator << (UnownedStringSlice const& str)
+		{
+			append(str);
+			return *this;
+		}
 		StringBuilder & operator << (const _EndLine)
 		{
 			Append('\n');
diff --git a/source/slang/compiler.cpp b/source/slang/compiler.cpp
index 66df01db0..9c1823f91 100644
--- a/source/slang/compiler.cpp
+++ b/source/slang/compiler.cpp
@@ -52,7 +52,7 @@
 
 // Includes to allow us to control console
 // output when writing assembly dumps.
-#include <fcntl.h> 
+#include <fcntl.h>
 #ifdef _WIN32
 #include <io.h>
 #else
@@ -97,6 +97,28 @@ namespace Slang
         }
     }
 
+    ComPtr<ISlangBlob> CompileResult::getBlob()
+    {
+        if(!blob)
+        {
+            switch(format)
+            {
+            case ResultFormat::None:
+            default:
+                break;
+
+            case ResultFormat::Text:
+                blob = createStringBlob(outputString);
+                break;
+
+            case ResultFormat::Binary:
+                blob = createRawBlob(outputBinary.Buffer(), outputBinary.Count());
+                break;
+            }
+        }
+        return blob;
+    }
+
     // EntryPointRequest
 
     TranslationUnitRequest* EntryPointRequest::getTranslationUnit()
@@ -149,6 +171,7 @@ namespace Slang
                     }
                 }
                 codeBuilder << "\"\n";
+
                 codeBuilder << sourceFile->content << "\n";
             }
 
@@ -886,7 +909,7 @@ String dissassembleDXILUsingDXC(
     void emitEntryPoints(
         TargetRequest*          /*targetReq*/)
     {
-        
+
     }
 
     void generateOutputForTarget(
diff --git a/source/slang/compiler.h b/source/slang/compiler.h
index 3a5f888fe..acbad9a7e 100644
--- a/source/slang/compiler.h
+++ b/source/slang/compiler.h
@@ -2,6 +2,7 @@
 #define RASTER_SHADER_COMPILER_H
 
 #include "../core/basic.h"
+#include "../core/slang-com-ptr.h"
 
 #include "diagnostics.h"
 #include "name.h"
@@ -99,9 +100,13 @@ namespace Slang
 
         void append(CompileResult const& result);
 
+        ComPtr<ISlangBlob> getBlob();
+
         ResultFormat format = ResultFormat::None;
         String outputString;
         List<uint8_t> outputBinary;
+
+        ComPtr<ISlangBlob> blob;
     };
 
     // Describes an entry point that we've been requested to compile
@@ -281,6 +286,14 @@ namespace Slang
 
     class Session;
 
+    /// Create a blob that will retain a string for its lifetime.
+    ///
+    ComPtr<ISlangBlob> createStringBlob(String const& string);
+
+    /// Create a blob that will retain (a copy of) raw data.
+    ///
+    ComPtr<ISlangBlob> createRawBlob(void const* data, size_t size);
+
     class CompileRequest : public RefObject
     {
     public:
@@ -348,6 +361,9 @@ namespace Slang
         DiagnosticSink mSink;
         String mDiagnosticOutput;
 
+        /// A blob holding the diagnostic output
+        ComPtr<ISlangBlob> diagnosticOutputBlob;
+
         // Files that compilation depended on
         List<String> mDependencyFilePaths;
 
@@ -368,6 +384,21 @@ namespace Slang
         // The resulting specialized IR module for each entry point request
         List<RefPtr<IRModule>> compiledModules;
 
+        /// File system implementation to use when loading files from disk.
+        ///
+        /// If this member is `null`, a default implementation that tries
+        /// to use the native OS filesystem will be used instead.
+        ///
+        ComPtr<ISlangFileSystem> fileSystem;
+
+        /// Load a file into memory using the configured file system.
+        ///
+        /// @param path The path to attempt to load from
+        /// @param outBlob A destination pointer to receive the loaded blob
+        /// @returns A `SlangResult` to indicate success or failure.
+        ///
+        SlangResult loadFile(String const& path, ISlangBlob** outBlob);
+
         CompileRequest(Session* session);
 
         ~CompileRequest();
@@ -394,6 +425,11 @@ namespace Slang
             int             translationUnitIndex,
             SourceFile*     sourceFile);
 
+        void addTranslationUnitSourceBlob(
+            int             translationUnitIndex,
+            String const&   path,
+            ISlangBlob*     sourceBlob);
+
         void addTranslationUnitSourceString(
             int             translationUnitIndex,
             String const&   path,
@@ -415,7 +451,7 @@ namespace Slang
         RefPtr<ModuleDecl> loadModule(
             Name*               name,
             String const&       path,
-            String const&       source,
+            ISlangBlob*         sourceBlob,
             SourceLoc const& loc);
 
         void loadParsedModule(
diff --git a/source/slang/preprocessor.cpp b/source/slang/preprocessor.cpp
index 301264632..3ec44fc28 100644
--- a/source/slang/preprocessor.cpp
+++ b/source/slang/preprocessor.cpp
@@ -1581,7 +1581,7 @@ static void HandleIncludeDirective(PreprocessorDirectiveContext* context)
     auto expandedDirectiveLoc = context->preprocessor->translationUnit->compileRequest->getSourceManager()->expandSourceLoc(directiveLoc);
     String pathIncludedFrom = expandedDirectiveLoc.getSpellingPath();
     String foundPath;
-    String foundSource;
+    ComPtr<ISlangBlob> foundSourceBlob;
 
     IncludeHandler* includeHandler = context->preprocessor->includeHandler;
     if (!includeHandler)
@@ -1590,7 +1590,7 @@ static void HandleIncludeDirective(PreprocessorDirectiveContext* context)
         GetSink(context)->diagnose(pathToken.loc, Diagnostics::noIncludeHandlerSpecified);
         return;
     }
-    auto includeResult = includeHandler->TryToFindIncludeFile(path, pathIncludedFrom, &foundPath, &foundSource);
+    auto includeResult = includeHandler->TryToFindIncludeFile(path, pathIncludedFrom, &foundPath, foundSourceBlob.writeRef());
 
     switch (includeResult)
     {
@@ -1611,7 +1611,7 @@ static void HandleIncludeDirective(PreprocessorDirectiveContext* context)
     // Push the new file onto our stack of input streams
     // TODO(tfoley): check if we have made our include stack too deep
 
-    SourceFile* sourceFile = context->preprocessor->getCompileRequest()->getSourceManager()->allocateSourceFile(foundPath, foundSource);
+    SourceFile* sourceFile = context->preprocessor->getCompileRequest()->getSourceManager()->allocateSourceFile(foundPath, foundSourceBlob);
 
     PreprocessorInputStream* inputStream = CreateInputStreamForSource(context->preprocessor, sourceFile);
     inputStream->parent = context->preprocessor->inputStream;
diff --git a/source/slang/preprocessor.h b/source/slang/preprocessor.h
index 64591ef03..0e30038bf 100644
--- a/source/slang/preprocessor.h
+++ b/source/slang/preprocessor.h
@@ -26,7 +26,7 @@ struct IncludeHandler
         String const& pathToInclude,
         String const& pathIncludedFrom,
         String* outFoundPath,
-        String* outFoundSource) = 0;
+        ISlangBlob** outFoundSourceBlob) = 0;
 };
 
 // Take a string of source code and preprocess it into a list of tokens.
diff --git a/source/slang/slang.cpp b/source/slang/slang.cpp
index 2d3e9ee77..24dff88a8 100644
--- a/source/slang/slang.cpp
+++ b/source/slang/slang.cpp
@@ -81,32 +81,29 @@ struct IncludeHandlerImpl : IncludeHandler
         String const& pathToInclude,
         String const& pathIncludedFrom,
         String* outFoundPath,
-        String* outFoundSource) override
+        ISlangBlob** outFoundSourceBlob) override
     {
         String path = Path::Combine(Path::GetDirectoryName(pathIncludedFrom), pathToInclude);
-        if (File::Exists(path))
+
+        if(SLANG_SUCCEEDED(request->loadFile(path, outFoundSourceBlob)))
         {
             *outFoundPath = path;
-            *outFoundSource = File::ReadAllText(path);
-
             request->mDependencyFilePaths.Add(path);
-
             return IncludeResult::Found;
         }
 
         for (auto & dir : request->searchDirectories)
         {
             path = Path::Combine(dir.path, pathToInclude);
-            if (File::Exists(path))
+
+            if(SLANG_SUCCEEDED(request->loadFile(path, outFoundSourceBlob)))
             {
                 *outFoundPath = path;
-                *outFoundSource = File::ReadAllText(path);
-
                 request->mDependencyFilePaths.Add(path);
-
                 return IncludeResult::Found;
             }
         }
+
         return IncludeResult::NotFound;
     }
 };
@@ -205,11 +202,168 @@ CompileRequest::~CompileRequest()
     types = decltype(types)();
 }
 
+// Allocate static const storage for the various interface IDs that the Slang API needs to expose
+static const Guid IID_ISlangUnknown = SLANG_UUID_ISlangUnknown;
+static const Guid IID_ISlangBlob    = SLANG_UUID_ISlangBlob;
+
+/** Base class for simple blobs.
+*/
+class BlobBase : public ISlangBlob
+{
+public:
+    BlobBase() {}
+    virtual ~BlobBase() {}
+
+    uint32_t referenceCount = 0;
+
+    // ISlangUnknown
+
+    SLANG_NO_THROW SlangResult SLANG_MCALL queryInterface(SlangUUID const& uuid, void** outObject) SLANG_OVERRIDE
+    {
+        if(uuid == IID_IComUnknown)
+        {
+            *(ISlangUnknown**)outObject = this;
+            addRef();
+            return SLANG_OK;
+        }
+        else if(uuid == IID_ISlangBlob)
+        {
+            *(ISlangBlob**)outObject = this;
+            addRef();
+            return SLANG_OK;
+        }
+
+        return SLANG_FAIL;
+    }
+
+    SLANG_NO_THROW uint32_t SLANG_MCALL addRef() SLANG_OVERRIDE
+    {
+        referenceCount++;
+        return 0;
+    }
+
+    SLANG_NO_THROW uint32_t SLANG_MCALL release() SLANG_OVERRIDE
+    {
+        if(--referenceCount == 0)
+        {
+            delete this;
+        }
+        return 0;
+    }
+
+};
+
+/** A blob that uses a `String` for its storage.
+*/
+class StringBlob : public BlobBase
+{
+public:
+    String string;
+
+    explicit StringBlob(String const& string)
+        : string(string)
+    {}
+
+    // ISlangBlob
+
+    SLANG_NO_THROW void const* SLANG_MCALL getBufferPointer() SLANG_OVERRIDE
+    {
+        return string.Buffer();
+    }
+
+    SLANG_NO_THROW size_t SLANG_MCALL getBufferSize() SLANG_OVERRIDE
+    {
+        return string.Length();
+    }
+};
+
+ComPtr<ISlangBlob> createStringBlob(String const& string)
+{
+    return ComPtr<ISlangBlob>(new StringBlob(string));
+}
+
+/** A blob that manages some raw data that it owns.
+*/
+class RawBlob : public BlobBase
+{
+public:
+    void* data;
+    size_t size;
+
+    ~RawBlob()
+    {
+        free(data);
+    }
+
+    // ISlangBlob
+
+    SLANG_NO_THROW void const* SLANG_MCALL getBufferPointer() SLANG_OVERRIDE
+    {
+        return data;
+    }
+
+    SLANG_NO_THROW size_t SLANG_MCALL getBufferSize() SLANG_OVERRIDE
+    {
+        return size;
+    }
+};
+
+
+ComPtr<ISlangBlob> createRawBlob(void const* inData, size_t size)
+{
+    void* dataCopy = malloc(size);
+    memcpy(dataCopy, inData, size);
+
+    RawBlob* rawBlob = new RawBlob();
+    rawBlob->data = dataCopy;
+    rawBlob->size = size;
+
+    return ComPtr<ISlangBlob>(rawBlob);
+}
+
+
+SlangResult CompileRequest::loadFile(String const& path, ISlangBlob** outBlob)
+{
+    // If there is a used-defined filesystem, then use that to load files.
+    //
+    if(fileSystem)
+    {
+        return fileSystem->loadFile(path.Buffer(), outBlob);
+    }
+
+    // Otherwise, fall back to a default implementation that uses the `core`
+    // libraries facilities for talking to the OS filesystem.
+    //
+    // TODO: we might want to conditionally compile these in, so that
+    // a user could create a build of Slang that doesn't include any OS
+    // filesystem calls.
+    //
+
+    if (!File::Exists(path))
+    {
+        return SLANG_FAIL;
+    }
+
+    try
+    {
+        String sourceString = File::ReadAllText(path);
+        ComPtr<ISlangBlob> sourceBlob = createStringBlob(sourceString);
+        *outBlob = sourceBlob.detach();
+
+        return SLANG_OK;
+    }
+    catch(...)
+    {
+    }
+    return SLANG_FAIL;
+
+}
+
 
 RefPtr<Expr> CompileRequest::parseTypeString(TranslationUnitRequest * translationUnit, String typeStr, RefPtr<Scope> scope)
 {
     Slang::SourceFile srcFile;
-    srcFile.content = typeStr;
+    srcFile.content = UnownedStringSlice(typeStr.begin(), typeStr.end());
     DiagnosticSink sink;
     sink.sourceManager = sourceManager;
     auto tokens = preprocessSource(
@@ -484,6 +638,16 @@ void CompileRequest::addTranslationUnitSourceFile(
     translationUnits[translationUnitIndex]->sourceFiles.Add(sourceFile);
 }
 
+void CompileRequest::addTranslationUnitSourceBlob(
+    int             translationUnitIndex,
+    String const&   path,
+    ISlangBlob*     sourceBlob)
+{
+    RefPtr<SourceFile> sourceFile = getSourceManager()->allocateSourceFile(path, sourceBlob);
+
+    addTranslationUnitSourceFile(translationUnitIndex, sourceFile);
+}
+
 void CompileRequest::addTranslationUnitSourceString(
     int             translationUnitIndex,
     String const&   path,
@@ -498,12 +662,17 @@ void CompileRequest::addTranslationUnitSourceFile(
     int             translationUnitIndex,
     String const&   path)
 {
-    String source;
-    try
-    {
-        source = File::ReadAllText(path);
-    }
-    catch (...)
+    // TODO: We need to consider whether a relative `path` should cause
+    // us to look things up using the registered search paths.
+    //
+    // This behavior wouldn't make sense for command-line invocations
+    // of `slangc`, but at least one API user wondered by the search
+    // paths were not taken into account by this function.
+    //
+
+    ComPtr<ISlangBlob> sourceBlob;
+    SlangResult result = loadFile(path, sourceBlob.writeRef());
+    if(SLANG_FAILED(result))
     {
         // Emit a diagnostic!
         mSink.diagnose(
@@ -513,10 +682,10 @@ void CompileRequest::addTranslationUnitSourceFile(
         return;
     }
 
-    addTranslationUnitSourceString(
+    addTranslationUnitSourceBlob(
         translationUnitIndex,
         path,
-        source);
+        sourceBlob);
 
     mDependencyFilePaths.Add(path);
 }
@@ -590,7 +759,7 @@ void CompileRequest::loadParsedModule(
 RefPtr<ModuleDecl> CompileRequest::loadModule(
     Name*               name,
     String const&       path,
-    String const&       source,
+    ISlangBlob*         sourceBlob,
     SourceLoc const&    srcLoc)
 {
     RefPtr<TranslationUnitRequest> translationUnit = new TranslationUnitRequest();
@@ -603,7 +772,7 @@ RefPtr<ModuleDecl> CompileRequest::loadModule(
     // TODO: decide which options, if any, should be inherited.
     translationUnit->compileFlags = 0;
 
-    RefPtr<SourceFile> sourceFile = getSourceManager()->allocateSourceFile(path, source);
+    RefPtr<SourceFile> sourceFile = getSourceManager()->allocateSourceFile(path, sourceBlob);
 
     translationUnit->sourceFiles.Add(sourceFile);
 
@@ -687,8 +856,8 @@ RefPtr<ModuleDecl> CompileRequest::findOrImportModule(
     String pathIncludedFrom = expandedLoc.getSpellingPath();
 
     String foundPath;
-    String foundSource;
-    IncludeResult includeResult = includeHandler.TryToFindIncludeFile(fileName, pathIncludedFrom, &foundPath, &foundSource);
+    ComPtr<ISlangBlob> foundSourceBlob;
+    IncludeResult includeResult = includeHandler.TryToFindIncludeFile(fileName, pathIncludedFrom, &foundPath, foundSourceBlob.writeRef());
     switch( includeResult )
     {
     case IncludeResult::NotFound:
@@ -715,7 +884,7 @@ RefPtr<ModuleDecl> CompileRequest::findOrImportModule(
     return loadModule(
         name,
         foundPath,
-        foundSource,
+        foundSourceBlob,
         loc);
 }
 
@@ -832,7 +1001,7 @@ Session::~Session()
     constExprRate = nullptr;
 
     destroyTypeCheckingCache();
-    
+
     builtinTypes = decltype(builtinTypes)();
     // destroy modules next
     loadedModuleCode = decltype(loadedModuleCode)();
@@ -895,6 +1064,16 @@ SLANG_API void spDestroyCompileRequest(
     delete req;
 }
 
+SLANG_API void spSetFileSystem(
+    SlangCompileRequest*    request,
+    ISlangFileSystem*       fileSystem)
+{
+    if(!request) return;
+    auto req = REQ(request);
+    req->fileSystem = fileSystem;
+}
+
+
 SLANG_API void spSetCompileFlags(
     SlangCompileRequest*    request,
     SlangCompileFlags       flags)
@@ -1021,6 +1200,25 @@ SLANG_API char const* spGetDiagnosticOutput(
     return req->mDiagnosticOutput.begin();
 }
 
+SLANG_API SlangResult spGetDiagnosticOutputBlob(
+        SlangCompileRequest*    request,
+        ISlangBlob**            outBlob)
+{
+    if(!request) return SLANG_ERROR_INVALID_PARAMETER;
+    if(!outBlob) return SLANG_ERROR_INVALID_PARAMETER;
+
+    auto req = REQ(request);
+
+    if(!req->diagnosticOutputBlob)
+    {
+        req->diagnosticOutputBlob = Slang::createStringBlob(req->mDiagnosticOutput);
+    }
+
+    Slang::ComPtr<ISlangBlob> resultBlob = req->diagnosticOutputBlob;
+    *outBlob = resultBlob.detach();
+    return SLANG_OK;
+}
+
 // New-fangled compilation API
 
 SLANG_API int spAddTranslationUnit(
@@ -1063,16 +1261,31 @@ SLANG_API void spAddTranslationUnitSourceFile(
         path);
 }
 
-// Add a source string to the given translation unit
 SLANG_API void spAddTranslationUnitSourceString(
     SlangCompileRequest*    request,
     int                     translationUnitIndex,
     char const*             path,
     char const*             source)
 {
+    if(!source) return;
+    spAddTranslationUnitSourceStringSpan(
+        request,
+        translationUnitIndex,
+        path,
+        source,
+        source + strlen(source));
+}
+
+SLANG_API void spAddTranslationUnitSourceStringSpan(
+    SlangCompileRequest*    request,
+    int                     translationUnitIndex,
+    char const*             path,
+    char const*             sourceBegin,
+    char const*             sourceEnd)
+{
     if(!request) return;
     auto req = REQ(request);
-    if(!source) return;
+    if(!sourceBegin) return;
     if(translationUnitIndex < 0) return;
     if(Slang::UInt(translationUnitIndex) >= req->translationUnits.Count()) return;
 
@@ -1081,10 +1294,34 @@ SLANG_API void spAddTranslationUnitSourceString(
     req->addTranslationUnitSourceString(
         translationUnitIndex,
         path,
-        source);
+        Slang::UnownedStringSlice(sourceBegin, sourceEnd));
+}
 
+SLANG_API void spAddTranslationUnitSourceBlob(
+    SlangCompileRequest*    request,
+    int                     translationUnitIndex,
+    char const*             path,
+    ISlangBlob*             sourceBlob)
+{
+    if(!request) return;
+    auto req = REQ(request);
+    if(!sourceBlob) return;
+    if(translationUnitIndex < 0) return;
+    if(Slang::UInt(translationUnitIndex) >= req->translationUnits.Count()) return;
+
+    if(!path) path = "";
+
+    req->addTranslationUnitSourceBlob(
+        translationUnitIndex,
+        path,
+        sourceBlob);
 }
 
+
+
+
+
+
 SLANG_API SlangProfileID spFindProfile(
     SlangSession*,
     char const*     name)
@@ -1267,6 +1504,35 @@ SLANG_API void const* spGetEntryPointCode(
     return data;
 }
 
+SLANG_API SlangResult spGetEntryPointCodeBlob(
+        SlangCompileRequest*    request,
+        int                     entryPointIndex,
+        int                     targetIndex,
+        ISlangBlob**            outBlob)
+{
+    if(!request) return SLANG_ERROR_INVALID_PARAMETER;
+    if(!outBlob) return SLANG_ERROR_INVALID_PARAMETER;
+
+    auto req = REQ(request);
+
+    int targetCount = (int) req->targets.Count();
+    if((targetIndex < 0) || (targetIndex >= targetCount))
+    {
+        return SLANG_ERROR_INVALID_PARAMETER;
+    }
+    auto targetReq = req->targets[targetIndex];
+
+    int entryPointCount = (int) req->entryPoints.Count();
+    if((entryPointIndex < 0) || (entryPointIndex >= entryPointCount))
+    {
+        return SLANG_ERROR_INVALID_PARAMETER;
+    }
+    Slang::CompileResult& result = targetReq->entryPointResults[entryPointIndex];
+
+    *outBlob = result.getBlob().detach();
+    return SLANG_OK;
+}
+
 SLANG_API char const* spGetEntryPointSource(
     SlangCompileRequest*    request,
     int                     entryPointIndex)
diff --git a/source/slang/source-loc.cpp b/source/slang/source-loc.cpp
index 80556ac37..3042c3bd4 100644
--- a/source/slang/source-loc.cpp
+++ b/source/slang/source-loc.cpp
@@ -1,6 +1,8 @@
 // source-loc.cpp
 #include "source-loc.h"
 
+#include "compiler.h"
+
 namespace Slang {
 
 String ExpandedSourceLoc::getPath() const
@@ -70,15 +72,18 @@ SourceRange SourceManager::allocateSourceRange(UInt size)
 
 SourceFile* SourceManager::allocateSourceFile(
     String const&   path,
-    String const&   content)
+    ISlangBlob*     contentBlob)
 {
-    UInt size = content.Length();
+    char const* contentBegin = (char const*) contentBlob->getBufferPointer();
+    UInt contentSize = contentBlob->getBufferSize();
+    char const* contentEnd = contentBegin + contentSize;
 
-    SourceRange sourceRange = allocateSourceRange(size);
+    SourceRange sourceRange = allocateSourceRange(contentSize);
 
     SourceFile* sourceFile = new SourceFile();
     sourceFile->path = path;
-    sourceFile->content = content;
+    sourceFile->contentBlob = contentBlob;
+    sourceFile->content = UnownedStringSlice(contentBegin, contentEnd);
     sourceFile->sourceRange = sourceRange;
 
     Entry entry;
@@ -91,6 +96,14 @@ SourceFile* SourceManager::allocateSourceFile(
     return sourceFile;
 }
 
+SourceFile* SourceManager::allocateSourceFile(
+    String const&   path,
+    String const&   content)
+{
+    ComPtr<ISlangBlob> contentBlob = createStringBlob(content);
+    return allocateSourceFile(path, contentBlob);
+}
+
 SourceLoc SourceManager::allocateSourceFileForLineDirective(
     SourceLoc const&    directiveLoc,
     String const&       path,
diff --git a/source/slang/source-loc.h b/source/slang/source-loc.h
index 83ad87633..88ec6b62a 100644
--- a/source/slang/source-loc.h
+++ b/source/slang/source-loc.h
@@ -3,6 +3,9 @@
 #define SLANG_SOURCE_LOC_H_INCLUDED
 
 #include "../core/basic.h"
+#include "../core/slang-com-ptr.h"
+
+#include "../../slang.h"
 
 namespace Slang {
 
@@ -73,8 +76,11 @@ public:
     // The logical file path to report for locations inside this span.
     String path;
 
-    // The actual contents of the file.
-    String content;
+    /// A blob that owns the storage for the file contents
+    ComPtr<ISlangBlob> contentBlob;
+
+    /// The actual contents of the file.
+    UnownedStringSlice content;
 
     // The range of source locations that the span covers
     SourceRange sourceRange;
@@ -134,6 +140,10 @@ struct SourceManager
 
     SourceFile* allocateSourceFile(
         String const&   path,
+        ISlangBlob*     content);
+
+    SourceFile* allocateSourceFile(
+        String const&   path,
         String const&   content);
 
     SourceLoc allocateSourceFileForLineDirective(