Add support for on-demand AST deserialization (#7482)

Note that this change does not actually *enable* on-demand deserialization of ASTs, because doing so is incompatible with the current compiler architecture where we have both an `ASTBuilder` and a `SharedASTBuilder`, and there are important invariants about how all AST nodes related to the core module must be created before those of any module using the core module.

Instead, this change simply adds the *infrastructure* for on-demand deserialization, and ensures that those code paths get used at runtime, but actually "demands" all of the nodes in a given serialized AST immediately as part of the deserialization process.

Important notes about the implementation approach:

* PR #7242 ensured that all of the code accessing the direct member declarations of a `ContainerDecl` went through a small(-ish) set of accessor methods. This change takes advantage of that work by further abstracting the storage of the direct member declarations out in a type, `ContainerDeclDirectMemberDecls`, which makes it easy to add custom serialization logic for just that type.

* The `ContainerDeclDirectMemberDecls` type also stores two pointers (one a `RefPtr` and the other a plain pointer) that are only used in the case where the members of a given `ContainerDecl` are being accessed through on-demand deserialization. This can be queried using the `isUsingOnDemandDeserialization()` method but any code accessing a `ContainerDecl` through the intended public API should never need to care about that detail.

* Many of the accessor methods that were added in PR #7242 now branch on whether `isUsingOnDemandDeserialization()` is set. The normal code path is unchanged, and the implementation logic for the on-demand-deserialization case is largely held in `slang-serialize-ast.cpp`, to keep it close to the definitions of the serialized data structures themselves.

* A few types in the `slang-ast-*.h` headers have had `FIDDLE()` annotations added to them, so that they can be used to synthesize some of the serialization logic that was previously hand-written.

* The `_registerBuiltinDeclsRec()` function (which is used to scan the built-in module ASTs for the various "magic" declarations that the `SharedASTBuilder` needs to know about) was factored a bit to support the way that registration needs to behave differently in the case of loading a serialized module (if we kept using the existing recursive search, then it would force every declaration in the core module to be loaded right away). The new `_collectBuiltinDeclsThatNeedRegistrationRec()` function mirrors the overall traversal pattern to produce a flat list that gets included in the serialized AST module. Note in particular that we no longer call `registerBuiltinDecls()` from within `_readBuiltinModule()`.

* The interface of the `Module` type was slightly expanded so that there is a more complete API for accessing the declarations exported from the module. Previously they could only be queried by their mangled name, but the new API also allows the entire list to be iterated over. The `ensureLookupAcceleratorBuilt()` method factors out the logic for building those data structures for a module. Note that in the case where on-demand deserialization is being used for a module, the `findExportedDeclByMandledName()` query will use serialized data directly, rather than build the lookup accelerators as C++ data structures (this is required if we are to avoid immediately deserializing all of the (exported) declarations in the core module as soon as it is loaded).

* A few methods related to loading serialized modules (e.g., `loadSerializedModule()`) have been updated so that along with a pointer to the serialized `ModuleChunk` (which, for those who aren't aware, is a pointer directly into the serialized bytes of the module file), they receive an `ISlangBlob` that refers to the entire blob holding the serialized data (which the `ModuleChunk` is part of). Passing this pointer down allows code running under these methods to retain a reference-counted pointer to the blob to stop the memory of the serialized module from being released until deserialization has been completed.

* The data types defined in `slang-fossil.h` have been overhauled significantly:

  * The most important change that is relevant to this work is the introduction of the `Fossilized<T>` template, which is used to statically map a "live" C++ type `T` to its binary fossilized representation. The `slang-fossil.h` file provides infrastructure allowing `Fossilized<T>` to be specialized for user-defined types, and also provides the necessary mappings for the core types like strings, arrays, and dictionaries.

  * A key point is that in C++ code, one can take a value of some type `Foo`, serialize it using a `Fossil::SerialWriter`, get a pointer to that serialized data, and then directly cast it to a `Fossilized<Foo>*` and navigate the serialized data directly (without deserializing it back into a `Foo`). For that process to work, any specialization of `Fossilized<T>` must be sure to match the layout that will be produced by the `serialize()` implementation for `T`, when writing to a `Fossil::SerialWriter`.

  * Another key change in the public interface of `slang-fossil.h` is that dynamically-typed traversal of the data used to be handled just with `FossilizedValRef`, but now uses a few different types. The `Fossil::ValRef<T>` and `Fossil::AnyValRef` types are used to capture the use cases that want reference-like behavior (basically a `Fossil::ValRef<T>` can be thought of as sort of like a `T&`), while `Fossil::ValPtr<T>` and `Fossil::AnyValPtr` are used for cases that want pointer like behavior (akin to `T*`).

* Then there are related changes in `slang-serialize-fossil.*`:

  * The implementation of `Fossil::SerialReader` has been changed to use `Fossil::AnyValPtr` in most places where it formerly used `FossilizedValRef`. Using pointers (that can be null) instead of a weird kind of pseudo-reference (that could still be null) to traverse things was making the code harder to follow than it ought to be, in terms of understanding the levels of indirection in various places.

  * Some of the state that was previously in `Fossil::SerialReader` has been split into `Fossil::ReadContext`. This type allows multiple `Fossil::SerialReader`s to be created to read from the same serialized blob(s), while maintaining a persistent mapping from fossilized data pointers to live object pointers. The `ReadContext` also maintains the work list of deferred deserialization actions waiting to be performed, and only flushes that list when the last currently-open `SerialReader` is about to go out of scope.

* In order to support the split of `Fossil::SerialReader` described above (and also to clean up something that didn't quite feel right in the original serialization design) the base serialization framework in `slang-serialize.h` has been tweaked so that a `Serializer` now wraps *two* pointers instead of just one. The first pointer continues to be an implementation of `ISerializerImpl`, which handles the actual reading/writing of data, while the other pointer is an explicit "context" pointer for operations that need additional user-defined context.

* Similar to the changes made to the accessors for direct member declarations in a `ContainerDecl`, the `Module::findExportedDeclByMangledName()` method was updated to conditionally execute a different code path in the case of a module that has been loaded from serialized data.

* Some improvements have been made to the fiddle tool:

  * Most importantly, the error-handling logic around Lua script execution has been cleaned up to better match correct Lua idiom. Native functions exposed to the Lua scripts have been changed to just use `lua_call` instead of `lua_pcall`, so rather than attempt to intercept Lua errors they will just automatically propagate them.

  * All Lua-related errors are caught at the top level, and reported in a way that uses the source location of the fiddle template that was being evaluated when the error was raised. In most cases, a Lua error should be accompanied by a stack trace of the Lua evluation state. The file paths and line numbers given should be accurate, but aren't directly double-clickable in the Visual Studio output panel, because they use a different format (a good future change might be to process the Lua stack trace and rewrite it into a format that is better for our needs).

  * Fixed a subtle bug where having "raw" content (parts of the template that should neither be evaluated nor emitted into the output) that consisted of only whitespace could result in a template being translated to invalid Lua code.

* The bulk of the change is, unsurprisingly, in `slang-serialize-ast.cpp`.

  * This file has been refactored enough to look like a complete rewrite. A lot of work has been put into comments that describe the overall approach being taken, so hopefully it can be understood even by somebody who wasn't familiar with the previous code. Some of these are just plain cleanups, rather than being directly related to on-demand serialization.

  * Where possible, the code for reading and writing types that needed custom serialization has been moved so that the read/write functions are next to one another, making it easier to visually confirm that the serialized representations match on the read and write sides.

  * Where possible, the serialization logic for all types (not just the AST nodes, as was the case before) is being generated via fiddle.

  * Rather than just defining `serialize()` overloads for each of the relevant types, the code now defines `Fossilized<...>` specializations for these types as well, to enable statically-typed in-memory traversal of the serialized data. Note, however, that for the most part the `Fossilized<...>` representation types are *not* being used by the code (really only the `ASTModuleInfo` and `ContainerDeclDirectMemberDeclsInfo` types are traversed directly). This can be considered more as work to prove out the design of the `Fossil<...>` template approach, and it may or may not end up being relevant in the future.

  * The trivial bit of work to enable on-demand deserialization is in `ASTSerialReadContext::handleContainerDeclDirectMemberDecls()` where, rather than recursively reading the contained declarations, the method effectively just grabs the current cursor of the `Fossil::SerialReader` (which is pointed into the fossilized data) and stashes it into the `ContainerDeclDirectMemberDecls`, along with a `RefPtr` to the `ASTSerialReadContext` itself. Those stashed pointers are what enables the accessors on `ContaienrDeclDirectMemberDecls` to look up information on-demand.

  * The more interesting bits of the approach mostly come at the end of the file, where the accessor operations for on-demand deserialization are implemented. Once all the relevant work has been done to write the data structures, and produce `Fossilized<...>` types with the right layout, the work itself may seem almost trivial: a little bit of array iteration, and a little bit of binary-search lookup.

  * As a reminder, all of this infrastructure for on-demand deserialization is now in place and able to be invoked by the rest of the compiler, but declarations are currently all being loaded eagerly. The `SLANG_DISABLE_ON_DEMAND_AST_DESERIALIZATION` macro is being used to enable a small bit of extra logic in `ASTSerialReadContext::_cleanUpASTNode` so that the "cleanup" on a just-deserialized `ContainerDecl` includes eagerly querying its list of direct member declarations, which will cause them to be recursively deserialized.

1 files changed, 224 insertions, 173 deletions

diff --git a/source/slang/slang-serialize-fossil.cpp b/source/slang/slang-serialize-fossil.cpp
index da1516399..003f4227a 100644
--- a/source/slang/slang-serialize-fossil.cpp
+++ b/source/slang/slang-serialize-fossil.cpp
@@ -56,7 +56,7 @@ void SerialWriter::_initialize(ChunkBuilder* chunk)
     // that the last field of the header is a relative pointer
     // to the root-value chunk.
     //
-    headerChunk->writeRelativePtr<Fossil::RelativePtrOffset>(rootValueChunk);
+    headerChunk->writeRelativePtr<FossilInt>(rootValueChunk);
 
     // The root value should always be a variant, and we want to
     // set up to write into it in a reasonable way.
@@ -140,7 +140,7 @@ void SerialWriter::handleFloat64(double& value)
 void SerialWriter::handleString(String& value)
 {
     auto size = value.getLength();
-    if (_shouldEmitWithPointerIndirection(FossilizedValKind::String))
+    if (_shouldEmitPotentiallyIndirectValueWithPointerIndirection())
     {
         if (size == 0)
         {
@@ -154,14 +154,14 @@ void SerialWriter::handleString(String& value)
 
             auto ptrLayout =
                 (ContainerLayoutObj*)_reserveDestinationForWrite(FossilizedValKind::Ptr);
-            _mergeLayout(ptrLayout->baseLayout, FossilizedValKind::String);
+            _mergeLayout(ptrLayout->baseLayout, FossilizedValKind::StringObj);
 
             _commitWrite(ValInfo::relativePtrTo(existingChunk));
             return;
         }
     }
 
-    _pushPotentiallyIndirectValueScope(FossilizedValKind::String);
+    _pushPotentiallyIndirectValueScope(FossilizedValKind::StringObj);
 
     auto data = value.getBuffer();
     _writeValueRaw(ValInfo::rawData(data, size + 1, 1));
@@ -176,7 +176,7 @@ void SerialWriter::handleString(String& value)
 
 void SerialWriter::beginArray()
 {
-    _pushContainerScope(FossilizedValKind::Array);
+    _pushContainerScope(FossilizedValKind::ArrayObj);
 }
 
 void SerialWriter::endArray()
@@ -186,7 +186,7 @@ void SerialWriter::endArray()
 
 void SerialWriter::beginDictionary()
 {
-    _pushContainerScope(FossilizedValKind::Dictionary);
+    _pushContainerScope(FossilizedValKind::DictionaryObj);
 }
 
 void SerialWriter::endDictionary()
@@ -258,7 +258,7 @@ void SerialWriter::endTuple()
 
 void SerialWriter::beginOptional()
 {
-    _pushIndirectValueScope(FossilizedValKind::Optional);
+    _pushIndirectValueScope(FossilizedValKind::OptionalObj);
 }
 
 void SerialWriter::endOptional()
@@ -266,7 +266,7 @@ void SerialWriter::endOptional()
     _popIndirectValueScope();
 }
 
-void SerialWriter::handleSharedPtr(void*& value, Callback callback, void* userData)
+void SerialWriter::handleSharedPtr(void*& value, Callback callback, void* context)
 {
     // Because we are writing, we only care about the
     // pointer that is already present in `value`.
@@ -305,7 +305,7 @@ void SerialWriter::handleSharedPtr(void*& value, Callback callback, void* userDa
     fossilizedObject->ptrLayout = ptrLayout;
     fossilizedObject->liveObjectPtr = liveObjectPtr;
     fossilizedObject->callback = callback;
-    fossilizedObject->userData = userData;
+    fossilizedObject->context = context;
 
     _fossilizedObjects.add(fossilizedObject);
     _mapLiveObjectPtrToFossilizedObject.add(liveObjectPtr, fossilizedObject);
@@ -313,15 +313,15 @@ void SerialWriter::handleSharedPtr(void*& value, Callback callback, void* userDa
     _commitWrite(ValInfo::relativePtrTo(chunk));
 }
 
-void SerialWriter::handleUniquePtr(void*& value, Callback callback, void* userData)
+void SerialWriter::handleUniquePtr(void*& value, Callback callback, void* context)
 {
     // We treat all pointers as shared pointers, because there isn't really
     // an optimized representation we would want to use for the unique case.
     //
-    handleSharedPtr(value, callback, userData);
+    handleSharedPtr(value, callback, context);
 }
 
-void SerialWriter::handleDeferredObjectContents(void* valuePtr, Callback callback, void* userData)
+void SerialWriter::handleDeferredObjectContents(void* valuePtr, Callback callback, void* context)
 {
     // Because we are already deferring writing of the *entirety* of
     // an object's members as part of how `handleSharedPtr()` works,
@@ -330,7 +330,7 @@ void SerialWriter::handleDeferredObjectContents(void* valuePtr, Callback callbac
     // (In practice the `handleDeferredObjectContents()` operation is
     // more for the benefit of reading than writing).
     //
-    callback(valuePtr, userData);
+    callback(valuePtr, this, context);
 }
 
 SerialWriter::LayoutObj* SerialWriter::_createSimpleLayout(FossilizedValKind kind)
@@ -356,7 +356,7 @@ SerialWriter::LayoutObj* SerialWriter::_createSimpleLayout(FossilizedValKind kin
     case FossilizedValKind::Float64:
         return new (_arena) SimpleLayoutObj(kind, 8);
 
-    case FossilizedValKind::String:
+    case FossilizedValKind::StringObj:
         return new (_arena) SimpleLayoutObj(kind);
 
     default:
@@ -369,23 +369,19 @@ SerialWriter::LayoutObj* SerialWriter::_createLayout(FossilizedValKind kind)
 {
     switch (kind)
     {
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Optional:
-    case FossilizedValKind::Dictionary:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::OptionalObj:
+    case FossilizedValKind::DictionaryObj:
         return new (_arena) ContainerLayoutObj(kind, nullptr);
 
     case FossilizedValKind::Ptr:
-        return new (_arena) ContainerLayoutObj(
-            kind,
-            nullptr,
-            sizeof(Fossil::RelativePtrOffset),
-            sizeof(Fossil::RelativePtrOffset));
+        return new (_arena) ContainerLayoutObj(kind, nullptr, sizeof(FossilInt), sizeof(FossilInt));
 
     case FossilizedValKind::Struct:
     case FossilizedValKind::Tuple:
         return new (_arena) RecordLayoutObj(kind);
 
-    case FossilizedValKind::Variant:
+    case FossilizedValKind::VariantObj:
         // A variant is being treated like a container in this context,
         // because it wants to be able to track the layout of what it
         // ended up holding...
@@ -403,7 +399,7 @@ SerialWriter::LayoutObj* SerialWriter::_createLayout(FossilizedValKind kind)
     case FossilizedValKind::UInt64:
     case FossilizedValKind::Float32:
     case FossilizedValKind::Float64:
-    case FossilizedValKind::String:
+    case FossilizedValKind::StringObj:
         {
             if (auto found = _simpleLayouts.tryGetValue(kind))
                 return *found;
@@ -435,7 +431,7 @@ SerialWriter::LayoutObj* SerialWriter::_mergeLayout(LayoutObj*& dst, FossilizedV
     // then we want to have a unique layout object for each
     // instance.
     //
-    if (kind == FossilizedValKind::Variant)
+    if (kind == FossilizedValKind::VariantObj)
     {
         auto src = _createLayout(kind);
         return src;
@@ -462,9 +458,9 @@ void SerialWriter::_mergeLayout(LayoutObj*& dst, LayoutObj* src)
 
     switch (src->getKind())
     {
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Optional:
-    case FossilizedValKind::Dictionary:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::OptionalObj:
+    case FossilizedValKind::DictionaryObj:
     case FossilizedValKind::Ptr:
         {
             auto dstContainer = (ContainerLayoutObj*)dst;
@@ -473,10 +469,10 @@ void SerialWriter::_mergeLayout(LayoutObj*& dst, LayoutObj* src)
         }
         break;
 
-    case FossilizedValKind::String:
+    case FossilizedValKind::StringObj:
         break;
 
-    case FossilizedValKind::Variant:
+    case FossilizedValKind::VariantObj:
         // Recursive merging should not be applied to variants;
         // each variant is unique until later deduplication.
         break;
@@ -557,7 +553,7 @@ Size SerialWriter::ValInfo::getAlignment() const
     switch (kind)
     {
     case Kind::RelativePtr:
-        return sizeof(Fossil::RelativePtrOffset);
+        return sizeof(FossilInt);
 
     case Kind::ContentsOfChunk:
         return chunk->getAlignment();
@@ -598,13 +594,13 @@ void SerialWriter::_popInlineValueScope()
 
 void SerialWriter::_pushVariantScope()
 {
-    _pushPotentiallyIndirectValueScope(FossilizedValKind::Variant);
+    _pushPotentiallyIndirectValueScope(FossilizedValKind::VariantObj);
 }
 
 void SerialWriter::_popVariantScope()
 {
     SLANG_ASSERT(_state.layout);
-    SLANG_ASSERT(_state.layout->kind == FossilizedValKind::Variant);
+    SLANG_ASSERT(_state.layout->kind == FossilizedValKind::VariantObj);
     auto variantLayout = (ContainerLayoutObj*)_state.layout;
     auto valueLayout = variantLayout->baseLayout;
     SLANG_ASSERT(valueLayout);
@@ -631,7 +627,7 @@ void SerialWriter::_popVariantScope()
 
 void SerialWriter::_pushPotentiallyIndirectValueScope(FossilizedValKind kind)
 {
-    if (_shouldEmitWithPointerIndirection(kind))
+    if (_shouldEmitPotentiallyIndirectValueWithPointerIndirection())
     {
         _pushIndirectValueScope(kind);
     }
@@ -647,12 +643,10 @@ ChunkBuilder* SerialWriter::_popPotentiallyIndirectValueScope()
     // conditional to select between the functions for the
     // indirect and inline cases.
 
-    auto valueLayout = _state.layout;
     auto valueChunk = _state.chunk;
     _popState();
 
-    auto valueKind = valueLayout->getKind();
-    if (_shouldEmitWithPointerIndirection(valueKind))
+    if (_shouldEmitPotentiallyIndirectValueWithPointerIndirection())
     {
         return _writeKnownIndirectValueSharedLogic(valueChunk);
     }
@@ -729,11 +723,14 @@ void SerialWriter::_writeValueRaw(ValInfo const& val)
 
     case ValInfo::Kind::RelativePtr:
         _ensureChunkExists();
-        _state.chunk->writeRelativePtr<Fossil::RelativePtrOffset>(val.chunk);
+        _state.chunk->writeRelativePtr<FossilInt>(val.chunk);
         break;
 
     case ValInfo::Kind::ContentsOfChunk:
         {
+            if (!val.chunk)
+                return;
+
             if (!_state.chunk)
             {
                 _state.chunk = val.chunk;
@@ -751,29 +748,14 @@ void SerialWriter::_writeValueRaw(ValInfo const& val)
     }
 }
 
-bool SerialWriter::_shouldEmitWithPointerIndirection(FossilizedValKind kind)
+bool SerialWriter::_shouldEmitPotentiallyIndirectValueWithPointerIndirection()
 {
-    switch (kind)
-    {
-    default:
-        return false;
-
-    case FossilizedValKind::Optional:
-        return true;
-
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
-    case FossilizedValKind::String:
-    case FossilizedValKind::Variant:
-        break;
-    }
-
     switch (_state.layout->getKind())
     {
     default:
         return true;
 
-    case FossilizedValKind::Optional:
+    case FossilizedValKind::OptionalObj:
     case FossilizedValKind::Ptr:
         return false;
     }
@@ -794,10 +776,10 @@ SerialWriter::LayoutObj*& SerialWriter::_reserveDestinationForWrite()
         break;
 
     case FossilizedValKind::Ptr:
-    case FossilizedValKind::Optional:
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
-    case FossilizedValKind::Variant:
+    case FossilizedValKind::OptionalObj:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::DictionaryObj:
+    case FossilizedValKind::VariantObj:
         {
             auto containerLayout = (ContainerLayoutObj*)_state.layout;
             auto& elementLayout = containerLayout->baseLayout;
@@ -849,17 +831,17 @@ void SerialWriter::_commitWrite(ValInfo const& val)
         }
         break;
 
-    case FossilizedValKind::Optional:
+    case FossilizedValKind::OptionalObj:
     case FossilizedValKind::Ptr:
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
-    case FossilizedValKind::Variant:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::DictionaryObj:
+    case FossilizedValKind::VariantObj:
         {
             auto elementIndex = _state.elementCount++;
 
             switch (outerKind)
             {
-            case FossilizedValKind::Optional:
+            case FossilizedValKind::OptionalObj:
             case FossilizedValKind::Ptr:
                 if (elementIndex > 0)
                 {
@@ -911,7 +893,10 @@ void SerialWriter::_flush()
 
         _state = State(fossilizedObject->ptrLayout, fossilizedObject->chunk);
 
-        fossilizedObject->callback(&fossilizedObject->liveObjectPtr, fossilizedObject->userData);
+        fossilizedObject->callback(
+            &fossilizedObject->liveObjectPtr,
+            this,
+            fossilizedObject->context);
     }
 
     // Once we've written out all the payload data, we can start to work on
@@ -921,7 +906,7 @@ void SerialWriter::_flush()
     for (auto variantInfo : _variants)
     {
         auto layoutChunk = _getOrCreateChunkForLayout(variantInfo.layout);
-        variantInfo.chunk->addPrefixRelativePtr<Fossil::RelativePtrOffset>(layoutChunk);
+        variantInfo.chunk->addPrefixRelativePtr<FossilInt>(layoutChunk);
     }
 }
 
@@ -964,22 +949,22 @@ ChunkBuilder* SerialWriter::_getOrCreateChunkForLayout(LayoutObj* layout)
         break;
 
     case FossilizedValKind::Ptr:
-    case FossilizedValKind::Optional:
+    case FossilizedValKind::OptionalObj:
         {
             auto containerLayout = (ContainerLayoutObj*)layout;
             auto elementLayout = containerLayout->baseLayout;
             auto elementLayoutChunk = _getOrCreateChunkForLayout(elementLayout);
-            chunk->writeRelativePtr<Fossil::RelativePtrOffset>(elementLayoutChunk);
+            chunk->writeRelativePtr<FossilInt>(elementLayoutChunk);
         }
         break;
 
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::DictionaryObj:
         {
             auto containerLayout = (ContainerLayoutObj*)layout;
             auto elementLayout = containerLayout->baseLayout;
             auto elementLayoutChunk = _getOrCreateChunkForLayout(elementLayout);
-            chunk->writeRelativePtr<Fossil::RelativePtrOffset>(elementLayoutChunk);
+            chunk->writeRelativePtr<FossilInt>(elementLayoutChunk);
 
             UInt32 elementStride = 0;
             if (elementLayout)
@@ -1004,7 +989,7 @@ ChunkBuilder* SerialWriter::_getOrCreateChunkForLayout(LayoutObj* layout)
             {
                 auto& field = recordLayout->fields[i];
                 auto fieldLayoutChunk = _getOrCreateChunkForLayout(field.layout);
-                chunk->writeRelativePtr<Fossil::RelativePtrOffset>(fieldLayoutChunk);
+                chunk->writeRelativePtr<FossilInt>(fieldLayoutChunk);
 
                 auto fieldOffset = UInt32(field.offset);
                 chunk->writeData(&fieldOffset, sizeof(fieldOffset));
@@ -1043,9 +1028,9 @@ bool SerialWriter::LayoutObjKey::operator==(LayoutObjKey const& that) const
     default:
         break;
 
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
-    case FossilizedValKind::Optional:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::DictionaryObj:
+    case FossilizedValKind::OptionalObj:
     case FossilizedValKind::Ptr:
         {
             auto thisContainer = (ContainerLayoutObj*)obj;
@@ -1117,9 +1102,9 @@ void SerialWriter::LayoutObjKey::hashInto(Hasher& hasher) const
     default:
         break;
 
-    case FossilizedValKind::Array:
-    case FossilizedValKind::Dictionary:
-    case FossilizedValKind::Optional:
+    case FossilizedValKind::ArrayObj:
+    case FossilizedValKind::DictionaryObj:
+    case FossilizedValKind::OptionalObj:
     case FossilizedValKind::Ptr:
         {
             auto container = (ContainerLayoutObj*)obj;
@@ -1152,16 +1137,83 @@ void SerialWriter::LayoutObjKey::hashInto(Hasher& hasher) const
 // SerialReader
 //
 
-SerialReader::SerialReader(FossilizedValRef valRef)
+SerialReader::SerialReader(
+    ReadContext& context,
+    Fossil::AnyValPtr valPtr,
+    InitialStateType initialState)
+    : _context(context)
 {
-    _state.type = State::Type::Root;
-    _state.baseValue = valRef;
+    // We track the number of active `SerialReader`s that
+    // are working with the same `ReadContext`, and will
+    // make use of this count in the destructor below.
+    //
+    context._readerCount++;
+
+    switch (initialState)
+    {
+    case InitialStateType::Root:
+        _state.type = State::Type::Root;
+        break;
+
+    case InitialStateType::PseudoPtr:
+        _state.type = State::Type::PseudoPtr;
+        break;
+    }
+
+    _state.baseValPtr = valPtr;
     _state.elementIndex = 0;
     _state.elementCount = 1;
 }
 
 SerialReader::~SerialReader()
 {
+    // If an application is designed to perform something
+    // like on-demand deserialization, it may create
+    // additional `SerialReader`s attached to the same
+    // `ReadContext`, potentially even in the body of a
+    // callback that was invoked by an operation on another
+    // `SerialReader` further up the stack.
+    //
+    // If we were to track the deferred actions that get
+    // enqueued on a per-`SerialReader` basis, and then
+    // flush them when the given `SerialReader` is destructed,
+    // it could potentially lead to very deep call stacks.
+    //
+    // Instead, we track a single list of deferred actions
+    // on the `ReadContext`, which means that we need to
+    // figure out when to actually flush that list.
+    //
+    // What is implemented here is a "last one out shuts the door"
+    // policy. When a `SerialReader` is being destroyed, before
+    // it decrements the count on the shared `ReadContext`, it
+    // checks to see if it is the last remaining `SerialReader`,
+    // in which case it takes responsibility for flushing the deferred
+    // actions that were enqueued by *all* of the readers.
+    //
+    // Note that the ordering here is critical: we check whether
+    // we are the last reader and, if so, perform the `_flush()`
+    // operation all *before* decrementing the counter. If we
+    // were to decrement the count before invoking `_flush()`
+    // then any nested `SerialReader`s that get created by the
+    // deferred actions would (incorrectly) believe themselves
+    // to be the "last one out" and try to perform their own
+    // `flush()`, which could quickly lead to unbounded
+    // recursion.
+    //
+    if (_context._readerCount == 1)
+    {
+        _flush();
+    }
+    _context._readerCount--;
+}
+
+Fossil::AnyValPtr SerialReader::readValPtr()
+{
+    return _readValPtr();
+}
+
+void SerialReader::flush()
+{
     _flush();
 }
 
@@ -1172,94 +1224,83 @@ SerializationMode SerialReader::getMode()
 
 void SerialReader::handleBool(bool& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedBoolVal>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleInt8(int8_t& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedInt8Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleInt16(int16_t& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedInt16Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleInt32(Int32& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedInt32Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleInt64(Int64& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedInt64Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleUInt8(uint8_t& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedUInt8Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleUInt16(uint16_t& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedUInt16Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleUInt32(UInt32& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedUInt32Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleUInt64(UInt64& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedUInt64Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleFloat32(float& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedFloat32Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleFloat64(double& value)
 {
-    auto valRef = _readValRef();
-    value = as<FossilizedFloat64Val>(valRef)->getValue();
+    handleSimpleVal(value);
 }
 
 void SerialReader::handleString(String& value)
 {
-    auto valRef = _readPotentiallyIndirectValRef();
-    if (!valRef)
+    auto valPtr = _readPotentiallyIndirectValPtr();
+    if (!valPtr)
     {
         value = String();
     }
     else
     {
-        value = as<FossilizedStringObj>(valRef)->getValue();
+        value = as<FossilizedStringObj>(valPtr)->get();
     }
 }
 
 void SerialReader::beginArray()
 {
-    auto valRef = _readPotentiallyIndirectValRef();
-    auto arrayRef = as<FossilizedContainerObj>(valRef);
+    auto valPtr = _readPotentiallyIndirectValPtr();
+    auto arrayPtr = as<FossilizedArrayObjBase>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Array;
-    _state.baseValue = valRef;
+    _state.baseValPtr = arrayPtr;
     _state.elementIndex = 0;
-    _state.elementCount = getElementCount(arrayRef);
+    _state.elementCount = arrayPtr->getElementCount();
 }
 
 void SerialReader::endArray()
@@ -1269,15 +1310,15 @@ void SerialReader::endArray()
 
 void SerialReader::beginDictionary()
 {
-    auto valRef = _readPotentiallyIndirectValRef();
-    auto dictionaryRef = as<FossilizedContainerObj>(valRef);
+    auto valPtr = _readPotentiallyIndirectValPtr();
+    auto dictionaryPtr = as<FossilizedDictionaryObjBase>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Dictionary;
-    _state.baseValue = valRef;
+    _state.baseValPtr = dictionaryPtr;
     _state.elementIndex = 0;
-    _state.elementCount = getElementCount(dictionaryRef);
+    _state.elementCount = dictionaryPtr->getElementCount();
 }
 
 void SerialReader::endDictionary()
@@ -1292,15 +1333,15 @@ bool SerialReader::hasElements()
 
 void SerialReader::beginStruct()
 {
-    auto valRef = _readValRef();
-    auto recordRef = as<FossilizedRecordVal>(valRef);
+    auto valPtr = _readValPtr();
+    auto recordPtr = as<FossilizedRecordVal>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Struct;
-    _state.baseValue = valRef;
+    _state.baseValPtr = valPtr;
     _state.elementIndex = 0;
-    _state.elementCount = getFieldCount(recordRef);
+    _state.elementCount = recordPtr->getFieldCount();
 }
 
 void SerialReader::endStruct()
@@ -1310,18 +1351,20 @@ void SerialReader::endStruct()
 
 void SerialReader::beginVariant()
 {
-    auto valRef = _readPotentiallyIndirectValRef();
-    auto variantRef = as<FossilizedVariantObj>(valRef);
-
-    auto contentValRef = getVariantContent(variantRef);
-    auto contentRecordRef = as<FossilizedRecordVal>(contentValRef);
+    auto valPtr = _readPotentiallyIndirectValPtr();
+    if (auto variantPtr = as<FossilizedVariantObj>(valPtr))
+    {
+        auto contentValPtr = getVariantContentPtr(variantPtr);
+        valPtr = contentValPtr;
+    }
+    auto recordPtr = as<FossilizedRecordVal>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Struct;
-    _state.baseValue = contentValRef;
+    _state.baseValPtr = recordPtr;
     _state.elementIndex = 0;
-    _state.elementCount = getFieldCount(contentRecordRef);
+    _state.elementCount = recordPtr->getFieldCount();
 }
 
 void SerialReader::endVariant()
@@ -1339,15 +1382,15 @@ void SerialReader::handleFieldKey(char const* name, Int index)
 
 void SerialReader::beginTuple()
 {
-    auto valRef = _readValRef();
-    auto recordRef = as<FossilizedRecordVal>(valRef);
+    auto valPtr = _readValPtr();
+    auto recordPtr = as<FossilizedRecordVal>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Tuple;
-    _state.baseValue = valRef;
+    _state.baseValPtr = recordPtr;
     _state.elementIndex = 0;
-    _state.elementCount = getFieldCount(recordRef);
+    _state.elementCount = recordPtr->getFieldCount();
 }
 
 void SerialReader::endTuple()
@@ -1357,15 +1400,15 @@ void SerialReader::endTuple()
 
 void SerialReader::beginOptional()
 {
-    auto valRef = _readIndirectValRef();
-    auto optionalRef = as<FossilizedOptionalObj>(valRef);
+    auto valPtr = _readIndirectValPtr();
+    auto optionalPtr = as<FossilizedOptionalObjBase>(valPtr);
 
     _pushState();
 
     _state.type = State::Type::Optional;
-    _state.baseValue = valRef;
+    _state.baseValPtr = optionalPtr;
     _state.elementIndex = 0;
-    _state.elementCount = Count(hasValue(optionalRef));
+    _state.elementCount = Count(optionalPtr->hasValue());
 }
 
 void SerialReader::endOptional()
@@ -1373,14 +1416,21 @@ void SerialReader::endOptional()
     _popState();
 }
 
-void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userData)
+void SerialReader::handleSharedPtr(void*& value, Callback callback, void* context)
 {
-    // The fossilized value at our cursor must be a pointer,
-    // and we can resolve what it is pointing to easily enough.
-    //
-    auto valRef = _readValRef();
-    auto ptrRef = as<FossilizedPtrVal>(valRef);
-    auto targetValRef = getPtrTarget(ptrRef);
+    Fossil::AnyValPtr targetValPtr;
+
+    if (_state.type == State::Type::PseudoPtr)
+    {
+        _state.type = State::Type::Root;
+        targetValPtr = _readValPtr();
+    }
+    else
+    {
+        auto valPtr = _readValPtr();
+        auto ptrPtr = as<FossilizedPtr<void>>(valPtr);
+        targetValPtr = ptrPtr->getTargetValPtr();
+    }
 
     // The logic here largely mirrors what appears in
     // `SerialWriter::handleSharedPtr`.
@@ -1388,7 +1438,7 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     // We first check for an explicitly written null pointer.
     // If we find one our work is very easy.
     //
-    if (!targetValRef)
+    if (!targetValPtr)
     {
         value = nullptr;
         return;
@@ -1397,7 +1447,7 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     // Now we need to check if we've previously read in
     // a reference to the same object.
     //
-    if (auto found = _mapFossilizedObjectPtrToObjectInfo.tryGetValue(targetValRef.getData()))
+    if (auto found = _context.mapFossilizedObjectPtrToObjectInfo.tryGetValue(targetValPtr.get()))
     {
         auto objectInfo = *found;
 
@@ -1440,9 +1490,9 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     // object index that has not yet been read at all.
     //
     auto objectInfo = RefPtr(new ObjectInfo());
-    _mapFossilizedObjectPtrToObjectInfo.add(targetValRef.getData(), objectInfo);
+    _context.mapFossilizedObjectPtrToObjectInfo.add(targetValPtr.get(), objectInfo);
 
-    objectInfo->fossilizedObjectRef = targetValRef;
+    objectInfo->fossilizedObjectPtr = targetValPtr;
 
     // We cannot return from this function until we have
     // stored a pointer into `value`, to represent the
@@ -1473,7 +1523,7 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     //
     _pushState();
     _state.type = State::Type::Object;
-    _state.baseValue = objectInfo->fossilizedObjectRef;
+    _state.baseValPtr = objectInfo->fossilizedObjectPtr;
     _state.elementIndex = 0;
     _state.elementCount = 1;
 
@@ -1491,7 +1541,7 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     // that objects and stores a pointer to it into the output
     // parameter.
     //
-    callback(&objectInfo->resurrectedObjectPtr, userData);
+    callback(&objectInfo->resurrectedObjectPtr, this, context);
 
     _popState();
 
@@ -1500,15 +1550,15 @@ void SerialReader::handleSharedPtr(void*& value, Callback callback, void* userDa
     value = objectInfo->resurrectedObjectPtr;
 }
 
-void SerialReader::handleUniquePtr(void*& value, Callback callback, void* userData)
+void SerialReader::handleUniquePtr(void*& value, Callback callback, void* context)
 {
     // We treat all pointers as shared pointers, because there isn't really
     // an optimized representation we would want to use for the unique case.
     //
-    handleSharedPtr(value, callback, userData);
+    handleSharedPtr(value, callback, context);
 }
 
-void SerialReader::handleDeferredObjectContents(void* valuePtr, Callback callback, void* userData)
+void SerialReader::handleDeferredObjectContents(void* valuePtr, Callback callback, void* context)
 {
     // Unlike the case in `SerialWriter::handleDeferredObjectContents()`,
     // we very much *do* want to delay invoking the callback until later.
@@ -1528,9 +1578,9 @@ void SerialReader::handleDeferredObjectContents(void* valuePtr, Callback callbac
     deferredAction.savedState = _state;
     deferredAction.resurrectedObjectPtr = valuePtr;
     deferredAction.callback = callback;
-    deferredAction.userData = userData;
+    deferredAction.context = context;
 
-    _deferredActions.add(deferredAction);
+    _context._deferredActions.add(deferredAction);
 }
 
 void SerialReader::_flush()
@@ -1538,7 +1588,7 @@ void SerialReader::_flush()
     // We need to flush any actions that were deferred
     // and are still pending.
     //
-    while (_deferredActions.getCount() != 0)
+    while (_context._deferredActions.getCount() != 0)
     {
         // TODO: For simplicity we are using the `_deferredActions`
         // array as a stack (LIFO), but it would be good to
@@ -1546,15 +1596,15 @@ void SerialReader::_flush()
         // large the array would need to grow for a FIFO vs. LIFO,
         // and pick the better option.
         //
-        auto deferredAction = _deferredActions.getLast();
-        _deferredActions.removeLast();
+        auto deferredAction = _context._deferredActions.getLast();
+        _context._deferredActions.removeLast();
 
         _state = deferredAction.savedState;
-        deferredAction.callback(deferredAction.resurrectedObjectPtr, deferredAction.userData);
+        deferredAction.callback(deferredAction.resurrectedObjectPtr, this, deferredAction.context);
     }
 }
 
-FossilizedValRef SerialReader::_readValRef()
+Fossil::AnyValPtr SerialReader::_readValPtr()
 {
     switch (_state.type)
     {
@@ -1563,7 +1613,7 @@ FossilizedValRef SerialReader::_readValRef()
         SLANG_ASSERT(_state.elementCount == 1);
         SLANG_ASSERT(_state.elementIndex == 0);
         _state.elementIndex++;
-        return _state.baseValue;
+        return _state.baseValPtr;
 
     case State::Type::Struct:
     case State::Type::Tuple:
@@ -1571,8 +1621,8 @@ FossilizedValRef SerialReader::_readValRef()
             SLANG_ASSERT(_state.elementIndex < _state.elementCount);
             auto index = _state.elementIndex++;
 
-            auto recordRef = as<FossilizedRecordVal>(_state.baseValue);
-            return getField(recordRef, index);
+            auto recordPtr = as<FossilizedRecordVal>(_state.baseValPtr);
+            return getAddress(recordPtr->getField(index));
         }
 
     case State::Type::Optional:
@@ -1580,8 +1630,8 @@ FossilizedValRef SerialReader::_readValRef()
             SLANG_ASSERT(_state.elementCount == 1);
             SLANG_ASSERT(_state.elementIndex == 0);
 
-            auto optionalRef = as<FossilizedOptionalObj>(_state.baseValue);
-            return getValue(optionalRef);
+            auto optionalPtr = as<FossilizedOptionalObjBase>(_state.baseValPtr);
+            return getAddress(optionalPtr->getValue());
         }
 
     case State::Type::Array:
@@ -1590,8 +1640,8 @@ FossilizedValRef SerialReader::_readValRef()
             SLANG_ASSERT(_state.elementIndex < _state.elementCount);
             auto index = _state.elementIndex++;
 
-            auto containerRef = as<FossilizedContainerObj>(_state.baseValue);
-            return getElement(containerRef, index);
+            auto containerPtr = as<FossilizedContainerObjBase>(_state.baseValPtr);
+            return Fossil::ValPtr(containerPtr->getElement(index));
         }
 
     default:
@@ -1600,24 +1650,25 @@ FossilizedValRef SerialReader::_readValRef()
     }
 }
 
-FossilizedValRef SerialReader::_readIndirectValRef()
+Fossil::AnyValPtr SerialReader::_readIndirectValPtr()
 {
-    auto ptrValRef = _readValRef();
-    auto ptrRef = as<FossilizedPtrVal>(ptrValRef);
+    auto baseValPtr = _readValPtr();
+    auto basePtrPtr = as<FossilizedPtr<void>>(baseValPtr);
 
-    auto valRef = getPtrTarget(ptrRef);
-    return valRef;
+    auto targetValPtr = basePtrPtr->getTargetValPtr();
+    return targetValPtr;
 }
 
 
-FossilizedValRef SerialReader::_readPotentiallyIndirectValRef()
+Fossil::AnyValPtr SerialReader::_readPotentiallyIndirectValPtr()
 {
-    auto valRef = _readValRef();
-    if (auto ptrRef = as<FossilizedPtrVal>(valRef))
+    auto baseValPtr = _readValPtr();
+    if (auto basePtrPtr = as<FossilizedPtr<void>>(baseValPtr))
     {
-        return getPtrTarget(ptrRef);
+        auto targetValRef = basePtrPtr->getTargetValRef();
+        return Fossil::ValPtr(targetValRef);
     }
-    return valRef;
+    return baseValPtr;
 }
 
 void SerialReader::_pushState()