A new approach to AST serialization (#6854)

* A new approach to AST serialization

This change completely overhauls the way that AST nodes are being serialized, and the offline source-code generation steps that enable that serialization.
In practice, this ends up being a complete overhaul of the way that *modules* are being serialized (not just the AST part), although things like the serialization format for the Slang IR and for source locations are not affected.

The rest of this commit message is broken down in to sections, in an attempt to help guide anybody looking at the code in how to make sense of all the changes.

The Old C++ Extractor
---------------------

AST serialization used to be driven by information scraped using the `slang-cpp-extractor` tool, which did an ad hoc parse of the C++ declarations of the AST node types and then generated a set of "X macros" that could be for macro-based code generation within the rest of the compiler.
While the existing approach was functional, it wasn't easy to understand or maintain, and it has been getting in the way of forward progress on other features we'd like to work on in the language and compiler.

This change removes the `slang-cpp-extractor` tool entirely.

Marking Up the AST Declarations
-------------------------------

The most notable change that contributors to the compiler may notice is the large number of invocations of a macro `FIDDLE()` on the declarations of the AST node types.

The basic idea is that only declarations (namespaces, types, fields) that are preceded by `FIDDLE()` are visible to the code generator tool.
So if somebody is working with the AST and wondering why a new node type isn't working, or why a field they added isn't being serialized correctly, it is probably because they need to add `FIDDLE()` in front of it.

Generating the Boilerplate Code
-------------------------------

The file `slang-ast-boilerplate.cpp` provides a good example of how the information extracted from the marked-up AST declarations gets used.
In that file, the `FIDDLE TEMPLATE` construct is used to generate type information for each of the AST node types.

Similar logic is used in `slang-ast-forward-declarations.h` to generate the declaration of the `ASTNodeType` enumeration, and forward-declare all the AST node classes.
For many parts of the code, simply including that file replaces the need for the old `slang-generated-*.h` files.

Replacing Visitors and Related Logic
------------------------------------

The old visitor types for the AST used the macros that were generated by `slang-cpp-extractor`, so something new was needed to replace them.
The same goes for the `SLANG_AST_NODE_VIRTUAL_CALL` macros.

The core of the solution implemented here is in `slang-ast-dispatch.h`.
Given a "dispatchable" AST node type (say, `Expr`), a call like:

```
ASTNodeDispatcher<Expr,R>(expr, [&](auto e) { return doSomething(e); })
```

is an expression of type `R`, which does the equivalent of something like:

```
switch(expr->getTag())
{
case ASTNodeType::VarExpr: return doSomething(static_cast<VarExpr*>(expr));
// ...
}
```

The `SLANG_AST_NODE_VIRTUAL_CALL` macro is now implemented in terms of `ASTNodeDispatcher`.

The implementation of the visitor types is more involved.
The code in this change retains some of the macro names from the original version, just to try and make the parallels more clear.
The visitor types are all implemented on top of the `ASTNodeDispatcher` approach, and use `FIDDLE TEMPLATE` to generate all the boilerplate `visit*()` method declarations.

Refactoring of `Linkage` Module Loading
---------------------------------------

Needing to revisit all the places where modules get deserialized made it clear that there is a lot of complexity and apparent duplication in the core routines on the `Linkage` that get used for loading modules.

This change tries to clean up some of that logic, but it is worth noting that there are two legacy features that get in the way of making things as clean as they should be:

* The `LoadedModuleDictionary` type that gets passed around a lot exists entirely to handle the corner case where somebody uses the Slang API to perform a compilation with multiple `TranslationUnitRequest`s in the same `FrontEndCompileRequest`, and one of the translation units `import`s the module defined by another of the translation units.

* There are a lot of special-case behaviors and routines entirely there to support the `ModuleLibrary` feature, although that feature should be considered deprecated (or at least subject to getting entirely re-designed down the line).

The basic idea of the cleanup is that all of the (non-deprecated) ways load a module from a serialized binary, or compile one from source should now bottleneck through `loadModuleImpl`, which then bifurcates into `loadSourceModuleImpl` for the compilation case and `loadBinaryModuleImpl` for the deserialization case.

High-Level Serialization Approach
---------------------------------

The old serialization logic used the [RIFF](https://en.wikipedia.org/wiki/Resource_Interchange_File_Format) format to encode the high-level structure of things, and this change retains that usage (and actually doubles down on the RIFF usage).

The old serialization system relied on the idea that for any given type `Foo` that wants to support serialization, there should be something like a `SerialFooData` type in C++, that can represent the state of a `Foo`, and then the actual serialization applied to that `SerialFooData`. This means that in most cases there are four pieces of code written:

* During serialization:
  * Copying the data of a `Foo` in memory over to a `SerialFooData` in memory
  * Writing the state of a `SerialFooData` into the serialized data stream

* During deserialization:
  * Reading the state of a `SerialFooData` from a serialized data stream
  * Copying the data of the `SerialFooData` in memory over to a `Foo`

The new logic gets rid of the intermediate `SerialFooData`.

In the serialization direction, we take a `Foo` and write it to the `RIFFContainer` directly, or using some other utilities layered on top of it.

In the deserialization direction, we have additional flexibility. Given a `RIFFContainer::Chunk*` that represents a serialized `Foo`, we often navigate through the in-memory representation of the RIFF data to get to the parts of the serialized value that we actually want/need, without needing to deserialize the entire `Foo`.

To support this kind of operation, this change introduces a few helper types like `ContainerChunkRef` an `ModuleChunkRef`, that are little more than typed wrappers around a `RIFFContainer::Chunk*`.

The Module "Container" Part
---------------------------

A serialized `Module` is encoded as a RIFF chunk, using logic in `slang-serialize-container.cpp` - both before and after this change.
This change reorganizes a lot of the code in that file, to account for the way that eliminating the intermediate `SerialContainerData` type streamlines the overall task of writing out the parts of the module.

In the deserialization logic... there isn't really much to do in `slang-serialize-container.cpp`. Most of the logic in `slang.cpp` and `slang-module-library.cpp` that pertains to deserializing modules uses the `ModuleChunkRef`-based approach, and simply extracts the pieces of the serialized module that it needs.

The Actual Serialization of the AST
-----------------------------------

The actual AST serialization logic is in `slang-serialize-ast.cpp`.
The basic approach in both the writing and reading directions is:

* Use the `FIDDLE TEMPLATE` system to generate a set of functions, one for each AST node type, that recursively invoke the read/write logic on each field of that node (after recursively invoking the case for its direct superclass)

* Use the `ASTNodeDispatcher` system to dispatch out to those functions whene reading or writing anything derived from `NodeBase`

* For now, handle all types *not* derived from `NodeBase` by hand.

There's a lot of room for improvement around that last item: it should be just as easy to generate the serialization and deserialization logic for other types that don't inherit from `NodeBase`, but the current change tries to err on the side of making the logic as explicit and simplistic as possible, rather than trying to get too clever too soon.

The actual serialization *format* used for the AST is almost comically simplistic: the code uses hierarchical RIFF chunks to emulate a JSON-like structure. This is a very wasteful representation (e.g., a `bool` or a null pointer each take up *8 bytes*), but the goal for now is to start with the simplest thing that could possibly work, and only add more cleverness once we are sure it won't get in the way of important future improvements (like lazy/on-demand deserialization or IR and AST, to improve compiler startup times).

The files `slang-serialize.{h,cpp}` have been co-opted to define a new pair of types `Encoder` and `Decoder` that are used for a more-or-less stream-oriented way or reading or writing RIFF chunks for the JSON-like structure.

Almost everything related to the actual AST serialization could do with a cleanup pass, and some time spent on picking good/better names for everything.

Smaller Stuff
-------------

* Cleaned up a lot of code that was using bare `ASTNodeType` or the extractor's `ReflectClassInfo` type to consistently use `SyntaxClass`.

* Fixed an apparent bug in how the destination-driven code genarator was handling `TryExpr`s

* Fixed an apparent bug in how the GLSL legalization pass was handling translation of certain `SV_*` semantics.

* format code

* fixup: template errors caught by non-VS compilers

* format code

* fixup: more template errors

* fixup: more stuff VS didn't catch

* fixup: it's amazing VS doesn't catch these...

* fixup: yet more template stuff VS ignores

* fixup: more VS template nonsense

* fixup: unreachable return macro usage

* fixup: more unreacable returns

* fixup: unused parameter

* fixup: strict aliasing

* fixup: allow missing entry point list chunk

* fixup: wasm build script

* fixup: AST changes since this PR was created

---------

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>
Co-authored-by: Yong He <yonghe@outlook.com>

1 files changed, 0 insertions, 1178 deletions

diff --git a/source/slang/slang-serialize.cpp b/source/slang/slang-serialize.cpp
index 2a1a92302..a1c555a9b 100644
--- a/source/slang/slang-serialize.cpp
+++ b/source/slang/slang-serialize.cpp
@@ -8,1182 +8,4 @@
 namespace Slang
 {
 
-const SerialClass* SerialClasses::add(const SerialClass* cls)
-{
-    List<const SerialClass*>& classes = m_classesByTypeKind[Index(cls->typeKind)];
-
-    if (cls->subType >= classes.getCount())
-    {
-        classes.setCount(cls->subType + 1);
-    }
-    else
-    {
-        if (classes[cls->subType])
-        {
-            SLANG_ASSERT(!"Type is already set");
-            return nullptr;
-        }
-    }
-
-    SerialClass* copy = _createSerialClass(cls);
-    classes[cls->subType] = copy;
-
-    return copy;
-}
-
-const SerialClass* SerialClasses::add(
-    SerialTypeKind kind,
-    SerialSubType subType,
-    const SerialField* fields,
-    Index fieldsCount,
-    const SerialClass* superCls)
-{
-    SerialClass cls;
-    cls.typeKind = kind;
-    cls.subType = subType;
-
-    cls.fields = fields;
-    cls.fieldsCount = fieldsCount;
-
-    // If the superCls is set it must be owned
-    SLANG_ASSERT(superCls == nullptr || isOwned(superCls));
-
-    cls.super = superCls;
-
-    // Set to invalid values for now
-    cls.alignment = 0;
-    cls.size = 0;
-    cls.flags = 0;
-
-    return add(&cls);
-}
-
-const SerialClass* SerialClasses::addUnserialized(SerialTypeKind kind, SerialSubType subType)
-{
-    List<const SerialClass*>& classes = m_classesByTypeKind[Index(kind)];
-
-    if (subType >= classes.getCount())
-    {
-        classes.setCount(subType + 1);
-    }
-    else
-    {
-        if (classes[subType])
-        {
-            SLANG_ASSERT(!"Type is already set");
-            return nullptr;
-        }
-    }
-
-    SerialClass* dst = m_arena.allocate<SerialClass>();
-
-    dst->typeKind = kind;
-    dst->subType = subType;
-
-    dst->size = 0;
-    dst->alignment = 0;
-
-    dst->fields = nullptr;
-    dst->fieldsCount = 0;
-    dst->flags = SerialClassFlag::DontSerialize;
-    dst->super = nullptr;
-
-    classes[subType] = dst;
-    return dst;
-}
-
-bool SerialClasses::isOwned(const SerialClass* cls) const
-{
-    const List<const SerialClass*>& classes = m_classesByTypeKind[Index(cls->typeKind)];
-    return cls->subType < classes.getCount() && classes[cls->subType] == cls;
-}
-
-SerialClass* SerialClasses::_createSerialClass(const SerialClass* cls)
-{
-    uint32_t maxAlignment = 1;
-    uint32_t offset = 0;
-
-    if (cls->super)
-    {
-        SLANG_ASSERT(isOwned(cls->super));
-
-        maxAlignment = cls->super->alignment;
-        offset = cls->super->size;
-    }
-
-    // Can't be 0
-    SLANG_ASSERT(maxAlignment != 0);
-    // Must be a power of 2
-    SLANG_ASSERT((maxAlignment & (maxAlignment - 1)) == 0);
-
-    // Check it is correctly aligned
-    SLANG_ASSERT((offset & (maxAlignment - 1)) == 0);
-
-    SerialField* dstFields = m_arena.allocateArray<SerialField>(cls->fieldsCount);
-
-    // Okay, go through fields setting their offset
-    const SerialField* srcFields = cls->fields;
-    for (Index j = 0; j < cls->fieldsCount; j++)
-    {
-        const SerialField& srcField = srcFields[j];
-        SerialField& dstField = dstFields[j];
-
-        // Copy the field
-        dstField = srcField;
-
-        uint32_t alignment = srcField.type->serialAlignment;
-        // Make sure the offset is aligned for the field requirement
-        offset = (offset + alignment - 1) & ~(alignment - 1);
-
-        // Save the field offset
-        dstField.serialOffset = uint32_t(offset);
-
-        // Move past the field
-        offset += uint32_t(srcField.type->serialSizeInBytes);
-
-        // Calc the maximum alignment
-        maxAlignment = (alignment > maxAlignment) ? alignment : maxAlignment;
-    }
-
-    // Align with maximum alignment
-    offset = (offset + maxAlignment - 1) & ~(maxAlignment - 1);
-
-    SerialClass* dst = m_arena.allocate<SerialClass>();
-    *dst = *cls;
-
-    dst->alignment = uint8_t(maxAlignment);
-    dst->size = uint32_t(offset);
-
-    dst->fields = dstFields;
-
-    return dst;
-}
-
-bool SerialClasses::isOk() const
-{
-    StringSlicePool pool(StringSlicePool::Style::Default);
-
-    for (const auto& classes : m_classesByTypeKind)
-    {
-        for (const SerialClass* cls : classes)
-        {
-            // It is possible potentially to have gaps
-            if (cls == nullptr)
-            {
-                continue;
-            }
-
-            if (cls->super && cls->super->typeKind != cls->typeKind)
-            {
-                // If has a super type, must be the same typeKind
-                return false;
-            }
-
-            // Make sure the fields are uniquely named
-
-            pool.clear();
-
-            {
-                const SerialClass* curCls = cls;
-
-                do
-                {
-                    for (Index i = 0; i < curCls->fieldsCount; ++i)
-                    {
-                        const SerialField& field = curCls->fields[i];
-
-                        StringSlicePool::Handle handle;
-                        if (pool.findOrAdd(UnownedStringSlice(field.name), handle))
-                        {
-                            return false;
-                        }
-                    }
-
-                    // Add the fields of the parent
-                    curCls = curCls->super;
-                } while (curCls);
-            }
-        }
-    }
-
-    return true;
-}
-
-
-SerialClasses::SerialClasses()
-    : m_arena(2097152)
-{
-}
-
-// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SerialWriter  !!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
-SerialWriter::SerialWriter(SerialClasses* classes, SerialFilter* filter, Flags flags)
-    : m_arena(2097152), m_classes(classes), m_filter(filter), m_flags(flags)
-{
-    // 0 is always the null pointer
-    m_entries.add(nullptr);
-    m_ptrMap.add(nullptr, 0);
-}
-
-struct SkipFunctionBodyRAII
-{
-    FunctionDeclBase* funcDecl = nullptr;
-    Stmt* oldBody = nullptr;
-    SkipFunctionBodyRAII(SerialWriter::Flags flags, const SerialClass* serialCls, const void* ptr)
-    {
-        if ((flags & SerialWriter::Flag::SkipFunctionBody) == 0)
-            return;
-
-        if (serialCls->typeKind != SerialTypeKind::NodeBase)
-            return;
-        auto cls = serialCls;
-        while (cls)
-        {
-            auto astNodeType = (ASTNodeType)cls->subType;
-            if (astNodeType == ASTNodeType::FunctionDeclBase)
-            {
-                funcDecl = (FunctionDeclBase*)ptr;
-                break;
-            }
-            cls = cls->super;
-        }
-        if (funcDecl)
-        {
-            oldBody = funcDecl->body;
-            // We always need to include body of unsafeForceInlineEarly functions
-            // since they will need to be available at IR lowering time of the
-            // user module for pre-linking inling.
-            if (!isUnsafeForceInlineFunc(funcDecl))
-            {
-                funcDecl->body = nullptr;
-            }
-        }
-    }
-    ~SkipFunctionBodyRAII()
-    {
-        if (funcDecl)
-        {
-            funcDecl->body = oldBody;
-        }
-    }
-};
-
-SerialIndex SerialWriter::writeObject(const SerialClass* serialCls, const void* ptr)
-{
-    if (serialCls->flags & SerialClassFlag::DontSerialize)
-    {
-        return SerialIndex(0);
-    }
-
-    if (serialCls->typeKind == SerialTypeKind::NodeBase &&
-        ReflectClassInfo::isSubClassOf(serialCls->subType, Val::kReflectClassInfo))
-    {
-        return writeValObject((Val*)ptr);
-    }
-
-    // If we are skipping function bodies, set the body field to nullptr, and
-    // restore it after serialization.
-    SkipFunctionBodyRAII clearFunctionBodyRAII(m_flags, serialCls, ptr);
-
-    // This pointer cannot be in the map
-    SLANG_ASSERT(m_ptrMap.tryGetValue(ptr) == nullptr);
-
-    typedef SerialInfo::ObjectEntry ObjectEntry;
-
-    ObjectEntry* nodeEntry = (ObjectEntry*)m_arena.allocateAligned(
-        sizeof(ObjectEntry) + serialCls->size,
-        SerialInfo::MAX_ALIGNMENT);
-
-    nodeEntry->typeKind = serialCls->typeKind;
-    nodeEntry->subType = serialCls->subType;
-    nodeEntry->_pad0 = 0;
-
-    nodeEntry->info = SerialInfo::makeEntryInfo(serialCls->alignment);
-
-    // We add before adding fields, so if the fields point to this, the entry will be set
-    auto index = _add(ptr, nodeEntry);
-
-    // Point to start of payload
-    uint8_t* serialPayload = (uint8_t*)(nodeEntry + 1);
-
-    if (m_flags & Flag::ZeroInitialize)
-    {
-        ::memset(serialPayload, 0, serialCls->size);
-    }
-
-    while (serialCls)
-    {
-        for (Index i = 0; i < serialCls->fieldsCount; ++i)
-        {
-            auto field = serialCls->fields[i];
-
-            // Work out the offsets
-            auto srcField = ((const uint8_t*)ptr) + field.nativeOffset;
-            auto dstField = serialPayload + field.serialOffset;
-
-            field.type->toSerialFunc(this, srcField, dstField);
-        }
-
-        // Get the super class
-        serialCls = serialCls->super;
-    }
-
-    return index;
-}
-
-SerialIndex SerialWriter::writeObject(const NodeBase* node)
-{
-    const SerialClass* serialClass =
-        m_classes->getSerialClass(SerialTypeKind::NodeBase, SerialSubType(node->astNodeType));
-    return writeObject(serialClass, (const void*)node);
-}
-
-SerialIndex SerialWriter::writeValObject(const Val* node)
-{
-    typedef SerialInfo::ValEntry ValEntry;
-
-    size_t size = node->getOperandCount() * sizeof(SerialInfo::SerialValOperand);
-    ValEntry* nodeEntry =
-        (ValEntry*)m_arena.allocateAligned(sizeof(ValEntry) + size, SerialInfo::MAX_ALIGNMENT);
-
-    nodeEntry->typeKind = SerialTypeKind::NodeBase;
-    nodeEntry->subType = (SerialSubType)node->astNodeType;
-    nodeEntry->operandCount = (uint32_t)node->getOperandCount();
-    nodeEntry->info = SerialInfo::makeEntryInfo(SerialInfo::MAX_ALIGNMENT);
-
-    // We add before adding fields, so if the fields point to this, the entry will be set
-    auto index = _add(node, nodeEntry);
-
-    ShortList<SerialIndex, 4> serializedOperands;
-
-    for (Index i = 0; i < node->getOperandCount(); i++)
-    {
-        auto operand = node->m_operands[i];
-        switch (operand.kind)
-        {
-        case ValNodeOperandKind::ConstantValue:
-            serializedOperands.add((SerialIndex)0);
-            break;
-        case ValNodeOperandKind::ValNode:
-        case ValNodeOperandKind::ASTNode:
-            serializedOperands.add(addPointer(operand.values.nodeOperand));
-            break;
-        }
-    }
-
-    SLANG_ASSERT(serializedOperands.getCount() == node->getOperandCount());
-
-    auto serialOperands = (SerialInfo::SerialValOperand*)(nodeEntry + 1);
-    for (Index i = 0; i < node->getOperandCount(); i++)
-    {
-        auto serialOperand = serialOperands + i;
-        auto operand = node->m_operands[i];
-        serialOperand->type = (int)operand.kind;
-        switch (operand.kind)
-        {
-        case ValNodeOperandKind::ConstantValue:
-            serialOperand->payload = operand.values.intOperand;
-            break;
-        case ValNodeOperandKind::ValNode:
-            serialOperand->payload = (uint64_t)serializedOperands[i];
-            break;
-        case ValNodeOperandKind::ASTNode:
-            serialOperand->payload = (uint64_t)serializedOperands[i];
-            break;
-        }
-    }
-    return index;
-}
-
-SerialIndex SerialWriter::writeObject(const RefObject* obj)
-{
-    const SerialRefObject* serialObj = as<const SerialRefObject>(obj);
-    if (!serialObj)
-    {
-        SLANG_ASSERT(!"Unhandled type");
-        return SerialIndex(0);
-    }
-
-    const ReflectClassInfo* classInfo = serialObj->getClassInfo();
-    SLANG_ASSERT(classInfo);
-
-    const SerialClass* serialClass =
-        m_classes->getSerialClass(SerialTypeKind::RefObject, SerialSubType(classInfo->m_classId));
-    return writeObject(serialClass, (const void*)obj);
-}
-
-void SerialWriter::setPointerIndex(const NodeBase* ptr, SerialIndex index)
-{
-    m_ptrMap.add(ptr, Index(index));
-}
-
-void SerialWriter::setPointerIndex(const RefObject* ptr, SerialIndex index)
-{
-    m_ptrMap.add(ptr, Index(index));
-}
-
-SerialIndex SerialWriter::addPointer(const NodeBase* node)
-{
-    // Null is always 0
-    if (node == nullptr)
-    {
-        return SerialIndex(0);
-    }
-    // Look up in the map
-    Index* indexPtr = m_ptrMap.tryGetValue(node);
-    if (indexPtr)
-    {
-        return SerialIndex(*indexPtr);
-    }
-
-    if (m_filter)
-    {
-        return m_filter->writePointer(this, node);
-    }
-    else
-    {
-        return writeObject(node);
-    }
-}
-
-SerialIndex SerialWriter::addPointer(const RefObject* obj)
-{
-    // Null is always 0
-    if (obj == nullptr)
-    {
-        return SerialIndex(0);
-    }
-    // Look up in the map
-    Index* indexPtr = m_ptrMap.tryGetValue(obj);
-    if (indexPtr)
-    {
-        return SerialIndex(*indexPtr);
-    }
-
-    // TODO(JS):
-    // Arguably the lookup for these types should be done the same way as arbitrary RefObject types
-    // and have a enum for them, such we can use a switch instead of all this casting
-
-    if (auto stringRep = dynamicCast<StringRepresentation>(obj))
-    {
-        SerialIndex index = addString(StringRepresentation::asSlice(stringRep));
-        m_ptrMap.add(obj, Index(index));
-        return index;
-    }
-    else if (auto name = dynamicCast<const Name>(obj))
-    {
-        return addName(name);
-    }
-
-    if (m_filter)
-    {
-        return m_filter->writePointer(this, obj);
-    }
-    else
-    {
-        return writeObject(obj);
-    }
-}
-
-SerialIndex SerialWriter::_addStringSlice(
-    SerialTypeKind typeKind,
-    SliceMap& sliceMap,
-    const UnownedStringSlice& slice)
-{
-    typedef ByteEncodeUtil Util;
-    typedef SerialInfo::StringEntry StringEntry;
-
-    if (slice.getLength() == 0)
-    {
-        return SerialIndex(0);
-    }
-
-    Index* indexPtr = sliceMap.tryGetValue(slice);
-    if (indexPtr)
-    {
-        return SerialIndex(*indexPtr);
-    }
-
-    // Okay we need to add the string
-
-    uint8_t encodeBuf[Util::kMaxLiteEncodeUInt32];
-    const int encodeCount = Util::encodeLiteUInt32(uint32_t(slice.getLength()), encodeBuf);
-
-    StringEntry* entry = (StringEntry*)m_arena.allocateUnaligned(
-        SLANG_OFFSET_OF(StringEntry, sizeAndChars) + encodeCount + slice.getLength());
-    entry->info = SerialInfo::EntryInfo::Alignment1;
-    entry->typeKind = typeKind;
-
-    uint8_t* dst = (uint8_t*)(entry->sizeAndChars);
-    for (int i = 0; i < encodeCount; ++i)
-    {
-        dst[i] = encodeBuf[i];
-    }
-
-    memcpy(dst + encodeCount, slice.begin(), slice.getLength());
-
-    // Make a key that will stay in scope -> it's actually just stored in the arena.
-    // NOTE! without terminating 0
-    UnownedStringSlice keySlice(((const char*)dst) + encodeCount, slice.getLength());
-
-    Index newIndex = m_entries.getCount();
-    sliceMap.add(keySlice, newIndex);
-
-    m_entries.add(entry);
-    return SerialIndex(newIndex);
-}
-
-SerialIndex SerialWriter::addString(const String& in)
-{
-    return addPointer(in.getStringRepresentation());
-}
-
-SerialIndex SerialWriter::addName(const Name* name)
-{
-    if (name == nullptr)
-    {
-        return SerialIndex(0);
-    }
-
-    // Look it up
-    Index* indexPtr = m_ptrMap.tryGetValue(name);
-    if (indexPtr)
-    {
-        return SerialIndex(*indexPtr);
-    }
-
-    SerialIndex index = addString(name->text);
-    m_ptrMap.add(name, Index(index));
-    return index;
-}
-
-SerialIndex SerialWriter::addSerialArray(
-    size_t elementSize,
-    size_t alignment,
-    const void* elements,
-    Index elementCount)
-{
-    typedef SerialInfo::ArrayEntry Entry;
-
-    if (elementCount == 0)
-    {
-        return SerialIndex(0);
-    }
-
-    SLANG_ASSERT(alignment >= 1 && alignment <= SerialInfo::MAX_ALIGNMENT);
-
-    // We must at a minimum have the alignment for the array prefix info
-    alignment = (alignment < SLANG_ALIGN_OF(Entry)) ? SLANG_ALIGN_OF(Entry) : alignment;
-
-    size_t payloadSize = elementCount * elementSize;
-
-    Entry* entry = (Entry*)m_arena.allocateAligned(sizeof(Entry) + payloadSize, alignment);
-
-    entry->typeKind = SerialTypeKind::Array;
-    entry->info = SerialInfo::makeEntryInfo(int(alignment));
-    entry->elementSize = uint16_t(elementSize);
-    entry->elementCount = uint32_t(elementCount);
-
-    memcpy(entry + 1, elements, payloadSize);
-
-    m_entries.add(entry);
-    return SerialIndex(m_entries.getCount() - 1);
-}
-
-static const uint8_t s_fixBuffer[SerialInfo::MAX_ALIGNMENT]{
-    0,
-};
-
-SlangResult SerialWriter::write(Stream* stream)
-{
-    const Int entriesCount = m_entries.getCount();
-
-    // Add a sentinal so we don't need special handling for
-    SerialInfo::Entry sentinal;
-    sentinal.typeKind = SerialTypeKind::String;
-    sentinal.info = SerialInfo::EntryInfo::Alignment1;
-
-    m_entries.add(&sentinal);
-    m_entries.removeLast();
-
-    SerialInfo::Entry** entries = m_entries.getBuffer();
-    // Note strictly required in our impl of List. But by writing this and
-    // knowing that removeLast cannot release memory, means the sentinal must be at the last
-    // position.
-    entries[entriesCount] = &sentinal;
-
-    {
-        size_t offset = 0;
-
-        SerialInfo::Entry* entry = entries[1];
-        // We start on 1, because 0 is nullptr and not used for anything
-        for (Index i = 1; i < entriesCount; ++i)
-        {
-            SerialInfo::Entry* next = entries[i + 1];
-
-            // Before writing we need to store the next alignment
-
-            const size_t nextAlignment = SerialInfo::getAlignment(next->info);
-            const size_t alignment = SerialInfo::getAlignment(entry->info);
-            SLANG_UNUSED(alignment);
-
-            entry->info = SerialInfo::combineWithNext(entry->info, next->info);
-
-            // Check we are aligned correctly
-            SLANG_ASSERT((offset & (alignment - 1)) == 0);
-
-            // When we write, we need to make sure it take into account the next alignment
-            const size_t entrySize = entry->calcSize(m_classes);
-
-            // Work out the fix for next alignment
-            size_t nextOffset = offset + entrySize;
-            nextOffset = (nextOffset + nextAlignment - 1) & ~(nextAlignment - 1);
-
-            size_t alignmentFixSize = nextOffset - (offset + entrySize);
-
-            // The fix must be less than max alignment. We require it to be less because we aligned
-            // each Entry to MAX_ALIGNMENT, and so < MAX_ALIGNMENT is the most extra bytes we can
-            // write
-            SLANG_ASSERT(alignmentFixSize < SerialInfo::MAX_ALIGNMENT);
-
-            SLANG_RETURN_ON_FAIL(stream->write(entry, entrySize));
-            // If we needed to fix so that subsequent alignment is right, write out extra bytes here
-            if (alignmentFixSize)
-            {
-                SLANG_RETURN_ON_FAIL(stream->write(s_fixBuffer, alignmentFixSize));
-            }
-
-            // Onto next
-            offset = nextOffset;
-            entry = next;
-        }
-    }
-
-    return SLANG_OK;
-}
-
-SlangResult SerialWriter::writeIntoContainer(FourCC fourCc, RiffContainer* container)
-{
-    typedef RiffContainer::Chunk Chunk;
-    typedef RiffContainer::ScopeChunk ScopeChunk;
-
-    {
-        ScopeChunk scopeData(container, Chunk::Kind::Data, fourCc);
-
-        {
-            // Sentinel so we don't need special handling for end of list
-            SerialInfo::Entry sentinal;
-            sentinal.typeKind = SerialTypeKind::String;
-            sentinal.info = SerialInfo::EntryInfo::Alignment1;
-
-            size_t offset = 0;
-            const Int entriesCount = m_entries.getCount();
-
-            {
-                m_entries.add(&sentinal);
-                m_entries.removeLast();
-                // Note strictly required in our impl of List. But by writing this and
-                // knowing that removeLast cannot release memory, means the sentinal must be at the
-                // last position.
-                m_entries.getBuffer()[entriesCount] = &sentinal;
-            }
-
-            SerialInfo::Entry* const* entries = m_entries.getBuffer();
-
-            SerialInfo::Entry* entry = entries[1];
-            // We start on 1, because 0 is nullptr and not used for anything
-            for (Index i = 1; i < entriesCount; ++i)
-            {
-                SerialInfo::Entry* next = entries[i + 1];
-
-                // Before writing we need to store the next alignment
-
-                const size_t nextAlignment = SerialInfo::getAlignment(next->info);
-                const size_t alignment = SerialInfo::getAlignment(entry->info);
-                SLANG_UNUSED(alignment);
-
-                entry->info = SerialInfo::combineWithNext(entry->info, next->info);
-
-                // Check we are aligned correctly
-                SLANG_ASSERT((offset & (alignment - 1)) == 0);
-
-                // When we write, we need to make sure it take into account the next alignment
-                const size_t entrySize = entry->calcSize(m_classes);
-
-                // Work out the fix for next alignment
-                size_t nextOffset = offset + entrySize;
-                nextOffset = (nextOffset + nextAlignment - 1) & ~(nextAlignment - 1);
-
-                size_t alignmentFixSize = nextOffset - (offset + entrySize);
-
-                // The fix must be less than max alignment. We require it to be less because we
-                // aligned each Entry to MAX_ALIGNMENT, and so < MAX_ALIGNMENT is the most extra
-                // bytes we can write
-                SLANG_ASSERT(alignmentFixSize < SerialInfo::MAX_ALIGNMENT);
-
-                container->write(entry, entrySize);
-                if (alignmentFixSize)
-                {
-                    container->write(s_fixBuffer, alignmentFixSize);
-                }
-
-                // Onto next
-                offset = nextOffset;
-                entry = next;
-            }
-        }
-    }
-
-    return SLANG_OK;
-}
-
-// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SerialInfo::Entry  !!!!!!!!!!!!!!!!!!!!!!!!
-
-size_t SerialInfo::Entry::calcSize(SerialClasses* serialClasses) const
-{
-    switch (typeKind)
-    {
-    case SerialTypeKind::ImportSymbol:
-    case SerialTypeKind::String:
-        {
-            auto entry = static_cast<const StringEntry*>(this);
-            const uint8_t* cur = (const uint8_t*)entry->sizeAndChars;
-            uint32_t charsSize;
-            int sizeSize = ByteEncodeUtil::decodeLiteUInt32(cur, &charsSize);
-            return SLANG_OFFSET_OF(StringEntry, sizeAndChars) + sizeSize + charsSize;
-        }
-    case SerialTypeKind::Array:
-        {
-            auto entry = static_cast<const ArrayEntry*>(this);
-            return sizeof(ArrayEntry) + entry->elementSize * entry->elementCount;
-        }
-    case SerialTypeKind::RefObject:
-    case SerialTypeKind::NodeBase:
-        {
-            auto entry = static_cast<const ObjectEntry*>(this);
-
-            auto serialClass = serialClasses->getSerialClass(typeKind, entry->subType);
-
-            if (ReflectClassInfo::isSubClassOf(entry->subType, Val::kReflectClassInfo))
-                return sizeof(ValEntry) +
-                       static_cast<const ValEntry*>(this)->operandCount * sizeof(SerialValOperand);
-
-            // Align by the alignment of the entry
-            size_t alignment = getAlignment(entry->info);
-            size_t size = sizeof(ObjectEntry) + serialClass->size;
-
-            size = size + (alignment - 1) & ~(alignment - 1);
-            return size;
-        }
-
-    default:
-        break;
-    }
-
-    SLANG_ASSERT(!"Unknown type");
-    return 0;
-}
-
-// !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! SerialReader  !!!!!!!!!!!!!!!!!!!!!!!!!!!!
-
-SerialReader::~SerialReader()
-{
-    for (const RefObject* obj : m_scope)
-    {
-        const_cast<RefObject*>(obj)->releaseReference();
-    }
-}
-
-const void* SerialReader::getArray(SerialIndex index, Index& outCount)
-{
-    if (index == SerialIndex(0))
-    {
-        outCount = 0;
-        return nullptr;
-    }
-
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-    const Entry* entry = m_entries[Index(index)];
-
-    switch (entry->typeKind)
-    {
-    case SerialTypeKind::Array:
-        {
-            auto arrayEntry = static_cast<const SerialInfo::ArrayEntry*>(entry);
-            outCount = Index(arrayEntry->elementCount);
-            return (arrayEntry + 1);
-        }
-    default:
-        break;
-    }
-
-    SLANG_ASSERT(!"Not an array");
-    outCount = 0;
-    return nullptr;
-}
-
-SerialPointer SerialReader::getPointer(SerialIndex index)
-{
-    if (index == SerialIndex(0))
-    {
-        return SerialPointer();
-    }
-
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-    const Entry* entry = m_entries[Index(index)];
-
-    const SerialPointer& ptr = m_objects[Index(index)];
-
-    switch (entry->typeKind)
-    {
-    case SerialTypeKind::String:
-        {
-            // Hmm. Tricky -> we don't know if will be cast as Name or String. Lets assume string.
-            String string = getString(index);
-            return SerialPointer(string.getStringRepresentation());
-        }
-    case SerialTypeKind::ImportSymbol:
-        {
-            if (ptr.m_kind == SerialTypeKind::Unknown)
-            {
-                // TODO(JS):
-                // Could have an error here, because import symbol was not set
-                // For now just return nullptr
-                return SerialPointer();
-            }
-            break;
-        }
-    default:
-        break;
-    }
-
-    return ptr;
-}
-
-SerialPointer SerialReader::getValPointer(SerialIndex index)
-{
-    if (index == SerialIndex(0))
-    {
-        return SerialPointer();
-    }
-
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-
-    SerialPointer& ptr = m_objects[Index(index)];
-
-    if (ptr.m_ptr)
-        return ptr;
-
-    const SerialInfo::ValEntry* entry = (SerialInfo::ValEntry*)m_entries[Index(index)];
-    ValNodeDesc desc;
-    desc.type = (ASTNodeType)entry->subType;
-    auto readPtr = (SerialInfo::SerialValOperand*)(entry + 1);
-    for (uint32_t i = 0; i < entry->operandCount; i++)
-    {
-        auto serialOperand = readPtr[i];
-        ValNodeOperand operand;
-        operand.kind = (ValNodeOperandKind)(serialOperand.type);
-        switch (operand.kind)
-        {
-        case ValNodeOperandKind::ConstantValue:
-            operand.values.intOperand = serialOperand.payload;
-            break;
-        case ValNodeOperandKind::ASTNode:
-            operand.values.nodeOperand =
-                (NodeBase*)getPointer((SerialIndex)serialOperand.payload).m_ptr;
-            break;
-        case ValNodeOperandKind::ValNode:
-            operand.values.nodeOperand =
-                (Val*)getValPointer((SerialIndex)serialOperand.payload).m_ptr;
-            break;
-        }
-        desc.operands.add(operand);
-    }
-    desc.init();
-    ptr.m_kind = SerialTypeKind::NodeBase;
-    ptr.m_ptr = this->m_objectFactory->getOrCreateVal(_Move(desc));
-    return ptr;
-}
-
-String SerialReader::getString(SerialIndex index)
-{
-    if (index == SerialIndex(0))
-    {
-        return String();
-    }
-
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-    const Entry* entry = m_entries[Index(index)];
-
-    // It has to be a string type
-    if (entry->typeKind != SerialTypeKind::String)
-    {
-        SLANG_ASSERT(!"Not a string");
-        return String();
-    }
-
-    RefObject* obj = m_objects[Index(index)].dynamicCast<RefObject>();
-
-    if (obj)
-    {
-        StringRepresentation* stringRep = dynamicCast<StringRepresentation>(obj);
-        if (stringRep)
-        {
-            return String(stringRep);
-        }
-        // Must be a name then
-        Name* name = dynamicCast<Name>(obj);
-        SLANG_ASSERT(name);
-        return name->text;
-    }
-
-    // Okay we need to construct as a string
-    UnownedStringSlice slice = getStringSlice(index);
-
-    StringRepresentation* stringRep = nullptr;
-
-    const Index length = slice.getLength();
-    if (length)
-    {
-        stringRep = StringRepresentation::createWithCapacityAndLength(length, length);
-        memcpy(stringRep->getData(), slice.begin(), length * sizeof(char));
-        addScope(stringRep);
-    }
-
-    m_objects[Index(index)] = stringRep;
-    return String(stringRep);
-}
-
-Name* SerialReader::getName(SerialIndex index)
-{
-    if (index == SerialIndex(0))
-    {
-        return nullptr;
-    }
-
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-    const Entry* entry = m_entries[Index(index)];
-
-    // It has to be a string type
-    if (entry->typeKind != SerialTypeKind::String)
-    {
-        SLANG_ASSERT(!"Not a string");
-        return nullptr;
-    }
-
-    RefObject* obj = m_objects[Index(index)].dynamicCast<RefObject>();
-
-    if (obj)
-    {
-        Name* name = dynamicCast<Name>(obj);
-        if (name)
-        {
-            return name;
-        }
-        // Can only be a string then
-        StringRepresentation* stringRep = dynamicCast<StringRepresentation>(obj);
-        SLANG_ASSERT(stringRep);
-
-        // I don't need to scope, as scoped in NamePool
-        name = m_namePool->getName(String(stringRep));
-
-        // Store as name, as can always access the inner string if needed
-        m_objects[Index(index)] = name;
-        return name;
-    }
-
-    UnownedStringSlice slice = getStringSlice(index);
-    String string(slice);
-    Name* name = m_namePool->getName(string);
-    // Don't need to add to scope, because scoped on the pool
-    m_objects[Index(index)] = name;
-    return name;
-}
-
-UnownedStringSlice SerialReader::getStringSlice(SerialIndex index)
-{
-    SLANG_ASSERT(SerialIndexRaw(index) < SerialIndexRaw(m_entries.getCount()));
-    const Entry* entry = m_entries[Index(index)];
-
-    // It has to be a string type
-    if (entry->typeKind == SerialTypeKind::String ||
-        entry->typeKind == SerialTypeKind::ImportSymbol)
-    {
-        auto stringEntry = static_cast<const SerialInfo::StringEntry*>(entry);
-
-        const uint8_t* src = (const uint8_t*)stringEntry->sizeAndChars;
-
-        // Decode the string
-        uint32_t size;
-        int sizeSize = ByteEncodeUtil::decodeLiteUInt32(src, &size);
-        return UnownedStringSlice((const char*)src + sizeSize, size);
-    }
-
-    // Can't be accessed as a slice
-    SLANG_ASSERT(!"Not accessible as a slice");
-    return UnownedStringSlice();
-}
-
-/* static */ SlangResult SerialReader::loadEntries(
-    const uint8_t* data,
-    size_t dataCount,
-    SerialClasses* serialClasses,
-    List<const Entry*>& outEntries)
-{
-    // Check the input data is at least aligned to the max alignment (otherwise everything cannot be
-    // aligned correctly)
-    SLANG_ASSERT((size_t(data) & (SerialInfo::MAX_ALIGNMENT - 1)) == 0);
-
-    outEntries.setCount(1);
-    outEntries[0] = nullptr;
-
-    const uint8_t* const end = data + dataCount;
-
-    const uint8_t* cur = data;
-    while (cur < end)
-    {
-        const Entry* entry = (const Entry*)cur;
-        outEntries.add(entry);
-
-        const size_t entrySize = entry->calcSize(serialClasses);
-        cur += entrySize;
-
-        // Need to get the next alignment
-        const size_t nextAlignment = SerialInfo::getNextAlignment(entry->info);
-
-        // Need to fix cur with the alignment
-        cur = (const uint8_t*)((size_t(cur) + nextAlignment - 1) & ~(nextAlignment - 1));
-    }
-
-    return SLANG_OK;
-}
-
-SlangResult SerialReader::constructObjects(NamePool* namePool)
-{
-    m_namePool = namePool;
-
-    m_objects.clearAndDeallocate();
-    m_objects.setCount(m_entries.getCount());
-    memset(m_objects.getBuffer(), 0, m_objects.getCount() * sizeof(void*));
-
-    // Go through entries, constructing objects.
-    for (Index i = 1; i < m_entries.getCount(); ++i)
-    {
-        const Entry* entry = m_entries[i];
-
-        switch (entry->typeKind)
-        {
-        case SerialTypeKind::ImportSymbol:
-            {
-                // We don't construct any object for an imported symbol.
-                // It will be the responsibility of external code to interpet the symbols and *set*
-                // the appopriate objects prior to a call to `deserializeObjects`
-                break;
-            }
-        case SerialTypeKind::String:
-            {
-                // Don't need to construct an object. This is probably a StringRepresentation, or a
-                // Name Will evaluate lazily.
-                break;
-            }
-        case SerialTypeKind::RefObject:
-        case SerialTypeKind::NodeBase:
-            {
-                auto objectEntry = static_cast<const SerialInfo::ObjectEntry*>(entry);
-
-                // Don't create object for Vals.
-                if (objectEntry->typeKind == SerialTypeKind::NodeBase &&
-                    ReflectClassInfo::isSubClassOf(objectEntry->subType, Val::kReflectClassInfo))
-                    break;
-
-                void* obj = m_objectFactory->create(objectEntry->typeKind, objectEntry->subType);
-                if (!obj)
-                {
-                    return SLANG_FAIL;
-                }
-                m_objects[i].set(entry->typeKind, obj);
-                break;
-            }
-        case SerialTypeKind::Array:
-            {
-                // Don't need to construct an object, as will be accessed and interpreted by the
-                // object that holds it
-                break;
-            }
-        }
-    }
-
-    return SLANG_OK;
-}
-
-SlangResult SerialReader::deserializeObjects()
-{
-    // Deserialize
-    for (Index i = 1; i < m_entries.getCount(); ++i)
-    {
-        const Entry* entry = m_entries[i];
-        // First see if there is anything to construct
-        SerialPointer& dstPtr = m_objects[i];
-        if (!dstPtr)
-        {
-            continue;
-        }
-        switch (entry->typeKind)
-        {
-        case SerialTypeKind::NodeBase:
-        case SerialTypeKind::RefObject:
-            {
-                auto objectEntry = static_cast<const SerialInfo::ObjectEntry*>(entry);
-                auto serialClass =
-                    m_classes->getSerialClass(objectEntry->typeKind, objectEntry->subType);
-                if (!serialClass)
-                {
-                    return SLANG_FAIL;
-                }
-                if (ReflectClassInfo::isSubClassOf(objectEntry->subType, Val::kReflectClassInfo))
-                    continue;
-
-                const uint8_t* src = (const uint8_t*)(objectEntry + 1);
-                uint8_t* dst = (uint8_t*)dstPtr.m_ptr;
-
-                // It must be constructed
-                SLANG_ASSERT(dst);
-
-                while (serialClass)
-                {
-                    for (Index j = 0; j < serialClass->fieldsCount; ++j)
-                    {
-                        auto field = serialClass->fields[j];
-                        auto fieldType = field.type;
-                        fieldType->toNativeFunc(
-                            this,
-                            src + field.serialOffset,
-                            dst + field.nativeOffset);
-                    }
-
-                    // Get the super class
-                    serialClass = serialClass->super;
-                }
-
-                break;
-            }
-        default:
-            break;
-        }
-    }
-
-    return SLANG_OK;
-}
-
-
-SlangResult SerialReader::load(const uint8_t* data, size_t dataCount, NamePool* namePool)
-{
-    // Load and place entries into entries table
-    SLANG_RETURN_ON_FAIL(loadEntries(data, dataCount));
-    // Construct all of the objects
-    SLANG_RETURN_ON_FAIL(constructObjects(namePool));
-    SLANG_RETURN_ON_FAIL(deserializeObjects());
-    return SLANG_OK;
-}
-
 } // namespace Slang