slang.git/source/slang/slang-serialize-container.h, branch master

Use fossil for IR serialization (#7619)

2025-07-08T02:36:52+00:00

* bottleneck ir module reading and writing * compute/simple working * more complex tests working * neaten * factor out SourceLoc serialization * document changes * Appease clang * Correct name serialization * remove unnecessary code * neaten * neaten

Remove some cruft/complexity from IR serialization (#7483)

2025-07-01T01:20:33+00:00

* Remove some cruft/complexity from IR serialization This is a very simple cleanup to unnecessary code paths and remove some flexibility that isn't actually needed, to hopefully simplify the task of more completely overhauling the approach to IR serialization in a later change. The concrete feature that gets removed here is a debug-only feature (which thus shouldn't be affecting any users of Slang) that was added long ago in the life of the compiler as we were working to truly separate the front- and back-ends. At the time there was a lot of code in the compiler back-end that still made use of AST-level data structures, and thus got in the way of our goal to support separate compilation and linking (such that final code generation can only depend on the IR, and not the AST). The option was used to cause the Slang IR to be serialized out and then read back in as part of compilation, to try and enforce that only the wanted constructs could pass through that bottleneck. The idea was only ever half implemented, however, because it made use of a secondary implementation path in IR serialization that supported serializing the "raw" source locations (which are heavily dependent on AST-level information, even down to the number of bytes in source files). This change removes the feature entirely, since it is no longer useful for its intended purpose, and its presence causes there to be entire second code path for source locations in IR serialization that would need to have test coverage if we wanted to be sure it kept working. In addition, our pre-existing infrastructure for module serialization had various options that have either stopped being useful, or were not really useful at the time they were introduced. For example: there are no places in the code today where we attempt to serialize out a module without including both the serialized AST and IR. If that was a feature that we ever supported, the relevant code got removed at some preceding point without breaking any of our tests or (seemingly) upsetting users. Similarly, the options being passed into writing of a serialized module included both a flag to control whether source locations should be serialized *and* a pointer to the `SourceManager` to use in that case... but it was only ever meaningful to set both, or neither. The option has been changed to just be the `SourceManager` pointer, and the name has been updated to reflect its very narrow intended use case. * format code * fixup * regenerate command line reference --------- Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com> Co-authored-by: Yong He

Cleanups related to RIFF support (#7041)

2025-05-12T17:28:05+00:00

A new approach to AST serialization (#6854)

2025-04-22T20:26:57+00:00

* A new approach to AST serialization This change completely overhauls the way that AST nodes are being serialized, and the offline source-code generation steps that enable that serialization. In practice, this ends up being a complete overhaul of the way that *modules* are being serialized (not just the AST part), although things like the serialization format for the Slang IR and for source locations are not affected. The rest of this commit message is broken down in to sections, in an attempt to help guide anybody looking at the code in how to make sense of all the changes. The Old C++ Extractor --------------------- AST serialization used to be driven by information scraped using the `slang-cpp-extractor` tool, which did an ad hoc parse of the C++ declarations of the AST node types and then generated a set of "X macros" that could be for macro-based code generation within the rest of the compiler. While the existing approach was functional, it wasn't easy to understand or maintain, and it has been getting in the way of forward progress on other features we'd like to work on in the language and compiler. This change removes the `slang-cpp-extractor` tool entirely. Marking Up the AST Declarations ------------------------------- The most notable change that contributors to the compiler may notice is the large number of invocations of a macro `FIDDLE()` on the declarations of the AST node types. The basic idea is that only declarations (namespaces, types, fields) that are preceded by `FIDDLE()` are visible to the code generator tool. So if somebody is working with the AST and wondering why a new node type isn't working, or why a field they added isn't being serialized correctly, it is probably because they need to add `FIDDLE()` in front of it. Generating the Boilerplate Code ------------------------------- The file `slang-ast-boilerplate.cpp` provides a good example of how the information extracted from the marked-up AST declarations gets used. In that file, the `FIDDLE TEMPLATE` construct is used to generate type information for each of the AST node types. Similar logic is used in `slang-ast-forward-declarations.h` to generate the declaration of the `ASTNodeType` enumeration, and forward-declare all the AST node classes. For many parts of the code, simply including that file replaces the need for the old `slang-generated-*.h` files. Replacing Visitors and Related Logic ------------------------------------ The old visitor types for the AST used the macros that were generated by `slang-cpp-extractor`, so something new was needed to replace them. The same goes for the `SLANG_AST_NODE_VIRTUAL_CALL` macros. The core of the solution implemented here is in `slang-ast-dispatch.h`. Given a "dispatchable" AST node type (say, `Expr`), a call like: ``` ASTNodeDispatcher(expr, [&](auto e) { return doSomething(e); }) ``` is an expression of type `R`, which does the equivalent of something like: ``` switch(expr->getTag()) { case ASTNodeType::VarExpr: return doSomething(static_cast(expr)); // ... } ``` The `SLANG_AST_NODE_VIRTUAL_CALL` macro is now implemented in terms of `ASTNodeDispatcher`. The implementation of the visitor types is more involved. The code in this change retains some of the macro names from the original version, just to try and make the parallels more clear. The visitor types are all implemented on top of the `ASTNodeDispatcher` approach, and use `FIDDLE TEMPLATE` to generate all the boilerplate `visit*()` method declarations. Refactoring of `Linkage` Module Loading --------------------------------------- Needing to revisit all the places where modules get deserialized made it clear that there is a lot of complexity and apparent duplication in the core routines on the `Linkage` that get used for loading modules. This change tries to clean up some of that logic, but it is worth noting that there are two legacy features that get in the way of making things as clean as they should be: * The `LoadedModuleDictionary` type that gets passed around a lot exists entirely to handle the corner case where somebody uses the Slang API to perform a compilation with multiple `TranslationUnitRequest`s in the same `FrontEndCompileRequest`, and one of the translation units `import`s the module defined by another of the translation units. * There are a lot of special-case behaviors and routines entirely there to support the `ModuleLibrary` feature, although that feature should be considered deprecated (or at least subject to getting entirely re-designed down the line). The basic idea of the cleanup is that all of the (non-deprecated) ways load a module from a serialized binary, or compile one from source should now bottleneck through `loadModuleImpl`, which then bifurcates into `loadSourceModuleImpl` for the compilation case and `loadBinaryModuleImpl` for the deserialization case. High-Level Serialization Approach --------------------------------- The old serialization logic used the [RIFF](https://en.wikipedia.org/wiki/Resource_Interchange_File_Format) format to encode the high-level structure of things, and this change retains that usage (and actually doubles down on the RIFF usage). The old serialization system relied on the idea that for any given type `Foo` that wants to support serialization, there should be something like a `SerialFooData` type in C++, that can represent the state of a `Foo`, and then the actual serialization applied to that `SerialFooData`. This means that in most cases there are four pieces of code written: * During serialization: * Copying the data of a `Foo` in memory over to a `SerialFooData` in memory * Writing the state of a `SerialFooData` into the serialized data stream * During deserialization: * Reading the state of a `SerialFooData` from a serialized data stream * Copying the data of the `SerialFooData` in memory over to a `Foo` The new logic gets rid of the intermediate `SerialFooData`. In the serialization direction, we take a `Foo` and write it to the `RIFFContainer` directly, or using some other utilities layered on top of it. In the deserialization direction, we have additional flexibility. Given a `RIFFContainer::Chunk*` that represents a serialized `Foo`, we often navigate through the in-memory representation of the RIFF data to get to the parts of the serialized value that we actually want/need, without needing to deserialize the entire `Foo`. To support this kind of operation, this change introduces a few helper types like `ContainerChunkRef` an `ModuleChunkRef`, that are little more than typed wrappers around a `RIFFContainer::Chunk*`. The Module "Container" Part --------------------------- A serialized `Module` is encoded as a RIFF chunk, using logic in `slang-serialize-container.cpp` - both before and after this change. This change reorganizes a lot of the code in that file, to account for the way that eliminating the intermediate `SerialContainerData` type streamlines the overall task of writing out the parts of the module. In the deserialization logic... there isn't really much to do in `slang-serialize-container.cpp`. Most of the logic in `slang.cpp` and `slang-module-library.cpp` that pertains to deserializing modules uses the `ModuleChunkRef`-based approach, and simply extracts the pieces of the serialized module that it needs. The Actual Serialization of the AST ----------------------------------- The actual AST serialization logic is in `slang-serialize-ast.cpp`. The basic approach in both the writing and reading directions is: * Use the `FIDDLE TEMPLATE` system to generate a set of functions, one for each AST node type, that recursively invoke the read/write logic on each field of that node (after recursively invoking the case for its direct superclass) * Use the `ASTNodeDispatcher` system to dispatch out to those functions whene reading or writing anything derived from `NodeBase` * For now, handle all types *not* derived from `NodeBase` by hand. There's a lot of room for improvement around that last item: it should be just as easy to generate the serialization and deserialization logic for other types that don't inherit from `NodeBase`, but the current change tries to err on the side of making the logic as explicit and simplistic as possible, rather than trying to get too clever too soon. The actual serialization *format* used for the AST is almost comically simplistic: the code uses hierarchical RIFF chunks to emulate a JSON-like structure. This is a very wasteful representation (e.g., a `bool` or a null pointer each take up *8 bytes*), but the goal for now is to start with the simplest thing that could possibly work, and only add more cleverness once we are sure it won't get in the way of important future improvements (like lazy/on-demand deserialization or IR and AST, to improve compiler startup times). The files `slang-serialize.{h,cpp}` have been co-opted to define a new pair of types `Encoder` and `Decoder` that are used for a more-or-less stream-oriented way or reading or writing RIFF chunks for the JSON-like structure. Almost everything related to the actual AST serialization could do with a cleanup pass, and some time spent on picking good/better names for everything. Smaller Stuff ------------- * Cleaned up a lot of code that was using bare `ASTNodeType` or the extractor's `ReflectClassInfo` type to consistently use `SyntaxClass`. * Fixed an apparent bug in how the destination-driven code genarator was handling `TryExpr`s * Fixed an apparent bug in how the GLSL legalization pass was handling translation of certain `SV_*` semantics. * format code * fixup: template errors caught by non-VS compilers * format code * fixup: more template errors * fixup: more stuff VS didn't catch * fixup: it's amazing VS doesn't catch these... * fixup: yet more template stuff VS ignores * fixup: more VS template nonsense * fixup: unreachable return macro usage * fixup: more unreacable returns * fixup: unused parameter * fixup: strict aliasing * fixup: allow missing entry point list chunk * fixup: wasm build script * fixup: AST changes since this PR was created --------- Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com> Co-authored-by: Yong He

Remove support for ad hoc Slang IR compression (#6834)

2025-04-16T19:28:39+00:00

* Remove support for ad hoc Slang IR compression This change is part of a larger effort to clean up the approach to serialization in the Slang compiler. The overall goal is to simplify and streamline all of the serialization-related logic, so that we are left with code that is less "clever," and easier to understand for contributors to the codebase. Removing support for compression of serialized Slang IR has benefits that include: * Reduction in code complexity: consider things like the subtle way that the `FOURCC`s for compressed chunks were being computed from the uncompressed versions, and the mental overhead that goes into understanding that, for anybody who would dare to touch this code. * Reduction in testing burden: there have been, de facto, two very different code paths for serialization of the Slang IR, and it is not clear that the existing test corpus for Slang has sufficient coverage for both options. By having only a single code path, every test that performs any amount of IR serialization helps with test coverage of that one path. * Opportunity to explore alternatives. This is perhaps a reiteration of the first point, but once the code is stripped down to the simplest thing that could possibly work (I am not claiming it has reached that point yet), it becomes easier for contributors to understand, and it becomes more tractable for somebody to come along with an improved approach that performs better (in either compression ratio or performance) while still being maintainable. In my own local setup, I found that removing support for Slang IR compression led to the `slang-core-module-generated.h` file increasing in size from 46.1MB to 47.4MB. This increase in the `.h` file size for the core library binary only resulted in a release build of `slang.dll` increasing from 20.0MB to 20.2MB. Removing the ad hoc compression support has almost no impact on the size of actual binary Slang modules *so long* as the additional LZ4 compression step is being applied to them. * format code --------- Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

format

2024-10-29T06:49:26+00:00

* format * Minor test fixes * enable checking cpp format in ci

Add slangc interface to compile and use ir modules. (#3615)

2024-02-24T00:39:46+00:00

* Add slangc interface to compile and use ir modules. * Fix glsl scalar layout settings not copied to target. * Fix. * Cleanups.

Add API for querying and reusing precompiled binary modules. (#3614)

2024-02-22T15:14:55+00:00

Support loading serialized modules. (#3588)

2024-02-15T08:05:51+00:00

* Support loading serialized modules. * Fix. * Fix vs solution files * Fix glsl module loading. * C++ fix. * Fix. * Try fix c++ error. * Try fix. * Fix. * Fix.

SPIR-V WIP (#3064)

2023-08-15T12:28:42+00:00

* Add type layout for structured buffer * Default to generating spirv directly * vk test for compute simple * Add spirv-dis as a downstream compiler * Emit Array types in SPIR-V * makevector for spirv * Dump whole spirv module on validation failure * register array types todo, use emitTypeInst * Neater formatting for unhandled inst printing * break out emitCompositeConstruct * Correct array type generation * neaten * Allow getElement for vector * Remove unused * Allow predicating target intrinsics on types * Consider functions with intrinsics to have definitions We need to specialize these if they are predicated on types * Correct array type generation * makeArray for spir-v * replace getElement with getElementPtr for spirv * Correct translation of field access for spirv * Push layouts to types for spirv * Spirv intrinsics * operator now makes a pointer * Add structured buffer of struct test * Preserve type layout in spirv structured buffer legalization * neaten * makeVectorFromScalar for SPIRV * placeholder for layouts on param groups * More type safe spirv op construction * Know that constants and types only go in one section * Remove emitTypeInst * Add todo for spirv sampling * Add links to spirv documentation on emit functions * OpTypeImage support for SPIR-V * Add simpler texture test for spirv * s/spirv_direct/spirv/g * Allow several string literals in target_intrinsic * Handle global params without a var layour for SPIR-V For example groupshared vars * uint spirv asm type * Add todo for isDefinition It is currently too broad * Some atomic op spirv intrinsics * Strip ConstantBuffer wrappers for spirv * Add todo for matrix annotations * Do not associate decorations insts with spirv counterparts * Correct entry point parameter generation * Spelling * Assert that fieldAddress is returning a pointer * Add error for existential type layout getting to spir-v emit * Add IRTupleTypeLayout Unused so far * Allow getElementPtr to work with vectors * Correct target name in test * Hide default spirv direct behind a premake option --default-spirv-direct=true * Do not insert space at start of intrinsic def * Correct asm rendering in tests * remove redundant option * Emit directly from direct test * Add source language options for spirv-dis * Add comments to spirv dis * Add dead debug print for before spirv module * Correct asm rendering in tests * s/spirv_direct/spirv/g * Only specialize intrinsic functions with predicates * regenerate vs projects * squash warnings * squash warnings * remove duplication * Silence warnings from msvc * squash warnings * Overload for zero sized array * More msvc warnings * warnings * Add spirv-tools to path for tests * Do not be specific about dxc version for diag test * Normalize line endings from spirv-dis * Correct filecheck matches * Temporarily disable two spirv tests Failing on CI, undebuggable hang :/ * Do not emit storage class more than once for spirv snippet * Do not pass spir-v to spirv-dis by stdin * Do not get spirv-dis output via stream, use file * normalize file endings in spirv-dis output