| Age | Commit message (Collapse) | Author |
|---|
|
* A new approach to AST serialization
This change completely overhauls the way that AST nodes are being serialized, and the offline source-code generation steps that enable that serialization.
In practice, this ends up being a complete overhaul of the way that *modules* are being serialized (not just the AST part), although things like the serialization format for the Slang IR and for source locations are not affected.
The rest of this commit message is broken down in to sections, in an attempt to help guide anybody looking at the code in how to make sense of all the changes.
The Old C++ Extractor
---------------------
AST serialization used to be driven by information scraped using the `slang-cpp-extractor` tool, which did an ad hoc parse of the C++ declarations of the AST node types and then generated a set of "X macros" that could be for macro-based code generation within the rest of the compiler.
While the existing approach was functional, it wasn't easy to understand or maintain, and it has been getting in the way of forward progress on other features we'd like to work on in the language and compiler.
This change removes the `slang-cpp-extractor` tool entirely.
Marking Up the AST Declarations
-------------------------------
The most notable change that contributors to the compiler may notice is the large number of invocations of a macro `FIDDLE()` on the declarations of the AST node types.
The basic idea is that only declarations (namespaces, types, fields) that are preceded by `FIDDLE()` are visible to the code generator tool.
So if somebody is working with the AST and wondering why a new node type isn't working, or why a field they added isn't being serialized correctly, it is probably because they need to add `FIDDLE()` in front of it.
Generating the Boilerplate Code
-------------------------------
The file `slang-ast-boilerplate.cpp` provides a good example of how the information extracted from the marked-up AST declarations gets used.
In that file, the `FIDDLE TEMPLATE` construct is used to generate type information for each of the AST node types.
Similar logic is used in `slang-ast-forward-declarations.h` to generate the declaration of the `ASTNodeType` enumeration, and forward-declare all the AST node classes.
For many parts of the code, simply including that file replaces the need for the old `slang-generated-*.h` files.
Replacing Visitors and Related Logic
------------------------------------
The old visitor types for the AST used the macros that were generated by `slang-cpp-extractor`, so something new was needed to replace them.
The same goes for the `SLANG_AST_NODE_VIRTUAL_CALL` macros.
The core of the solution implemented here is in `slang-ast-dispatch.h`.
Given a "dispatchable" AST node type (say, `Expr`), a call like:
```
ASTNodeDispatcher<Expr,R>(expr, [&](auto e) { return doSomething(e); })
```
is an expression of type `R`, which does the equivalent of something like:
```
switch(expr->getTag())
{
case ASTNodeType::VarExpr: return doSomething(static_cast<VarExpr*>(expr));
// ...
}
```
The `SLANG_AST_NODE_VIRTUAL_CALL` macro is now implemented in terms of `ASTNodeDispatcher`.
The implementation of the visitor types is more involved.
The code in this change retains some of the macro names from the original version, just to try and make the parallels more clear.
The visitor types are all implemented on top of the `ASTNodeDispatcher` approach, and use `FIDDLE TEMPLATE` to generate all the boilerplate `visit*()` method declarations.
Refactoring of `Linkage` Module Loading
---------------------------------------
Needing to revisit all the places where modules get deserialized made it clear that there is a lot of complexity and apparent duplication in the core routines on the `Linkage` that get used for loading modules.
This change tries to clean up some of that logic, but it is worth noting that there are two legacy features that get in the way of making things as clean as they should be:
* The `LoadedModuleDictionary` type that gets passed around a lot exists entirely to handle the corner case where somebody uses the Slang API to perform a compilation with multiple `TranslationUnitRequest`s in the same `FrontEndCompileRequest`, and one of the translation units `import`s the module defined by another of the translation units.
* There are a lot of special-case behaviors and routines entirely there to support the `ModuleLibrary` feature, although that feature should be considered deprecated (or at least subject to getting entirely re-designed down the line).
The basic idea of the cleanup is that all of the (non-deprecated) ways load a module from a serialized binary, or compile one from source should now bottleneck through `loadModuleImpl`, which then bifurcates into `loadSourceModuleImpl` for the compilation case and `loadBinaryModuleImpl` for the deserialization case.
High-Level Serialization Approach
---------------------------------
The old serialization logic used the [RIFF](https://en.wikipedia.org/wiki/Resource_Interchange_File_Format) format to encode the high-level structure of things, and this change retains that usage (and actually doubles down on the RIFF usage).
The old serialization system relied on the idea that for any given type `Foo` that wants to support serialization, there should be something like a `SerialFooData` type in C++, that can represent the state of a `Foo`, and then the actual serialization applied to that `SerialFooData`. This means that in most cases there are four pieces of code written:
* During serialization:
* Copying the data of a `Foo` in memory over to a `SerialFooData` in memory
* Writing the state of a `SerialFooData` into the serialized data stream
* During deserialization:
* Reading the state of a `SerialFooData` from a serialized data stream
* Copying the data of the `SerialFooData` in memory over to a `Foo`
The new logic gets rid of the intermediate `SerialFooData`.
In the serialization direction, we take a `Foo` and write it to the `RIFFContainer` directly, or using some other utilities layered on top of it.
In the deserialization direction, we have additional flexibility. Given a `RIFFContainer::Chunk*` that represents a serialized `Foo`, we often navigate through the in-memory representation of the RIFF data to get to the parts of the serialized value that we actually want/need, without needing to deserialize the entire `Foo`.
To support this kind of operation, this change introduces a few helper types like `ContainerChunkRef` an `ModuleChunkRef`, that are little more than typed wrappers around a `RIFFContainer::Chunk*`.
The Module "Container" Part
---------------------------
A serialized `Module` is encoded as a RIFF chunk, using logic in `slang-serialize-container.cpp` - both before and after this change.
This change reorganizes a lot of the code in that file, to account for the way that eliminating the intermediate `SerialContainerData` type streamlines the overall task of writing out the parts of the module.
In the deserialization logic... there isn't really much to do in `slang-serialize-container.cpp`. Most of the logic in `slang.cpp` and `slang-module-library.cpp` that pertains to deserializing modules uses the `ModuleChunkRef`-based approach, and simply extracts the pieces of the serialized module that it needs.
The Actual Serialization of the AST
-----------------------------------
The actual AST serialization logic is in `slang-serialize-ast.cpp`.
The basic approach in both the writing and reading directions is:
* Use the `FIDDLE TEMPLATE` system to generate a set of functions, one for each AST node type, that recursively invoke the read/write logic on each field of that node (after recursively invoking the case for its direct superclass)
* Use the `ASTNodeDispatcher` system to dispatch out to those functions whene reading or writing anything derived from `NodeBase`
* For now, handle all types *not* derived from `NodeBase` by hand.
There's a lot of room for improvement around that last item: it should be just as easy to generate the serialization and deserialization logic for other types that don't inherit from `NodeBase`, but the current change tries to err on the side of making the logic as explicit and simplistic as possible, rather than trying to get too clever too soon.
The actual serialization *format* used for the AST is almost comically simplistic: the code uses hierarchical RIFF chunks to emulate a JSON-like structure. This is a very wasteful representation (e.g., a `bool` or a null pointer each take up *8 bytes*), but the goal for now is to start with the simplest thing that could possibly work, and only add more cleverness once we are sure it won't get in the way of important future improvements (like lazy/on-demand deserialization or IR and AST, to improve compiler startup times).
The files `slang-serialize.{h,cpp}` have been co-opted to define a new pair of types `Encoder` and `Decoder` that are used for a more-or-less stream-oriented way or reading or writing RIFF chunks for the JSON-like structure.
Almost everything related to the actual AST serialization could do with a cleanup pass, and some time spent on picking good/better names for everything.
Smaller Stuff
-------------
* Cleaned up a lot of code that was using bare `ASTNodeType` or the extractor's `ReflectClassInfo` type to consistently use `SyntaxClass`.
* Fixed an apparent bug in how the destination-driven code genarator was handling `TryExpr`s
* Fixed an apparent bug in how the GLSL legalization pass was handling translation of certain `SV_*` semantics.
* format code
* fixup: template errors caught by non-VS compilers
* format code
* fixup: more template errors
* fixup: more stuff VS didn't catch
* fixup: it's amazing VS doesn't catch these...
* fixup: yet more template stuff VS ignores
* fixup: more VS template nonsense
* fixup: unreachable return macro usage
* fixup: more unreacable returns
* fixup: unused parameter
* fixup: strict aliasing
* fixup: allow missing entry point list chunk
* fixup: wasm build script
* fixup: AST changes since this PR was created
---------
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>
Co-authored-by: Yong He <yonghe@outlook.com>
|
|
* format
* Minor test fixes
* enable checking cpp format in ci
|
|
* Support mixture of precompiled and non-precompiled modules
This changes the implementation of precompile DXIL modules to
accept combinations of modules with precompiled DXIL, ones without,
and ones with a mixture of precompiled DXIL and Slang IR.
During precompilation, module IR is analyzed to find public functions
which appear to be capable of being compiled as HLSL, and those
functions are given a HLSLExport decoration, ensuring they are emitted
as HLSL and preserved in the precompiled DXIL blob. The IR for those
functions is then tagged with a new decoration AvailableInDXIL, which
marks that their implementation is present in the embedded DXIL blob.
The DXIL blob is attached to the IR as before, inside a EmbeddedDXIL
BlobLit instruction.
The logic that determines whether or not functions should be
precompiled to DXIL is a placeholder at this point, returning true
always. A subsequent change will add selection criteria.
During module linking, the full module IR is available, as well
as the optional EmbeddedDXIL blob. The IR for functions implemented
by the blob are tagged with AvailableInDXIL in the module IR.
After linking the IR for all modules to program level IR, the IR for
the functions marked AvailableInDXIL are deleted from the linked IR,
prior to emitting HLSL and compiling linking the result.
This change also changes the point of time when the module IR is
checked for EmbeddedDXIL blobs. Instead of happening at load time
as before, it happens during immediately before final linking, meaning
that the blob does not need to be independently stored with the module
separate from the IR as was done previously.
Work on #4792
* Clean up debug prints
* Call isSimpleHLSLDataType stub
* Address feedback on precompiled dxil support
Allow for IR filtering both before and after linking.
Only mark AvailableInDXIL those functions which pass
both filtering stages. Functions are corrlated using
mangled function names.
Rather than delete functions entirely when linking with
libraries that include precompiled DXIL, instead convert
the IR function definitions to declarations by gutting
them, removing child blocks.
* Use artifact metadata and name list instead of linkedir hack
* Use String instead of UnownedStringSlice
* Update tests
* Renaming
* Minor edits
* Don't fully remove functions post-link
* Unexport before collecting metadata
|
|
* Add embedded precompiled binary IR ops
Add IR operations to embed precompiled DXIL or SPIR-V blobs
into IR. Adds a BlobLit literal that is mostly identical to
StringLit except for its inability to be displayed, e.g.
in dumped IR. In the future, the blob might be dumped as
hexadecimal, but for now it is summarized as "<binary blob>".
* EmbeddedDXIL and SPIR-V options
The options, '-embed-dxil' and '-embed-spirv' in slangc, will
cause a target dxil or spirv to be compiled and stored in the
translation unit IR when written to a slang-module. Subsequent
changes actually implement the options.
* Per-translation unit DXIL precompilation
When -embed-dxil is specified, perform a precompilation to DXIL of
each TU, linked only with stdlib. Embed the resulting DXIL for
the TU in a IR op. Being part of IR, the precompiled DXIL can be
serialized to disk in a slang-module.
Upon loading slang-modules, the new IR op will be searched for and
the precompiled DXIL blob is saved with the loaded Module. During
linking, if all the Modules have precompiled blobs they will be
sent to the downstream compile commands as libraries instead of
source, skipping the downstream compilation, using DXC only for
linking.
Fixes Issue #4580
* Remove placeholder embedded SPIRV support
Code was added only to sketch out how other precompiled bins
will be supported.
* Remove the rest of the SPIRV placeholder support
* Fix warnings, test error on non-windows
* Remove lib_6_6 hack, add dxil_lib capability
* Allocate blob value from irmodule memarena
* Add null check after memarena allocation
* Restore the request->e2erequest code path for generatewholeprogram
* Update capability handling, move EmbedDXIL enum to end to preserve abi
* Remove lib_6_6 hack
* Move ICompileRequest functions to end
|
|
* Add slangc interface to compile and use ir modules.
* Fix glsl scalar layout settings not copied to target.
* Fix.
* Cleanups.
|
|
* Support loading serialized modules.
* Fix.
* Fix vs solution files
* Fix glsl module loading.
* C++ fix.
* Fix.
* Try fix c++ error.
* Try fix.
* Fix.
* Fix.
|
|
* #include an absolute path didn't work - because paths were taken to always be relative.
* WIP with hierarchical enums.
* Some small fixes and improvements around artifact desc related types.
* Improvements around hierarchical enum.
* Fixes to get Artifact types refactor to be able to execute tests.
* Attempt to better categorize PTX.
* Work around for potentially unused function warning.
* Typo fix.
* Simplify Artifact header.
* Small improvements around Artifact kind/payload/style.
* Added IDestroyable/ICastable
* Add IArtifactList.
* First impl of IArtifactUtil.
* Use the ICastable interface for IArtifactRepresentation.
* Added IArtifactRepresentation & IArtifactAssociated.
* Add SLANG_OVERRIDE to avoid gcc/clang warning.
* Fix calling convention issue on win32.
* Fix missing SLANG_OVERRIDE.
* First attempt at file abstraction around Artifact.
* Added creation of lock file.
* Move functionality for determining file paths to the IArtifactUtil.
Add casting to ICastable.
* Added some casting/finding mechanisms.
* Simplify IArtifact interface, and use Items for file reps.
* Fix problem with libraries on DXIL.
* Split out ArtifactRepresentation.
* Move ArtifactDesc functionality to ArtifactDescUtil. ArtifactInfoUtil becomes ArtifactDescUtil.
* Split implementations from the interfaces for Artifact.
* Use TypeTextUtil for target name outputting.
* Add artifact impls.
* Add ICastableList
* Added UnknownCastableAdapter
* Make ISlangSharedLibrary derive from ICastable, and remain backwards compatible with slang-llvm.
* Refactor Representation on Artifact.
* Make our ISlangBlobs also derive from ICastable.
Make ISlangBlob atomic ref counted.
* Split out CastableList and related types, and placed in core.
* Small fixes around IArtifact.
Improve IArtifact docs.
First impl of getChildren for IArtifact.
* Documentation improvements for Artifact related types.
* Fix typo.
* Special case adding a ICastableList to a LazyCastableList.
* Small simplification of LazyCastableList, by adding State member.
* Removed the ILockFile interface because IFileArtifactRepresentation can be used.
* Implement DiagnosticsArtifactRepresentation.
* Added PostEmitMetadataArtifactRepresentation
* Add searching by predicate.
Added handling of accessing Artifact as ISharedLibrary
* Fix typo.
* Add find to IArtifacgtList.
Fix some missing SLANG_NO_THROW.
* Small improvements around ArtifactDesc types.
* Another small change around ArtifactKind.
* Some more shuffling of ArtifactDesc.
* Make IArtifact castable
Remove IArtifactList
Made IArtifactContainer derive from IArtifact
Made ModuleLibrary atomic ref counted/given IModuleLibrary interface.
* Must call _requireChildren before any children access.
* Fix missing SLANG_MCALL on castAs.
* Fix missing SLANG_OVERRIDE.
* Added IArtifactHandler
* Use ICastable for basis of scope/lookup.
|
|
* #include an absolute path didn't work - because paths were taken to always be relative.
* WIP with hierarchical enums.
* Some small fixes and improvements around artifact desc related types.
* Improvements around hierarchical enum.
* Fixes to get Artifact types refactor to be able to execute tests.
* Attempt to better categorize PTX.
* Work around for potentially unused function warning.
* Typo fix.
* Simplify Artifact header.
* Small improvements around Artifact kind/payload/style.
* Added IDestroyable/ICastable
* Add IArtifactList.
* First impl of IArtifactUtil.
* Use the ICastable interface for IArtifactRepresentation.
* Added IArtifactRepresentation & IArtifactAssociated.
* Add SLANG_OVERRIDE to avoid gcc/clang warning.
* Fix calling convention issue on win32.
* Fix missing SLANG_OVERRIDE.
* First attempt at file abstraction around Artifact.
* Added creation of lock file.
* Move functionality for determining file paths to the IArtifactUtil.
Add casting to ICastable.
* Added some casting/finding mechanisms.
* Simplify IArtifact interface, and use Items for file reps.
* Fix problem with libraries on DXIL.
* Split out ArtifactRepresentation.
* Move ArtifactDesc functionality to ArtifactDescUtil. ArtifactInfoUtil becomes ArtifactDescUtil.
* Split implementations from the interfaces for Artifact.
* Use TypeTextUtil for target name outputting.
* Add artifact impls.
* Add ICastableList
* Added UnknownCastableAdapter
* Make ISlangSharedLibrary derive from ICastable, and remain backwards compatible with slang-llvm.
* Refactor Representation on Artifact.
* Make our ISlangBlobs also derive from ICastable.
Make ISlangBlob atomic ref counted.
* Fix typo.
|
|
* #include an absolute path didn't work - because paths were taken to always be relative.
* WIP with hierarchical enums.
* Some small fixes and improvements around artifact desc related types.
* Improvements around hierarchical enum.
* Fixes to get Artifact types refactor to be able to execute tests.
* Attempt to better categorize PTX.
* Work around for potentially unused function warning.
* Typo fix.
* Simplify Artifact header.
* Small improvements around Artifact kind/payload/style.
* Added IDestroyable/ICastable
* Add IArtifactList.
* First impl of IArtifactUtil.
* Use the ICastable interface for IArtifactRepresentation.
* Added IArtifactRepresentation & IArtifactAssociated.
* Add SLANG_OVERRIDE to avoid gcc/clang warning.
* Fix calling convention issue on win32.
* Fix missing SLANG_OVERRIDE.
|
|
* #include an absolute path didn't work - because paths were taken to always be relative.
* Compile to a dxil library.
* Added CompileProduct.
* Support handling of ModuleLibrary.
* CacheBehavior -> Cache
* Use CompileProduct for -r references.
* CompileProduct -> Artifact.
* Determining an artifact type on binding.
* Determine binary linkability.
* Added Artifact::exists.
* Added ArtifactKeep.
* Small fixes.
* Small improvements to Artifact.
* Add zip extension.
* Fix some comments.
* Fix multiple adding of PublicDecoration.
Make public output export for DXIL/lib.
Add checking for simpleDecorations such that only added once.
* Use 'whole program' to identify library build.
* Move slang-artifact into compiler-core.
* Split out Keep free functions.
* Artifact::Keep -> ArtifactKeep.
* Handle libraries as artifacts.
* Add -target dxil so test infrastructure knows it needs DXC.
* Linking working in DXC.
* Improve handling around emit for 'export'.
* Add comment around Artifact name.
* Render test working with linking.
* Improvements around Artifact handling.
* Add ArtifactPayloadInfo.
* Small tidy up around artifact.
* Split out code to get info about Artifacts into artifact-info.cpp/.h
* IArtifact interface and IArtifactInstance interface.
* Fix small issues.
* Fix compilation warning issue.
* Fix missing SLANG_OVERRIDE.
* Small fixes to make compilation work on Visual Studio 2022.
* Small improvements to Artifact interface/naming.
* Added Desc with each element in IArchive to allow more flexibility in usage.
* Fix clang warning issue.
* Add ArtifactPayload::Diagnostics
* More discussion around IArtifact usage.
* Re-add slang-artifact.h which was removed during merge.
* Fix typo identified in review.
|
|
* #include an absolute path didn't work - because paths were taken to always be relative.
* Compile to a dxil library.
* Added CompileProduct.
* Support handling of ModuleLibrary.
* CacheBehavior -> Cache
* Use CompileProduct for -r references.
* CompileProduct -> Artifact.
* Determining an artifact type on binding.
* Determine binary linkability.
* Added Artifact::exists.
* Added ArtifactKeep.
* Small fixes.
* Small improvements to Artifact.
* Add zip extension.
* Fix some comments.
* Fix multiple adding of PublicDecoration.
Make public output export for DXIL/lib.
Add checking for simpleDecorations such that only added once.
* Use 'whole program' to identify library build.
* Move slang-artifact into compiler-core.
* Split out Keep free functions.
* Artifact::Keep -> ArtifactKeep.
* Handle libraries as artifacts.
* Add -target dxil so test infrastructure knows it needs DXC.
* Linking working in DXC.
* Improve handling around emit for 'export'.
* Add comment around Artifact name.
* Render test working with linking.
Co-authored-by: Theresa Foley <10618364+tangent-vector@users.noreply.github.com>
|
