| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* A new approach to AST serialization
This change completely overhauls the way that AST nodes are being serialized, and the offline source-code generation steps that enable that serialization.
In practice, this ends up being a complete overhaul of the way that *modules* are being serialized (not just the AST part), although things like the serialization format for the Slang IR and for source locations are not affected.
The rest of this commit message is broken down in to sections, in an attempt to help guide anybody looking at the code in how to make sense of all the changes.
The Old C++ Extractor
---------------------
AST serialization used to be driven by information scraped using the `slang-cpp-extractor` tool, which did an ad hoc parse of the C++ declarations of the AST node types and then generated a set of "X macros" that could be for macro-based code generation within the rest of the compiler.
While the existing approach was functional, it wasn't easy to understand or maintain, and it has been getting in the way of forward progress on other features we'd like to work on in the language and compiler.
This change removes the `slang-cpp-extractor` tool entirely.
Marking Up the AST Declarations
-------------------------------
The most notable change that contributors to the compiler may notice is the large number of invocations of a macro `FIDDLE()` on the declarations of the AST node types.
The basic idea is that only declarations (namespaces, types, fields) that are preceded by `FIDDLE()` are visible to the code generator tool.
So if somebody is working with the AST and wondering why a new node type isn't working, or why a field they added isn't being serialized correctly, it is probably because they need to add `FIDDLE()` in front of it.
Generating the Boilerplate Code
-------------------------------
The file `slang-ast-boilerplate.cpp` provides a good example of how the information extracted from the marked-up AST declarations gets used.
In that file, the `FIDDLE TEMPLATE` construct is used to generate type information for each of the AST node types.
Similar logic is used in `slang-ast-forward-declarations.h` to generate the declaration of the `ASTNodeType` enumeration, and forward-declare all the AST node classes.
For many parts of the code, simply including that file replaces the need for the old `slang-generated-*.h` files.
Replacing Visitors and Related Logic
------------------------------------
The old visitor types for the AST used the macros that were generated by `slang-cpp-extractor`, so something new was needed to replace them.
The same goes for the `SLANG_AST_NODE_VIRTUAL_CALL` macros.
The core of the solution implemented here is in `slang-ast-dispatch.h`.
Given a "dispatchable" AST node type (say, `Expr`), a call like:
```
ASTNodeDispatcher<Expr,R>(expr, [&](auto e) { return doSomething(e); })
```
is an expression of type `R`, which does the equivalent of something like:
```
switch(expr->getTag())
{
case ASTNodeType::VarExpr: return doSomething(static_cast<VarExpr*>(expr));
// ...
}
```
The `SLANG_AST_NODE_VIRTUAL_CALL` macro is now implemented in terms of `ASTNodeDispatcher`.
The implementation of the visitor types is more involved.
The code in this change retains some of the macro names from the original version, just to try and make the parallels more clear.
The visitor types are all implemented on top of the `ASTNodeDispatcher` approach, and use `FIDDLE TEMPLATE` to generate all the boilerplate `visit*()` method declarations.
Refactoring of `Linkage` Module Loading
---------------------------------------
Needing to revisit all the places where modules get deserialized made it clear that there is a lot of complexity and apparent duplication in the core routines on the `Linkage` that get used for loading modules.
This change tries to clean up some of that logic, but it is worth noting that there are two legacy features that get in the way of making things as clean as they should be:
* The `LoadedModuleDictionary` type that gets passed around a lot exists entirely to handle the corner case where somebody uses the Slang API to perform a compilation with multiple `TranslationUnitRequest`s in the same `FrontEndCompileRequest`, and one of the translation units `import`s the module defined by another of the translation units.
* There are a lot of special-case behaviors and routines entirely there to support the `ModuleLibrary` feature, although that feature should be considered deprecated (or at least subject to getting entirely re-designed down the line).
The basic idea of the cleanup is that all of the (non-deprecated) ways load a module from a serialized binary, or compile one from source should now bottleneck through `loadModuleImpl`, which then bifurcates into `loadSourceModuleImpl` for the compilation case and `loadBinaryModuleImpl` for the deserialization case.
High-Level Serialization Approach
---------------------------------
The old serialization logic used the [RIFF](https://en.wikipedia.org/wiki/Resource_Interchange_File_Format) format to encode the high-level structure of things, and this change retains that usage (and actually doubles down on the RIFF usage).
The old serialization system relied on the idea that for any given type `Foo` that wants to support serialization, there should be something like a `SerialFooData` type in C++, that can represent the state of a `Foo`, and then the actual serialization applied to that `SerialFooData`. This means that in most cases there are four pieces of code written:
* During serialization:
* Copying the data of a `Foo` in memory over to a `SerialFooData` in memory
* Writing the state of a `SerialFooData` into the serialized data stream
* During deserialization:
* Reading the state of a `SerialFooData` from a serialized data stream
* Copying the data of the `SerialFooData` in memory over to a `Foo`
The new logic gets rid of the intermediate `SerialFooData`.
In the serialization direction, we take a `Foo` and write it to the `RIFFContainer` directly, or using some other utilities layered on top of it.
In the deserialization direction, we have additional flexibility. Given a `RIFFContainer::Chunk*` that represents a serialized `Foo`, we often navigate through the in-memory representation of the RIFF data to get to the parts of the serialized value that we actually want/need, without needing to deserialize the entire `Foo`.
To support this kind of operation, this change introduces a few helper types like `ContainerChunkRef` an `ModuleChunkRef`, that are little more than typed wrappers around a `RIFFContainer::Chunk*`.
The Module "Container" Part
---------------------------
A serialized `Module` is encoded as a RIFF chunk, using logic in `slang-serialize-container.cpp` - both before and after this change.
This change reorganizes a lot of the code in that file, to account for the way that eliminating the intermediate `SerialContainerData` type streamlines the overall task of writing out the parts of the module.
In the deserialization logic... there isn't really much to do in `slang-serialize-container.cpp`. Most of the logic in `slang.cpp` and `slang-module-library.cpp` that pertains to deserializing modules uses the `ModuleChunkRef`-based approach, and simply extracts the pieces of the serialized module that it needs.
The Actual Serialization of the AST
-----------------------------------
The actual AST serialization logic is in `slang-serialize-ast.cpp`.
The basic approach in both the writing and reading directions is:
* Use the `FIDDLE TEMPLATE` system to generate a set of functions, one for each AST node type, that recursively invoke the read/write logic on each field of that node (after recursively invoking the case for its direct superclass)
* Use the `ASTNodeDispatcher` system to dispatch out to those functions whene reading or writing anything derived from `NodeBase`
* For now, handle all types *not* derived from `NodeBase` by hand.
There's a lot of room for improvement around that last item: it should be just as easy to generate the serialization and deserialization logic for other types that don't inherit from `NodeBase`, but the current change tries to err on the side of making the logic as explicit and simplistic as possible, rather than trying to get too clever too soon.
The actual serialization *format* used for the AST is almost comically simplistic: the code uses hierarchical RIFF chunks to emulate a JSON-like structure. This is a very wasteful representation (e.g., a `bool` or a null pointer each take up *8 bytes*), but the goal for now is to start with the simplest thing that could possibly work, and only add more cleverness once we are sure it won't get in the way of important future improvements (like lazy/on-demand deserialization or IR and AST, to improve compiler startup times).
The files `slang-serialize.{h,cpp}` have been co-opted to define a new pair of types `Encoder` and `Decoder` that are used for a more-or-less stream-oriented way or reading or writing RIFF chunks for the JSON-like structure.
Almost everything related to the actual AST serialization could do with a cleanup pass, and some time spent on picking good/better names for everything.
Smaller Stuff
-------------
* Cleaned up a lot of code that was using bare `ASTNodeType` or the extractor's `ReflectClassInfo` type to consistently use `SyntaxClass`.
* Fixed an apparent bug in how the destination-driven code genarator was handling `TryExpr`s
* Fixed an apparent bug in how the GLSL legalization pass was handling translation of certain `SV_*` semantics.
* format code
* fixup: template errors caught by non-VS compilers
* format code
* fixup: more template errors
* fixup: more stuff VS didn't catch
* fixup: it's amazing VS doesn't catch these...
* fixup: yet more template stuff VS ignores
* fixup: more VS template nonsense
* fixup: unreachable return macro usage
* fixup: more unreacable returns
* fixup: unused parameter
* fixup: strict aliasing
* fixup: allow missing entry point list chunk
* fixup: wasm build script
* fixup: AST changes since this PR was created
---------
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
| |
* Move switch statement bodies to their own lines
* format
---------
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
| |
* format
* Minor test fixes
* enable checking cpp format in ci
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Move the file public header files to `include` dir
Close the issue (#4635).
Move the following headers files to a `include` dir
located at root dir of slang repo:
slang-com-helper.h -> include/slang-com-helper.h
slang-com-ptr.h -> include/slang-com-ptr.h
slang-gfx.h -> include/slang-gfx.h
slang.h -> include/slang.h
Change cmake/SlangTarget.cmake to add include path to
every target, and change the source file to use
"#include <slang.h>" to include the public headers.
The source code update is by the script like follow:
```
fileNames_slang=$(grep -r "\".*slang\.h\"" source/ -l)
for fileName in "${fileNames_slang[@]}"
do
echo "$fileName"
sed -i "s/\".*slang\.h\"/\"slang\.h\"/" $fileName
done
```
* Fix the test issues
* Fix cpu test issues by adding include seach path
* Update cmake to not add include path for every target
Also change "#include <slang.h>" to "include "slang.h" " to
make the coding style consistent with other slang code.
* Change public include to private include for unit-test and slang-glslang
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Refactor Stream. Working on all tests.
* Split out CharEncode.
* Make method names lower camel.
m_prefix in Writer/Reader
* Tidy up around CharEncode interface.
* Small improvements around encode/decode.
* Better use of types.
* Remove readLine from TextReader.
* Remove exceptions from Stream/Text handling.
* Fix some typos.
* Fix tabbing.
* Fix missing override.
* Remove remaining exception throw/catch via using signal mechanism.
* Remove exceptions that are not used anymore.
* Document the Stream interface.
* Remove index for decoding 'get byte' function.
* Fix CharReader -> ByteReader.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Testing out use of lz4.
* Added ICompressionSystem, and LZ4 implementation.
* Add support for deflate compression.
Simplify compression interface - to make more easily work across apis.
* WIP on CompressedFileSystem.
* ImplicitDirectoryCollector
* SubStringIndexMap - > StringSliceIndexMap.
* WIP save stdlib in different containers.
* Support for different archive types for stdlib.
* Fix project.
* CompressedFileSystem -> ArchiveFileSystem.
Added CompressionSystemType::None
* Added ArchiveFileSystem
* Fix problem RiffFileSystem load withoug compression system.
* Test archive types.
Improve diagnostic message.
* Fix typo in testing file system archives.
* Split out archive detection.
* Fix gcc warning issue.
* Fix warning.
* RiffArchiveFileSystem -> RiffFileSystem
Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Test if blob is returned.
* Rename serialize files so can be grouped.
* StringRepresentationCache -> SerialStringTable
* Split out SerialStringTable from slang-serialize-ir
* First pass at reorganizing serialization/containers. Remain some issues about debug info.
* Fix bug in calculating sourceloc.
* Improve calcFixSourceLoc
* Make allocations for payload RiffContainer align to at least 8 bytes. This is important for read, if the payload can contain 8 byte aligned data. Note this has no effect on Riff file format alignment rules.
* Improve comments around RiffContainer and alignment.
* Remove SerialStringTable, can just use StringSlicePool instead.
* Add flags to control what is output in SerialContainer.
Turn off AST output for obfuscated code.
Lazily create astClasses when doing write container serialization.
* Typo fix for Clang/Linux.
* Fixes that came out of review
* TranslationUnit -> Module
* TargetModule -> TargetComponent
* PAYLOAD_MIN_ALIGNMENT -> kPayloadMinAlignment
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Test if blob is returned.
* Rename serialize files so can be grouped.
* StringRepresentationCache -> SerialStringTable
* Split out SerialStringTable from slang-serialize-ir
* First pass at reorganizing serialization/containers. Remain some issues about debug info.
* Fix bug in calculating sourceloc.
* Improve calcFixSourceLoc
* Make allocations for payload RiffContainer align to at least 8 bytes. This is important for read, if the payload can contain 8 byte aligned data. Note this has no effect on Riff file format alignment rules.
* Improve comments around RiffContainer and alignment.
* Remove SerialStringTable, can just use StringSlicePool instead.
* Typo fix for Clang/Linux.
Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fields from upper to lower case in slang-ast-decl.h
* Lower camel field names in slang-ast-stmt.h
* Fix fields in slang-ast-expr.h
* slang-ast-type.h make fields lowerCamel.
* slang-ast-base.h members functions lowerCamel.
* Method names in slang-ast-type.h to lowerCamel.
* GetCanonicalType -> getCanonicalType
* Substitute -> substitute
* Equals -> equals
ToString -> toString
* ParentDecl -> parentDecl
Members -> members
* * Make hash code types explicit
* Use HashCode as return type of GetHashCode
* Added conversion from double to int64_t
* Split Stable from other hash functions
* toHash32/64 to convert a HashCode to the other styles.
GetHashCode32/64 -> getHashCode32/64
GetStableHashCode32/64 -> getStableHashCode32/64
* Other Get/Stable/HashCode32/64 fixes
* GetHashCode -> getHashCode
* Equals -> equals
* CreateCanonicalType -> createCanonicalType
* Catches of polymorphic types should be through references otherwise slicing can occur.
* Fixes for newer verison of gcc.
Fix hashing problem on gcc for Dictionary.
* Another fix for GetHashPos
* Fix signed issue around GetHashPos
|
| |
|
|
|
|
|
|
|
|
|
| |
* * Added option to get random bytes from RandomGenerator
* Added ability to allocate only in current block on MemoryArena
* Allowed RiffContainer to not allocate new Data blocks, if can just extend the Data it has (because it's at the end of current block and there is space for the new data).
* Added coverage for change on Riff unit test
* Add test coverage for allocations over multiple Data blocks.
* Improve comment on riff unit tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Add basic support for entry points in `.slang-lib` files.
The basic idea here is that when writing out a `.slang-lib` file based on a compile request, we include new sections in the generated RIFF that represent the entry points that were requested. The entry-point information is serialized in an entirely ad hoc fashion (a future change might clean it up to use the `OffsetContainer` machinery), and contains the name, profile, and mangled symbol name of an entry point.
When deserializing this information, we create a list of "extra" entry points that gets attached to the front-end compile requests. These "extra" entry points get turned into `EntryPoint` objects at the same place in the code that entry points specified on the command line or via API would be checked, but the extra entry points bypass the semantic checking and just create "dummy" `EntryPoint` objects.
Aside: the ability for a compile request to end up with entry points that weren't originally specified via API or command-line is not new. We already had support for compiling a translation unit with entry points entirely specified via `[shader(...)]` attributes, and this new support tries to function similarly.
Because the "dummy" entry points don't retain AST-level information, several parts of the code have been modified to defensively check for `EntryPoint` objects without a matching AST declaration, and skip over them.
The main place where this creates a problem is paramete binding, where ignoring the dummy entry point is appropriate since we currently assume linked-in library code has been laid out manually.
One small cleanup here is that the `-r` command-line flag and the `spAddLibraryReference` API functio now bottleneck through a common routine to do their work, so that they both gain the new behavior without needing copy-paste programming.
In order to keep the existing test case for library linking with entry points working, I had to add a flag to the `render-test` tool so that it can skip specifying entry point names as part of the compile request it creates. In that case it must instead assume that the entry points will be added to the compile request via other means. This logic is a bit magical, and hints that we should be looking for other ways to expose the library linking functionality over time.
* fixup: remove alignment assertion
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Added RiffReadHelper
* Move type to fourCC in Chunk simplifies some code.
* Make MemoryArena able to track external blocks.
Allow ownership of Data to vary.
Changed IR serialization to use moved allocations to avoid copies.
As it turns out all of the array writes could use unowned data, but doing so requires the IRData to stay in scope longer than IRSerialData, which it does at the moment - but perhaps needs better naming or a control for the feature.
* Write out slang-module container.
* WIP on -r option.
Loading modules - with -r.
* Making the serialized-module run (without using imported module).
* Split compiling module from the test.
* Separate module compilation with a function working.
* Remove serialization test as not used.
* Fix warning on gcc.
* Updated test to have types across module boundary.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
* Added RiffReadHelper
* Move type to fourCC in Chunk simplifies some code.
* Make MemoryArena able to track external blocks.
Allow ownership of Data to vary.
Changed IR serialization to use moved allocations to avoid copies.
As it turns out all of the array writes could use unowned data, but doing so requires the IRData to stay in scope longer than IRSerialData, which it does at the moment - but perhaps needs better naming or a control for the feature.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* WIP RiffContainer.
* WIP riff container.
* Testing out RiffContainer.
* * Naming improvements
* Visitor functions
* Ability to dump riffs.
* Renamed RiffChunk to RiffHeader
* Remove m_ prefix on RiffHeader members.
* Riff stream reading writing.
Simple test of riff reading/writing.
* Fix Riff alignment issue.
Make IR serialization use the RiffContainer API.
* Improve documentation.
* Remove SubChunk fuctionality as not needed with RiffContainer.
|
| |
|
|
|
|
|
| |
* * Added semantic versioning and hashing test on repro data.
* Added RiffSemanticVersion type
* Fix linux build warning.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
* Added spEnableReproCapture to the API.
* Added MemoryStreamBase - which can be used to read from without copyin the data.
Added the missing Repro API functions - spEnableReproCapture and spExtractRepro.
Added support for serializing output filenames.
* Improved naming around Stream.
Brought Stream and sub types closer to code conventions.
* Renamed content -> contents in Stream.
|
|
|
* WIP on serialize/save state.
* Relative string encoding.
* Added RelativeContainer unit test.
Split out RelativeContainer into core.
* Fix bug in RelativeString encoding.
* More work around relative container.
* Fix checks.
* Use RelativeBase for safe access.
Use malloc/free/realloc instead of List.
* Add natvis support for relative types.
* Setting up of state (not includes) writing of repro state.
* Capture after spCompile.
* Writing SourceFile and file system files.
Added -dump-repo
* First pass at loading state.
* First pass at reading repro.
* Small optimization around Safe32Ptr
* Refactor how repro data is stored - to make saving off the files more simple, by having all all backed by 'files'.
Make file loading always set up PathInfo so we get uniqueIdentifier info.
* Generate unique file names.
* Added RelativeFileSystem
Added saveFile to ISlangFileSystemExt and implemented for interfaces
Added mechanism to save of files (and manifest)
* Added ability to replace files in repo with directory holding their contents.
* Add support for entry points.
* Fix problem compiling on linux.
* Added SIMPLE_EX option, where everything on command line must be specified.
* Fix typo in unit test for relative container.
* Fix another typo in unit test for RelativeContainer.
* Fix small bugs.
* Fix release unused variable issue in slang-state-serialize.cpp
* Fix checking for SIMPLE_EX in testing, else broke COMMAND_LINE_SIMPLE.
* Fix warnings on 32 bit debug build.
* Added import-subdir-search-path-repro.slang test. Although disabled for now as writes to root of slang project.
* Remove wrong version of import-subdir-search-path-repro.slang
* Added import-subdir-search-path-repro.slang
|
