summaryrefslogtreecommitdiff
path: root/source/slang/slang-serialize-container.cpp
AgeCommit message (Collapse)Author
2024-10-29formatEllie Hermaszewska
* format * Minor test fixes * enable checking cpp format in ci
2024-08-05Initial support for precompiled DXIL in slang-modules (#4755)cheneym2
* Add embedded precompiled binary IR ops Add IR operations to embed precompiled DXIL or SPIR-V blobs into IR. Adds a BlobLit literal that is mostly identical to StringLit except for its inability to be displayed, e.g. in dumped IR. In the future, the blob might be dumped as hexadecimal, but for now it is summarized as "<binary blob>". * EmbeddedDXIL and SPIR-V options The options, '-embed-dxil' and '-embed-spirv' in slangc, will cause a target dxil or spirv to be compiled and stored in the translation unit IR when written to a slang-module. Subsequent changes actually implement the options. * Per-translation unit DXIL precompilation When -embed-dxil is specified, perform a precompilation to DXIL of each TU, linked only with stdlib. Embed the resulting DXIL for the TU in a IR op. Being part of IR, the precompiled DXIL can be serialized to disk in a slang-module. Upon loading slang-modules, the new IR op will be searched for and the precompiled DXIL blob is saved with the loaded Module. During linking, if all the Modules have precompiled blobs they will be sent to the downstream compile commands as libraries instead of source, skipping the downstream compilation, using DXC only for linking. Fixes Issue #4580 * Remove placeholder embedded SPIRV support Code was added only to sketch out how other precompiled bins will be supported. * Remove the rest of the SPIRV placeholder support * Fix warnings, test error on non-windows * Remove lib_6_6 hack, add dxil_lib capability * Allocate blob value from irmodule memarena * Add null check after memarena allocation * Restore the request->e2erequest code path for generatewholeprogram * Update capability handling, move EmbedDXIL enum to end to preserve abi * Remove lib_6_6 hack * Move ICompileRequest functions to end
2024-04-11Fix the issues when compiling slang to library (#3936)kaizhangNV
2024-03-19Fix inconsistent digest of precompiled module. (#3796)Yong He
2024-03-18Fix name mangling and source file finding logic for precompiled module ↵Yong He
validation. (#3784) * Fix name mangling. * Fix source validation. * Caching and search path fixes.
2024-03-01warnings (#3653)Ellie Hermaszewska
* Remove unused code * unused variable warnings * unused variable --------- Co-authored-by: Yong He <yonghe@outlook.com>
2024-02-26Allow default values for `extern` symbols. (#3632)Yong He
* Allow default values for `extern` symbols. * Fix. * Fix test.
2024-02-23Add slangc interface to compile and use ir modules. (#3615)Yong He
* Add slangc interface to compile and use ir modules. * Fix glsl scalar layout settings not copied to target. * Fix. * Cleanups.
2024-02-22Add API for querying and reusing precompiled binary modules. (#3614)Yong He
2024-02-20Refactor compiler option representations. (#3598)Yong He
* Refactor compiler option representation. * Fix binary compatibility. * Add a test for specifying compiler options at link time. * Fix binary compatibility. * Fix binary compatibility. * Fix backward compatibility on matrix layout. * Fix. * Fix. * Fix. * Fix gfx. * Fix gfx. * Fix dynamic dispatch. * Polish.
2024-02-15Support loading serialized modules. (#3588)Yong He
* Support loading serialized modules. * Fix. * Fix vs solution files * Fix glsl module loading. * C++ fix. * Fix. * Try fix c++ error. * Try fix. * Fix. * Fix.
2023-09-26Fix for epoch/ASTBuilder* nullptr issue (#3240)jsmall-nvidia
* Fix issue with failing tests tests/serialization/serialized-module-test.slang tests/serialization/extern/extern-test.slang * Fix issue with session destruction order on Session. * Improve comment.
2023-08-09Clean up and improve Val deduplication performance. (#3069)Yong He
* Clean up and improve Val deuplication performance. * Fix. * Fix. * Fix. * Fix. --------- Co-authored-by: Yong He <yhe@nvidia.com>
2023-08-07Fix `Val` deduplication bug. (#3050)Yong He
* Fix `Val` deduplication bug. * Fix * Concat stdlib files into a single module. * Remove unnecessary logic in `resolve`. --------- Co-authored-by: Yong He <yhe@nvidia.com>
2023-08-04Redesign `DeclRef` and systematic `Val` deduplication (#3049)Yong He
* Redesign DeclRef + Deduplicate Val. * Update project files * Fix warning. * Fix. * Fix. * Remove `Val::_equalsImplOverride`. * Rmove `Val::_getHashCodeOverride`. * Remove `semanticVisitor` param from `resolve`. * Cleanups. --------- Co-authored-by: Yong He <yhe@nvidia.com>
2023-04-26Fix most of the disabled warnings on gcc/clang (#2839)Ellie Hermaszewska
2023-04-25StringBuilder to lowerCamel (#2840)jsmall-nvidia
* #include an absolute path didn't work - because paths were taken to always be relative. * WIP lowerCamel Dictionary. * WIP more lowerCamel fixes for Dictionary. * Add/Remove/Clear * GetValue/Contains * Fix tabs in dictionary. Count -> getCount * Fix fields with caps. * Key -> key Value -> value Use m_ for members where appropriate. Use lowerCamel in linked list. * Some small fixes/improvements to Dictionary. * Kick CI. * Small tidy on String. * Append -> append * ToString -> toString ProduceString -> produceString * Small fixes. * StringToXXX -> stringToXXX * Fix typo introduced by Append -> append. * Made intToAscii do reversal at the end. --------- Co-authored-by: Yong He <yonghe@outlook.com>
2023-04-25Dictionary using lowerCamel (#2835)jsmall-nvidia
* #include an absolute path didn't work - because paths were taken to always be relative. * WIP lowerCamel Dictionary. * WIP more lowerCamel fixes for Dictionary. * Add/Remove/Clear * GetValue/Contains * Fix tabs in dictionary. Count -> getCount * Fix fields with caps. * Key -> key Value -> value Use m_ for members where appropriate. Use lowerCamel in linked list. * Some small fixes/improvements to Dictionary. * Kick CI.
2022-06-13Language Server improvements. (#2269)Yong He
* Language Server improvements. - Improve parser robustness around `attribute_syntax`. - Exclude instance members in a static query. - Coloring accessors - Improved signature help cursor range check. * Add expected test result. * Language server: support configuring predefined macros. * Fix constructor highlighting. * Improving performance by supporting incremental text change notifications. * Fix UTF16 positions and highlighting of constructor calls. * Add completion suggestions for HLSL semantics. * Fix tests. * Fix: don't skip static variables in a static query. * Include literal init expr value in hover text. * Fix scenarios where completion failed to trigger. * Fixing language server protocol field initializations. Co-authored-by: Yong He <yhe@nvidia.com>
2021-01-05Use "capability" system to select VKRT extension (#1647)Tim Foley
* Use "capability" system to select VKRT extension Slang currently supports translation of ray tracing shader code to Vulkan GLSL code that uses the `GL_NV_ray_tracing` extension. A multi-vendor equivalent of that extension has been released as `GL_EXT_ray_tracing` and we want Slang to support that extension as well. At the simplest, making the change from one extension to the other is just a matter of changing a few strings, since it does not appear that anything of significance was changed at the GLSL level (or even in SPIR-V). Where this gets trickier is when we have users who want us to support *both* extensions, and to be able to switch between them. The solution we've implemented here more or less amounts to: * If you don't tell the compiler which extension to use, it will default to `GL_EXT_ray_tracing` (the newer multi-vendor one). * If you explicitly want the older extension, you can opt into it using the `-profile` option or via a new API for explicitly adding capabilities to your target. Making that work required a few different kinds of changes: * The options parsing and public API needed ways to add optional capabilities to a target. * During GLSL code emit, we can check the capabilities that were added to the target to see if the `GL_NV_ray_tracing` extension was explicitly enabled and, if not, default to using the `GL_EXT_ray_tracing` names for things. This step is needed because some of the modifiers/attributes involved in the extension have to be handled explicitly in the code generator rather than implicitly as part of mapping intrinsic functions. * We add two different translations to the relevant operatiosn in the stdlib, one marked with each of the extensions. If profile/capability-based overload resolution can be relied on to pick the right one, this should Just Work. * Next, a bunch of work had to go into making capability-based overloading Just Work for the purposes of this change. There's been a nearly complete reworking of the implementation of `CapabilitySet` here to make it more suitable for our needs. * The tests that were using ray tracing translation for Vulkan needed to be updated. For some of them I updated their baselines to use `GL_EXT_ray_tracing` so that they can test the new path. For others, I updated the command line for the test case so that it explicitly opts into using `GL_NV_ray_tracing`. The result is that we have some coverage of each extension. I would have liked to have each test run in both modes, but our pass-through glslang support doesn't support `-D` options, so I couldn't take that step easily. This change does *not* add support for `GL_EXT_ray_query`, the extension that supports "DXR 1.1" style queries under Vulkan. Adding support for that extension should hopefully be a smaller step because it doesn't have the same multiple-extensions issue. This change does *not* address a lot of possible avenues for improvement or cleanup around the capability system. It focuses only on those changes that are necessary to make the ray tracing feature work and leaves the rest for future work. * fixup: infinite loop * Comment-only change to retrigger TC build
2020-11-18Serialized stdlib working (#1603)jsmall-nvidia
* #include an absolute path didn't work - because paths were taken to always be relative. * Mangling/module name extraction for GenericDecl * Add comment on SerialFilter to explain re-enabling Stmt. * Support setting up SyntaxDecl when reconstructed after deserialization. * Improvements to setup SyntaxDecl. * Fix typo so can read compressed SourceLocs. * Fix issue with SourceManger.
2020-11-05Standard library save/loadable (#1592)jsmall-nvidia
* #include an absolute path didn't work - because paths were taken to always be relative. * Fix handling of access modifiers inside type definition. * Fix access problem for AST node. Make dumping produce a single function with switch, to potentially make available without Dump specific access. * WIP on serialization design doc. * Remove project references to previously generated files. * More docs on serialization design. * Improve serialization documentation. Remove unused function from IRSerialReader. * Small fixes around naming. Remove long comment from slang-serialize.h - as covered in serialization.md * Remove long comment in slang-serialize.h as covered in serialization.md * More information about doing replacements on read for AST and problems surrounding. * Typo fix. * Spelling fixes. * Value serialize. * Value types with inheritence. * Use value reflection serial conversion for more AST types * Use automatic serialization on more of AST. * Get the types via decltype, simplifies what the extractor has to do. * Update the serialization.md for the value serialization. * Small doc improvements. * Update project. * Remove ImportExternalDecl type Added addImportSymbol and ImportSymbol type Fixed bug in container which meant it wouldn't read back AST module * Because of change of how imports and handled, store objects as SerialPointers. * First pass symbol lookup from mangled names. * Cache current module looked up from mangled name. * Fix SourceLoc bug. Improve comments. * Added diagnostic on mangled symbol not being found * Fix typo. * WIP serializing stdlib. * WIP serializing stdlib in. * Fix problem serializing arrays that hold data that is already serialized. * Remove clash of names in MagicTypeModifier. * Make conversion from char to String explicit. Fix reference count issue with SerialReader. * Add code to save/load stdlib. * Use return code to avoid warning - SerialContainerUtil::write(module, options, &stream)) * Make all String numeric ctors explicit. Added isChar to UnownedStringSlice. Added operator== for UnownedStringSlice to String to avoid need to convert to String and allocate. * Add error check to readAllText. * tabs -> spaces on String.h * tab -> spaces String.cpp * Remove msg for StringBuilder, just build inplace for exceptions. * Check SerialClasses - for name clashes. Renamed Modifier::name as Modifier::keywordName * Handling of extensions when deserializing AST - updating the moduleDecl->mapTypeToCandidateExtensions Co-authored-by: Tim Foley <tim.foley.is@gmail.com>
2020-10-29Handling imported/exporting symbols from serialized modules (#1589)jsmall-nvidia
* #include an absolute path didn't work - because paths were taken to always be relative. * Fix handling of access modifiers inside type definition. * Fix access problem for AST node. Make dumping produce a single function with switch, to potentially make available without Dump specific access. * WIP on serialization design doc. * Remove project references to previously generated files. * More docs on serialization design. * Improve serialization documentation. Remove unused function from IRSerialReader. * Small fixes around naming. Remove long comment from slang-serialize.h - as covered in serialization.md * Remove long comment in slang-serialize.h as covered in serialization.md * More information about doing replacements on read for AST and problems surrounding. * Typo fix. * Spelling fixes. * Value serialize. * Value types with inheritence. * Use value reflection serial conversion for more AST types * Use automatic serialization on more of AST. * Get the types via decltype, simplifies what the extractor has to do. * Update the serialization.md for the value serialization. * Small doc improvements. * Update project. * Remove ImportExternalDecl type Added addImportSymbol and ImportSymbol type Fixed bug in container which meant it wouldn't read back AST module * Because of change of how imports and handled, store objects as SerialPointers. * First pass symbol lookup from mangled names. * Cache current module looked up from mangled name. * Fix SourceLoc bug. Improve comments. * Added diagnostic on mangled symbol not being found * Fix typo. Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>
2020-10-06Use Reflection for (Serial)RefObject Serialization (#1567)jsmall-nvidia
* First pass at generalizing serializer. * Split out ReflectClassInfo * Use the general ReflectClassInfo * Fix some typos in debug generalized serialization. * Add calculation of classIds. Make distinct addCopy/add on SerialClasses. * Write up of more generalized serialization * WIP to transition from ASTSerialReader/Writer etc to generalized SerialReader/Writer and associated types. * Improvements to SerialExtraObjects. Keep RefObjects in scope in factory * Compiles with Serial refactor - doesn't quite work yet. * First pass serialization appears to work with refector. * Split out type info for general slang types. * Split out slang-serialize-misc-type-info.h * DebugSerialData -> SerialSourecLocData DebugSerialReader -> SerialSourceLocReader DebugSerialWriter -> SerialSourceLocWriter * Remove unused template that only compiles on VS. * Fix warning around unused function on non-VS. * Improve output of type names that are in scopes in C++ extractor. Update premake5.lua to run generation for RefObject derived types. * C++ extractor working on RefObject type. * Split out serialization functionality that spans different types into slang-serialization-factory.cpp/.h Put AST type info into header. Removed RefObjectSerialSubType - use RefObjectType Add filtering for RefObject derived types Remove construction and filteringhacks. * Set up field serialization for SerialRefObject derived types. * Fix template problem compiling on Clang/Gcc * Work in progress to make Value types work. * Added slang-value-reflect.cpp
2020-09-30Generalizing Serialization (#1563)jsmall-nvidia
* First pass at generalizing serializer. * Split out ReflectClassInfo * Use the general ReflectClassInfo * Fix some typos in debug generalized serialization. * Add calculation of classIds. Make distinct addCopy/add on SerialClasses. * Write up of more generalized serialization * WIP to transition from ASTSerialReader/Writer etc to generalized SerialReader/Writer and associated types. * Improvements to SerialExtraObjects. Keep RefObjects in scope in factory * Compiles with Serial refactor - doesn't quite work yet. * First pass serialization appears to work with refector. * Split out type info for general slang types. * Split out slang-serialize-misc-type-info.h * DebugSerialData -> SerialSourecLocData DebugSerialReader -> SerialSourceLocReader DebugSerialWriter -> SerialSourceLocWriter * Remove unused template that only compiles on VS. * Fix warning around unused function on non-VS.
2020-09-18Serialization fixes based on review of #1547 (#1551)jsmall-nvidia
* Test if blob is returned. * Rename serialize files so can be grouped. * StringRepresentationCache -> SerialStringTable * Split out SerialStringTable from slang-serialize-ir * First pass at reorganizing serialization/containers. Remain some issues about debug info. * Fix bug in calculating sourceloc. * Improve calcFixSourceLoc * Make allocations for payload RiffContainer align to at least 8 bytes. This is important for read, if the payload can contain 8 byte aligned data. Note this has no effect on Riff file format alignment rules. * Improve comments around RiffContainer and alignment. * Remove SerialStringTable, can just use StringSlicePool instead. * Add flags to control what is output in SerialContainer. Turn off AST output for obfuscated code. Lazily create astClasses when doing write container serialization. * Typo fix for Clang/Linux. * Fixes that came out of review * TranslationUnit -> Module * TargetModule -> TargetComponent * PAYLOAD_MIN_ALIGNMENT -> kPayloadMinAlignment
2020-09-18Control container serialization with SerialOptionFlags (#1550)jsmall-nvidia
* Test if blob is returned. * Rename serialize files so can be grouped. * StringRepresentationCache -> SerialStringTable * Split out SerialStringTable from slang-serialize-ir * First pass at reorganizing serialization/containers. Remain some issues about debug info. * Fix bug in calculating sourceloc. * Improve calcFixSourceLoc * Make allocations for payload RiffContainer align to at least 8 bytes. This is important for read, if the payload can contain 8 byte aligned data. Note this has no effect on Riff file format alignment rules. * Improve comments around RiffContainer and alignment. * Remove SerialStringTable, can just use StringSlicePool instead. * Add flags to control what is output in SerialContainer. Turn off AST output for obfuscated code. Lazily create astClasses when doing write container serialization. * Typo fix for Clang/Linux.
2020-09-17Share debug information between AST and IR (#1547)jsmall-nvidia
* Test if blob is returned. * Rename serialize files so can be grouped. * StringRepresentationCache -> SerialStringTable * Split out SerialStringTable from slang-serialize-ir * First pass at reorganizing serialization/containers. Remain some issues about debug info. * Fix bug in calculating sourceloc. * Improve calcFixSourceLoc * Make allocations for payload RiffContainer align to at least 8 bytes. This is important for read, if the payload can contain 8 byte aligned data. Note this has no effect on Riff file format alignment rules. * Improve comments around RiffContainer and alignment. * Remove SerialStringTable, can just use StringSlicePool instead. * Typo fix for Clang/Linux. Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>