| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Initial plan
* Fix pragma warning not working with multifile modules
- Check if DiagnosticSink already has a WarningStateTracker before creating new one
- This preserves pragma warning state across __include'd files
- Add regression tests for multifile pragma warnings
Co-authored-by: csyonghe <2652293+csyonghe@users.noreply.github.com>
* Add additional test cases for nested pragma warnings
- Test nested __include scenarios with pragma warning directives
- Verify pragma warnings work correctly with multiple levels of includes
Co-authored-by: csyonghe <2652293+csyonghe@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: csyonghe <2652293+csyonghe@users.noreply.github.com>
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Language version + tuple syntax.
* Fix compile error.
* regenerate documentation Table of Contents
* Fix.
* regenerate command line reference
* Fix.
* Fix.
* Fix more test failures.
* revert empty line change,
* Retrigger CI
* #version->#lang
* Update source/core/slang-type-text-util.cpp
Co-authored-by: ArielG-NV <159081215+ArielG-NV@users.noreply.github.com>
* Remove comments.
* Fix parsing logic.
* Fix parser.
* Fix parser.
* update test comment
* Update options.
* regenerate documentation Table of Contents
* regenerate command line reference
---------
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>
Co-authored-by: ArielG-NV <159081215+ArielG-NV@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Implemented #pragma warning
Based on https://learn.microsoft.com/en-us/cpp/preprocessor/warning?view=msvc-170
* Make #pragma warning work with #includes.
- SourceLoc are not sorted by inclusion order.
- Construct a mapping from SourceLoc to "absolute locations" that are sorted by inclusion order (roughly represents a location in a raw file with all #include resolved).
- The absolute location can be used in the pragma warning timeline
* Added preprocessor #pragma warning tests.
- Fixed #pragma warning (push / pop) SourceLoc
- Fixed unused directiveLoc in #pragma warning parsing
* #pragma warning: Added some comments and fixed some typos
* Cleaned #pragma warning preprocessor implementation.
---------
Co-authored-by: Yong He <yonghe@outlook.com>
|
| | |
|
| |
|
|
|
|
|
|
|
| |
* Move switch statement bodies to their own lines
* format
---------
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
| |
* format
* Minor test fixes
* enable checking cpp format in ci
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Squash redundant move warnings
* Move C interface from slang.h to slang-deprecated.h
spGetBuildTagString remains, because it's useful to have before the
global session exists.
This C API is used quite pervasively in the C++ helpers (for example
slang::UserAttribute. It's not trivial to move these to
slang-deprecated.h as they're entangled with some enums which are
themselves used elsewhere in the compiler.
The fact that these helpers use the C API can be viewed as an
implementation detail for now, and this usage moved to slang-deprecated
in due course.
Closes https://github.com/shader-slang/slang/issues/4758
* Squash warnings for our usage of our deprecated API
---------
Co-authored-by: Yong He <yonghe@outlook.com>
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
* Unify GLSL and HLSL buffer block parsing.
Automatic GLSL module recognition.
* Fix.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Parse glsl buffer blocks to GLSLInterfaceBlockDecl
* Parse glsl local size layout declarations
* Parse (and ignore) glsl version directives
* spelling
* Better l-value interpretation for glsl interface blocks
* Better l-value interpretation for glsl interface blocks
* Add compile flag for enabling glsl
* Parse and ignore precision modifiers.
* Automatically import `glsl` module for compatiblity.
* Complete vector and matrix types for glsl
* Remove generated file from repo
* Bump .gitignore
* do not mark out globals as params
* Synthesize entrypoint layout from global inout vars.
* update test result.
* Allow HLSL semantic on global variables.
* Fix.
* Fix test.
* Fix win32 compile error.
* Add more builtin input/output and texture intrinsics.
* Add struct/array constructor syntax.
* Skip `#extension` lines.
* overide operator * for matrix/vector multiplication.
* Add `matrixCompMult`.
* Parse modifiers in for loop init var declr.
* Add more glsl intrinsics, add stage into to var layout.
* Allow `int[3] x` syntax.
* Fix array type syntax.
---------
Co-authored-by: Ellie Hermaszewska <ellieh@nvidia.com>
Co-authored-by: Yong He <yhe@nvidia.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Correct namespace for getClockFrequency
* missing const
* Add missing assignment operator
* Remove unused variables
* Return correct modified variable
* Use stable hash code for file system identity
* terse static_assert
* Structured binding for map iteration
* Make (==) and getHashCode const on many structs
* Add ConstIterator for LinkedList
* Replace uses of ItemProxy::getValue with Dictionary::at
* Extract list of loads from gradientsMap before updating it
* Const correctness in type layout
* Add unordered_dense hashmap submodule
* Use wyhash or getHashCode in slang-hash.h
* refactor slang-hash.h
* Use ankerl/unordered_dense as a hashmap implementation
Notable changes:
- The subscript operator returns a reference directly to the value,
rather than a lazy ItemProxy (pair of dict pointer and key)
slang-profile time (95% over 10 runs):
- Before: 6.3913906 (±0.0746)
- After: 5.9276123 (±0.0964)
* 64 bit hash for strings
So they have the same hash as char buffers with the same contents
* Narrowing warnings for gcc to match msvc
* revert back to c++17
* Correct c++ version for msvc
* Use path to unordered_dense which keeps tests happy
* Do not assign to and read from map in same expression
* Remove redundant map operations in primal-hoist
* Split out stable hash functions into slang-stable-hash.h
* 64 bit hash by default
* regenerate vs projects
* Correct return type from HashSetBase::getCount()
* correct width for call to Dictionary::reserve
* Use stable hash for obfuscated module ids
* Signed int for reserve
* clearer variable naming
* Parameterize Dictionary on hash and equality functors
* Allow heterogenous lookup for Dictionary
* missing const
* Use set over operator[] in some places
* Remove unused function
* s/at/getValue
|
| |
|
|
|
|
|
|
|
|
|
| |
* A more way robust way to handle resource consumption might use multiple `kind`s on GLSL emit.
* Improve method naming and some comments.
* Small consistency fix.
* Fix issue with #elif evaluation.
* Add a test.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* WIP lowerCamel Dictionary.
* WIP more lowerCamel fixes for Dictionary.
* Add/Remove/Clear
* GetValue/Contains
* Fix tabs in dictionary.
Count -> getCount
* Fix fields with caps.
* Key -> key
Value -> value
Use m_ for members where appropriate.
Use lowerCamel in linked list.
* Some small fixes/improvements to Dictionary.
* Kick CI.
* Small tidy on String.
* Append -> append
* ToString -> toString
ProduceString -> produceString
* Small fixes.
* StringToXXX -> stringToXXX
* Fix typo introduced by Append -> append.
* Made intToAscii do reversal at the end.
---------
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* WIP lowerCamel Dictionary.
* WIP more lowerCamel fixes for Dictionary.
* Add/Remove/Clear
* GetValue/Contains
* Fix tabs in dictionary.
Count -> getCount
* Fix fields with caps.
* Key -> key
Value -> value
Use m_ for members where appropriate.
Use lowerCamel in linked list.
* Some small fixes/improvements to Dictionary.
* Kick CI.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix a bug in Path::find
* Fix code formatting
* Fix LockFile and add LockFileGuard
* Add PersistentCache and unit test
* Replace file path dependency list with source file dependency list
* Add note on ordering in Module/FileDependencyList
* Remove old shader cache code
* Refactor shader cache implementation
* Temporarily skip unit tests reading/writing files
* Fix warning
* Reenable lock file test
* Rename shader cache tests and disable crashing test
* Testing
* Stop using Path::getCanonical
* Fix persistent cache lock and test
* Fix threading issues
* Move adding file dependency hashes to getEntryPointHash()
* Fix handling of #include files
* Allow specifying additional search paths for gfx testing device
* Work on shader cache tests
* Update project files
* Revive shader cache graphics tests
* Split graphics pipeline test
* Fix compilation
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Draft FileStream-based implementation for updating cache file
* File streams fully integrated into shader cache code paths; Tests will not run unless file system is on disk as file streams do not play nicely with in-memory
* Brought old code back as fallback path, but tests need to ensure previous is freed first
* Testing structure updated, beginning cleanup work
* All tests working
* Cleanup changes
* Removed an extra tab at the end of a line
* Cleanup change
* Undo externals change
* Removed redundant logic for OS vs memory file system handling of the shader cache; Removed extra helper function left over from old cache implementation
* Reverted performance change to generate contents hashes when modules are being loaded as this code path is not always followed; Contents hashing now uses a combination of hashing and checking the last modified time for all file dependencies, only reading in and hashing the contents of all files if the last modified hash does not match
* Added handling to Module::updateContentsBasedHash for file dependencies which are not from a physical source file on disk; Added test for above
Co-authored-by: Lucy Chen <lucchen@nvidia.com>
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Preliminary graphics shader test
* Added test checking that a graphics shader is correctly split into two different entries
* Removed testing only addition
* Changed RequirementDictionary to an OrderedDictionary and added SerialTypeInfo for OrderedDictionary; Added entry point mangled name to the dependency hash
* Added test covering failure case discovered as part of Falcor integration
* Changed DifferentiableTypeSemanticContext::m_mapTypeToIDifferentiableWitness to an OrderedDictionary
* Added serializedAST field to Module in order to save serialized ASTs to avoid reserialization as much as possible; Added classes field to Session in order to save the output of SerialClassesUtil::create to avoid recreating as much as possible
* Changed AST hashing to hash the contents of a Module's file dependencies; Renamed all references to AST hashing to contents hashing
* Further cleanup
* Moved contents hash computation up to Linkage::loadModule and added field to Module to save the computed contents digest
* Changed PreprocessorHandler::handleFileDependency to optionally take an ISlangBlob* containing file contents and changed FrontEndPreprocessorHandler::handleFileDependency to add the source code for an included file to the module's contents digest
* Removed extraneous addToDigest call
* Fixed accidental removal of source code hash for module being loaded
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Make DownstreamCompileOptions use POD types.
* CharSliceAllocator -> SliceAllocator
Added SliceConverter
CharSliceCaster -> SliceCaster
* First attempt at zero terminating around blobs.
* Fix clang warning.
* Add SlangTerminatedChars
Make Blob implementations support it.
Make most blobs 'terminated'.
* Fix bug setting up sourceFiles for CommandLineDownstreamCompiler.
* Traffic in TerminatedCharSlice for sourceFiles.
Use ArtifactDesc to generate temporary file names for source.
* Fix typo in testing for shared library/C++.
* Working with source being passed as artifacts to DownstreamCompiler.
* Use artifacts in SourceManager/SourceFile.
* Support infering extension from the original file extension.
* * Infer extension if can't determine from the artifact type
* Split IOSFile/IExtFile representations
* Move responsibility for creating OS file to the handler.
* Disable the check memory path.
* Remove artifact from SourceFile.
Lazily generate SourceFile from artifacts as needed.
* Fix some small bugs.
* Remove maybeAddArtifact.
* Load artifacts if repro capture is enabled.
* Remove adding by string, because doing so means source will be allocated twice or there is a potential race around ref counting to the contained String.
* Add built in source as a blob.
* Fix warning.
* Make StringBlob own the contents if moved.
Fix some compilation issues.
* Share StringBlob uniqueness code.
* Do move unique on Ctor.
* Change MoveUnique to not have any values.
* MoveUnique can more sensibly be a struct.
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Make DownstreamCompileOptions use POD types.
* CharSliceAllocator -> SliceAllocator
Added SliceConverter
CharSliceCaster -> SliceCaster
* First attempt at zero terminating around blobs.
* Fix clang warning.
* Add SlangTerminatedChars
Make Blob implementations support it.
Make most blobs 'terminated'.
* Fix bug setting up sourceFiles for CommandLineDownstreamCompiler.
* Traffic in TerminatedCharSlice for sourceFiles.
Use ArtifactDesc to generate temporary file names for source.
* Fix typo in testing for shared library/C++.
* Working with source being passed as artifacts to DownstreamCompiler.
* Use artifacts in SourceManager/SourceFile.
* Support infering extension from the original file extension.
* * Infer extension if can't determine from the artifact type
* Split IOSFile/IExtFile representations
* Move responsibility for creating OS file to the handler.
* Disable the check memory path.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Make DownstreamCompileOptions use POD types.
* CharSliceAllocator -> SliceAllocator
Added SliceConverter
CharSliceCaster -> SliceCaster
* First attempt at zero terminating around blobs.
* Fix clang warning.
* Add SlangTerminatedChars
Make Blob implementations support it.
Make most blobs 'terminated'.
* Fix bug setting up sourceFiles for CommandLineDownstreamCompiler.
* Traffic in TerminatedCharSlice for sourceFiles.
Use ArtifactDesc to generate temporary file names for source.
* Fix typo in testing for shared library/C++.
* Working with source being passed as artifacts to DownstreamCompiler.
* Use artifacts in SourceManager/SourceFile.
* Support infering extension from the original file extension.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change adds support for variadic macros in the C-style
preprocessor, e.g.:
#define DEBUG(MSG, ...) print(__FILE__, __LINE__, MSG, __VA_ARGS__)
Similar to the gcc preprocessor, this feature supports both named
variadic macro parameters and unnamed ones (which then default to
`__VA_ARGS__`.
The implementation work is mostly straightforward, although there are a
few subtle design choices worth mentioning:
* A variadic macro is represented by it having a variadic *parameter*
that is part of the ordinary parameter list.
* Argument parsing does *not* detect whether the macro being invoked is
variadic and collect/combine arguments to form a single argument value
for the variadic parameter. This is motivated by the need for some
extensions to differentiate a variadic parameter receiving a single
empty argument vs. zero arguments.
* Because any reference to the variadic parameter needs to expand to the
comma-separated arguments that match it, the logic for turning a macro
parameter reference into a list of tokens has been factored out into a
subroutine that handles the details.
* The choice in the earlier refactor to have a macro invocation collect
all the argument tokens (including the intervening commas) into a
single token list seems to pay off here, because it means that the
tokens in the expansion of a variadic parameter reference were already
stored contiguously.
* The special-case logic for handling an empty argument list had to be
tweaked again to ensure that an empty argument list is treated as
having zero arguments for the variadic parameter. Note that
historically C did not define the behavior of this case, and always
required at least one argument for any variadic macro parameter.
* The logic for checking whether the number of arguments to a macro
invocation is valid needed to handle variadic and non-variadic macros
as distinct cases. There really isn't much overlap in how the checks
need to work, even if we tried to change the underling representation.
The main missing feature here is any way to discard a comma in a macro
body that appears before a variadic parameter reference, e.g.:
#define DEBUG(...) print("debug:", __VA_ARGS__)
In this case, an empty invocation list `DEBUG()` will expand to
`print("debug:",)` - a call with a trailing comma in the argument list.
If users end up needing a way to discard commas in cases like this, we
have many options we can consider. This change does not implement any of
them to keep the initial work as minimal as possible.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix a bug in preprocessor "busy" logic
This bug manifested in both incorrect preprocessor output for certain complicated cases, and also (more importantly) a use-after-free issue.
One of the "clever" design choices in the Slang preprocessor implementation is that the set of "busy" macros during expansion is implicitly defined by a linked list of those invocations that are actively being read from as part of the input stack. This logic works very well for checking whether a macro name is busy before triggering expansion, and for computing what macros should be considered busy during expansion of an object-like macro.
The problem case here was with function-like macros where the preprocessor was re-using the same list of busy macros that had been fetched when reading the macro name, but doing so *after* all the macro argument tokens had been read. Because additional tokens had been read from the input stream stack, there was no guarantee that the invocations that had been active before were still live.
The new logic computes the set of busy macros fresh before starting expansion of a function-like macro invocation. A test is included to ensure that the case that showed the use-after-free bug has been fixed.
In addition, the new logic is careful to compute the busy macros only based on where subsequent tokens will come from and not based on any macros that might have contributed to the argument tokens themselves. A test case has been added that relies on getting this detail correct.
* Update slang-preprocessor.cpp
Remove a test typo.
Co-authored-by: jsmall-nvidia <jsmall@nvidia.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Overhaul the preprocessor
The old Slang preprocessor was based on a simple mental model that tried to unify two parts of macro expansion:
* Scanning for macro invocations in a sequence of tokens
* Producing the expanded tokens for a macro expansion by substituting arguments into its body
The basic was that substitution of macro arguments into a macro definition is superficially similar to top-level macro expansion, just with an environment where the macro arguments act like `#define`s for the corresponding parameter names. That approach was "clever" and could conceivably have been extended to include a lot of advanced preprocessor features (e.g., a preprocessor-level `lambda` would be easy to support!), but it was basically impossible to make it correctly handle all the corner cases of the full C/C++ preprocessor.
The fundamental problem with the old approach was that it conflated the two parts of expansion listed above into one implementation, while the various special cases of the C/C++ preprocessor rely on treating the two cases very differently. The new approach here (which is somewhere between a refactor and a full rewrite of the preprocessor) changes things up in a few key ways:
* The abstraction still cares a lot about streams of tokens, but it now treats the top level streams (`InputFile`s) as fairly different from the lower-level streams (`InputStream`s)
* Macro expansion is handled as a dedicated type of stream that wraps another stream. This allows macro expansion to be applied to anything, and supports cases where multiple rounds of macro expansion are required by the spec.
* Macro *invocations* and the substitution of their arguments are now handled by a completely new system.
* Macro arguments are no longer treated as if they were `#define`s
* The macro body/definition is analyzed at definition time to detect various kinds of issues, and to derive a list of "ops" that make it easier to "play back" the definition at substitution time
* Token pasting and stringizing are now only handled in macro definitions (rather than being allowed anywhere), and their use cases are restricted to only those that make sense (e.g., you can't stringize anythign except a macro parameter, because anything else wouldn't make sense)
The key new types here are the `ExpansionInputStream` which handles scanning for macro invocations, and the `MacroInvocation` type, which handles playing back the macro body with substitutions.
The `ExpansionInputStream` is the easier of the two to understand. By refactoring it to use a single token of lookahead, the one major detail it had to deal with before (abandoning expansion of a function-like macro if the macro name was not followed by `(`) is significantly easier to manage.
The more subtle part is the `MacroInvocation` type, and most of the complexity there is around handling of token pasting, and the fact that either or both of the operands to a token paste might be empty.
Many of the test cases that exposed the problems in the preprocessor have been moved from `current-bugs` to `preprocessor` since they now work correctly.
* debugging: enable extractor command line dump
* fixup
* fixup
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Split out StringEscapeUtil.
* Added StringEscapeUtil.
* Fix typo in unix quoting type.
* Small comment improvements.
* Try to fix linux linking issue.
* Fix typo.
* Attempt to fix linux link issue.
* Update VS proj even though nothing really changed.
* Fix another typo issue.
* Fix for windows issue.
Fixed bug.
* Make separate Utils for escaping.
* Fix typo.
* Split out into StringEscapeHandler.
* Windows shell does handle removing quotes (so remove code to remove them).
* Handle unescaping if not initiating using the shell.
* Slight improvement around shell like decoding.
* Simplify command extraction.
* Add shared-library category type.
* Fix bug in command extraction.
* Typo in transcendental category.
* Enable unit-test on in smoke test category.
* Make parsing failing output as a failing test.
* Fixes for transcendental tests. Disable tests that do not work.
* Changed category parsing.
* Removed the TestResult parameter from _gatherTestsForFile.
Made testsList only output.
* Remove testing if all tests were disabled.
* Fix typo.
* Disable path canonical test on linux because CI issue.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Split out compiler-core initially with just slang-source-loc.cpp
* More lexer, name, token to compiler-core.
* Split Lexer and Core diagnostics.
* Move slang-file-system to core.
* Add slang-file-system to core.
* More DownstreamCompiler into compiler-core
* Fix typo.
* Add compiler-core to bootstrap proj.
* Small fixes to premake
* For linux try with compiler-core
* Remove compiler-core from examples.
* Added NameConventionUtil to compiler-core
* Add global function to CharUtil to *hopefully* avoid linking issue.
* Hack to make linkage of CharUtil work on linux.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* First pass support for __LINE__ and __FILE__.
* Test include handling with __FILE__
Fix diagnostic compare when input is empty.
* Fix some issues in preprocessor handling of special macros like __LINE__
Add a more complex test.
* Use CONCAT2 in tests, because preprocessor doesn't quite get parameter expansion correct.
* Make __FILE__ and __LINE__ behave more like Clang/Gcc.
* A test for preprocessor bug.
* Fix __LINE__ and __FILE__ in macro expansion, should be initiating location.
* Fix some comments.
* Small tidy up around builtin macros.
* Small improvements for macro type names.
Escape found paths.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The underlying problem here requires that we have an object-like macro with an expansion that starts with a non-identifier token:
```
```
Then we need a function-like macro that uses a token paste in a way that can expand to that object-like macro:
```
```
Finally, for the specific case a user ran into, we need to invoke that function-like macro in the context of a preprocessor conditional expression:
```
// ...
#error "unimplemented"
```
The way a problem manifest is that the preprocessor logic that handles conditional expressions tries to "peek" one token ahead and see what is coming, and while the peeking logic handles macro expansion it does *not* handle token pasting right now. That means that the peek operation sees `MY_FEATURE` and assumes that it is seeing an identifier in a preprocessor conditional that doesn't have a macro expansion. The logic then goes on to read the token, but what it gets back is *not* an identifier, and is instead the numeric literal token `1`, because the reading logic handles token pasting.
The quick fix I applied here is to make the logic that deals with preprocessor conditionals go ahead and automatically consume a token from the input, and then decide what to do based on that token, so that it always makes use of the reading logic that handles token pasting.
The lingering problem is that we still have cases in the preprocessor that use the peeking logic which doesn't handle pasting, and we might find that those cases have reason to want the same kind of expansion behavior we needed here. A more systematic fix would be to have the peeking logic automatically handle token pasting as well as macro expansion, but doing so would be a more complicated change because detecting the `##` when peeking ahead requires two tokens of lookahead, and our current implementation only assumes we can support one.
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* #include an absolute path didn't work - because paths were taken to always be relative.
* Improve diagnostic for token pasting.
* Token paste location test.
* Output include hierarchy.
* WIP on includes hierarchy.
* Improved include hierarchy output - to handle source files without tokens.
Improved test case.
* Small comment improvements.
Fixed a typo with not returning a reference.
* Slight simplification of the ViewInitiatingHierarchy, by adding GetOrAddValue to Dictionary.
* Remove the need for ViewInitiatingHierarchy type.
* Improve output of path in diagnostic for includes hierarchy.
* Remove comment in diagnostic for token-paste-location.slang
* Update command line docs to include `-output-includes`
Co-authored-by: Yong He <yonghe@outlook.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
Based on review feedback from #1556, this change updates the Slang preprocessor so that it is no longer coupled to policy details from higher levels of the software stack. In particular, the preprocessor used to:
* Deal with updating the list of file paths that a `Module` depends on.
* (As of #1556) detect NVAPI-related macro definitions and use them to construct an AST-level `Modifier` attached to the `ModuleDecl`.
This change introduces a callback interface where the `Preprocessor` calls out to a `PreprocessorHandler` at certain points during execution, allowing the handler to introduce custom logic that suits a particular high-level use case.
This change also removes the dependence of the preprocessor on the `Linkage`, because in practice only a small number of its sub-objects were needed. As a convenience, a wrapper function that takes a `Linkage` was left in place so that the existing call sites didn't have to change very much.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In some cases, functionality is available as either a GLSL extension for Vulkan/SPIR-V, or through the NVAPI system for D3D. This situation creates complications because while GLSL extensions are generally all supported by the open-source glslang compiler (which we can bundle and ship), NVAPI operations are exposed through a specific header (`nvHLSLExtns.h`) that ships as part of the NVAPI SDK.
When a user wants to explicitly use NVAPI-provided operations in their shader code, there are no major complications for Slang; the user sets up their include paths, `#include`s the relevant header, calls functions in it, and lets Slang deal with the details of compilation.
The challenge for Slang arises when we want to provide a cross-platform interface in our standard library (e.g., the `RWByteAddressBuffer.InterlockedAddF32` method that was recently added) that uses either a GLSL extension (when compiling for Vulkan/SPIR-V) or an NVAPI (when compiling to DXBC or DXIL). In that case, the code *generated* by Slang now has a dependency on NVAPI, and we need to somehow emit a `#include` directive that pulls it in when invoking fxc or dxc. Because we do not (and seemingly cannot) bundle the NVAPI header with the compiler, we have to rely on ther user to have it available and to somehow communicate to Slang where it is.
Exposing portable routines that sometimes use NVAPI currently creates two main challenges:
1. The user is forced to interact with the "prelude" mechanism in the compiler, which allows the programmer to define code in a given target language that gets prepended to the Slang-generated code. While the prelude mechanism is powerful, it is also hard for users to integrate into their workflow, and our experience so far is that users want something that Just Works.
2. If the user writes code that uses some of our abstract operations that layer on NVAPI *and* they also want to use NVAPI explicitly, they end up with two copies of the NVAPI header (one included by the Slang front-end, and another included by the downstream fxc/dxc compiler). This puts the user in the situation of (a) having to ensure that they set the defines like `NV_SHADER_EXTN_SLOT` consistently both when invoking Slang and when adding their prelude, and (b) even if they do make the definitions consistent, they run into the problem that fxc/dxc complain about overlapping register bindings on the two copies of the `g_NvidiaExt` global shader paraemter that the NVAPI header declares.
This change attempts to resolve both issues by adding a lot of "do what I mean" logic to the compiler to try to ease things in the common case. In particular:
1. The user no longer needs to use the "prelude" mechanism when using NVAPI. The compiler now embeds a default prelude for HLSL output, which will `#include` the NVAPI header if and only if the generated code needs NVAPI access because of portable standard library routines that were used.
2. The user can mix-and-match explicit NVAPI use and stdlib functions that compile to use NVAPI. The register/space to be used by NVAPI when included via prelude is now set based on whatever the user set via the preprocessor so that it should automatically be consistent between both cases. Furthermore, the code we emit for the declaration of `g_NvidiaExt` when compiling explicit NVAPI use is set up to be conditional, so that it is skipped in the case where the prelude will pull in its own declaration of that parameter.
The way all this is achieved involves a lot of moving pieces:
* We now have an HLSL prelude, which mostly just serves to `#include "nvHLSLExtns.h"` in the case where NVAPI support is needed downstream.
* Standard library operations that require NVAPI for their implementation on HLSL include a new `[__requiresNVAPI]` attribute.
* The preprocessor has been extended so that after tokenizing an input file it looks up the NVAPI-relevant macros in the resulting environment, and if they are set it attached a modifier (`NVAPISlotModifier1) to the AST `ModuleDecl` that is based on their values. Logic is added to detect if multiple input files specify values for the macros in ways that conflict.
* The semantic checking step is extended so that it detects the "magic" NVAPI declarations (the `g_NvidiaExt` paramter and the `NvShaderExtnStruct` type that it uses) and attaches a modifier to them so that they can be identified as such in later steps.
* Parameter binding is extended to collect a list of the AST modifiers that reflect NVAPI binding, and to reserve the relevant register(s) so that ordinary user-defined parameters cannot conflict with them.
* IR lowering translates the three new AST modifiers related to NVAPI over to IR equivalents.
* IR linking is extended to make sure that it clones any `IRNVAPISlotDecoration`s attached to the input modules. The pass intentionally does not care where the modifiers came from; it just collects them all and leaves it to downstream code to sort out what they mean.
* Emit logic is extended to have a notion of "prelude directives" which are preprocessor directives that should come *before* the prelude in the generated code, because they can impact the way that the prelude compiles. This is done so that we don't have to introduce ad hoc logic for each downstream compiler to set any relevant `-D` flags (e.g., both fxc and dxc would need to duplicate such logic for NVAPI support).
* The HLSL source emitter is extended to track whether it emits any operations that require NVAPI support.
* The HLSL source emitter is extended to emit prelude directives based on whether NVAPI is needed and, if it is, to also set the register and space that NVAPI should use based on what was stored in the decoration(s) on the IR module.
* The HLSL source emitter is extended so that it detects global instructions that represent "magic" NVAPI constructs , and emit them as conditional definitions so that they are skipped when NVAPI is included via the prelude.
* The handling of requires capabilities during emit logic was cleaned up a bit so that more logic is shared across targets, and also so that the same logic is used both when emitting a function declaration/definition and when emitting a call to an instrinsic function (which won't get declared/defined).
|
| |
|
|
|
|
|
|
| |
nvAPI -> NVAPI
nvAPIPath -> nvapiPath
DxcIncludeHandler don't reference count.
nv-api-path -> nvapi-path
Co-authored-by: Tim Foley <tfoleyNV@users.noreply.github.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix a preprocessor bug affecting X-macros
Fixes #1435
This bug exhibited as nondeterministic output from the preprocessor in release builds, but using a debug build it was narrowed down to a use-after-free issue.
The core problem is subtle, but relates to how we set up the linked list that represents the "busy" status of macros in a particular expansion environment.
Consider this scenario:
```hlsl
X(A)
```
The flow we expect from the preprocessor is something like:
1. Read the `X` token in `X(A)` and recognize the start of a function-like macro invocation. Create an expansion environment for `X`, with the global environment as a parent, read in the arguments (just `A`), and push that expansion onto the stack.
2. Read the `M` token that starts the expansion of `M`, and recognize it as an invocation of the object-like macro representing the argument `M`. Create an expansion environment for the definition of `M` (which is just `A`), and push it onto the stack.
3. Read the token `A` from the expansion for the argument `M`, and recognize it as an invocation of the function-like macro `A`. Create an expansion environemnt for `A`, with the current environment as its parent, read in the arguments (just `0`), and push that expansion onto the stack.
4. Read the token `y` from the expansion for `A`, and recognize it as an invocation of the object-like macro representing the argument `y`. Create an expansion environment for the definition of `y` (which is just `0`) and push it onto the stack.
5. Read `0`.
6. Read a bunch of end-of-file tokens that cause all of these expansions to be popped.
That all looks fine as written, but the gotcha is that the input stream for the expansion in step (2) is only a single token (`A`), which means that during step (3) the current input stream at the time we *create* the macro expansion for `A` is at the end of its input, and by the time we've read in the macro arguments that expansion will have been popped.
The problem, then, is that the logic for setting up the stack of "busy" macros was being performed at the beginning of the expansion (the part referred to as "create an expansion" above), when it should only have been set up as part of pushing the xpansion onto the stack (since at that point we have a guarantee that the parent expansion cannot be popped until the child expansion has been).
The fix here is thus pretty simple: we already have distinct operations for `initializeMacroExpansion()` and `pushMacroExpansion()`, and I simply moved the logic for setting up the "busy" state from the former to the latter.
* fixup: typo
|
| |
|
|
|
|
| |
* Token size on 64 bits is 24 bytes (from 40). On 32 bits is 16 bytes from 24.
* Added hasContent method to Token.
Some other small improvements around Token.
|
| |
|
|
|
|
|
|
| |
* Upper camel -> lowerCamel
m_ prefix members where appropriate
_ prefix module local functions
* m_ prefix members in Lexer. Fit's standard because type has methods/ctor.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Fix for a macro expansion bug
The work in #1177 added a notion of a macro being "busy" to prevent errors around recursively-defined macros:
#define BAR(X) BAR(0)
BAR(A) /* should yield BAR(0) and not infinite-loop */
The fix was to place an `isBusy` flag on each macro, setting it to `true` when an expansion of that macro gets pushed onto the stack of input streams, and back to `false` when the expansion gets popped from the stack of input streams.
That approach meant that we had to pre-expand macro arguments to avoid incorrectly treating a macro as busy for nested-but-not-recursive invocations:
#define NEG(X) (-X)
NEG(NEG(3)) // should expand to (-(-3)) and not (-NEG(3))
The subtle bug that arose with the `isBusy` flag is that the current preprocessor design doesn't always pop input streams when we might expect. For input like the following:
BAR(A) BAR(B)
The preprocessor can be in a state where the stack of input streams look like:
// top input stream:
BAR(X) => BAR(0)
----------------^
// bottom input stream
BAR(A) BAR(B)
------^
That is, the first macro expansion is still "open," even though it is at the end of its input. When the preprocesor looks ahead and sees `BAR` as the next token and asks whether it should be expanded, the `isBusy` state on the BAR macro hasn't yet been unset, so expansion gets skipped.
Aside: it is completely reasonable to ask at this point whether we should just "fix" the behavior of not popping input streams that are at their end. We should consider doing that, but the current code goes to some lengths to preserve the current behavior and I do *not* recall why we had found it necessary. Any attempt to change that behavior should come along with lots of testing.
This change instead adds a different approach for tracking the busy-ness of macros that doesn't rely on mutable state in the macro, and thus works even without pre-expansion of macro arguments.
In the Slang preprocessor, macro names are always looked up in *environments*, where each environment maps names to macros, and has an option parent environment.
There is a global environment used for most cases, but when expanding a function-like macro we would create an "argument environment" for it that maps the parameter names to their argument tokens.
By expanding the environment type to include an (optional) field for a macro that is made "busy" by that environment, we can check if a macro is busy in a given environment by checking if the given macro has been associated with any of the environments on the chain of parent environments.
A function-like macro expansion could then attach the macro to the "argument environment" and automatically make itself busy for the purposes of expansion of its body.
To extend the same functionality to *all* macros, the "argument environment" was changes to an "expansion environment" that always gets used for a macro expansion and always has the identity of the macro attached.
Because the arguments to a function-like macro have their own environment for expansion that bypasses the argument/expansion environment of the function-like macro, the macro will show as busy in its own body, but not in the expansion of its arguments. Thus the pre-expansion of macro arguments is not needed with this change.
Aside: it might not be clear how this design avoids the problem of the unpopped input streams that aren't at their end. The answer is that the "current" environment for the preprocessor is already taken as the environment of the top-most input stream that is *not at its end*. This means that the new test for busy-ness benefits from the existing logic that was in place to deal with non popping input streams as soon as then end. (This is not to say that the current input-stream behavior isn't questionable)
This change includes a new test case for the behavior that was broken, and expands the test case from #1177 to also test object-like macros.
* Fix to handle the mutually-recursive case
The revised "busy" logic was unable to handle a mutually-recursive macro like:
#define ABC XYZ
#define XYZ ABC
This change restores the ability to handle mutually recursive cases, and makes sure that we test that case.
The crux of the fix relies on splitting the environment and busy-macro tracking into more separate data structures.
Rather than the list of environments directly trakcing busy-ness via the chain of parent environments, an environment now stores a separate linked list of macros that should be considered busy in that environment.
Decoupling the two lists allows for an important change: when expanding an ordinary macro (either object-like or function-like, but *not* a function argument) the parent *environment* comes from the point where the macro was defined (this is required for lookup to make sense), but the parent list of *busy macros* comes from the current input stream at the point where expansion takes place. That change ensures that non-argument macros always properly "stack up" the list of busy macros.
|
| |
|
|
|
|
|
| |
* First pass fix of macro expansion logic to stop recursive application (causting a recursive loop), whilst also allowing application on parameters to a macro.
* Added recursive-macro test.
Fixed macro application example.
|
|
|
* Prefixing source files in source/slang with slang-
* Prefix source in source/slang with slang- prefix.
* Rename core source files with slang- prefix.
* Update project files.
* Fix problems from automatic merge.
|
