summaryrefslogtreecommitdiffstats
path: root/source/core/slang-token-reader.h
Commit message (Collapse)AuthorAge
* formatEllie Hermaszewska2024-10-29
| | | | | | | * format * Minor test fixes * enable checking cpp format in ci
* Allow parsing some SPIR-V enums in intrinsics (#3062)Ellie Hermaszewska2023-08-08
| | | | | | | * Add TokenReader::AdvanceIf overlaod for TokenType * Add some spirv defs to parser * Add comment
* StringBuilder to lowerCamel (#2840)jsmall-nvidia2023-04-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * #include an absolute path didn't work - because paths were taken to always be relative. * WIP lowerCamel Dictionary. * WIP more lowerCamel fixes for Dictionary. * Add/Remove/Clear * GetValue/Contains * Fix tabs in dictionary. Count -> getCount * Fix fields with caps. * Key -> key Value -> value Use m_ for members where appropriate. Use lowerCamel in linked list. * Some small fixes/improvements to Dictionary. * Kick CI. * Small tidy on String. * Append -> append * ToString -> toString ProduceString -> produceString * Small fixes. * StringToXXX -> stringToXXX * Fix typo introduced by Append -> append. * Made intToAscii do reversal at the end. --------- Co-authored-by: Yong He <yonghe@outlook.com>
* Dictionary using lowerCamel (#2835)jsmall-nvidia2023-04-25
| | | | | | | | | | | | | | | | | | | | | | | | | * #include an absolute path didn't work - because paths were taken to always be relative. * WIP lowerCamel Dictionary. * WIP more lowerCamel fixes for Dictionary. * Add/Remove/Clear * GetValue/Contains * Fix tabs in dictionary. Count -> getCount * Fix fields with caps. * Key -> key Value -> value Use m_ for members where appropriate. Use lowerCamel in linked list. * Some small fixes/improvements to Dictionary. * Kick CI.
* Added NativeStringType (#2252)jsmall-nvidia2022-05-27
| | | | | | | | | | | | | | | | | | | * #include an absolute path didn't work - because paths were taken to always be relative. * Use TerminatedUnownedStringSlice for literals in output C++. * Remove Escape/Unescape functions used in slang-token-reader.cpp Add target type of 'host-cpp' etc to map to the target types. * Fix some corner cases around string encoding. * Added unit test for string escaping. Fixed some assorted escaping bugs. * Updated test output. * Added decode test. * Stop using hex output, to get around 'greedy' aspect. Use octal instead.
* Allow slangc to generate exe from .slang file. (#2170)Yong He2022-03-28
|
* Further implementation of SPIRV direct emit. (#1920)Yong He2021-08-12
| | | | | | | | | | | | | | * Further implementation of SPIRV direct emit. This change implements: - Struct, Vector, Matrix and Unsized Array types. - Basic arithmetic opcodes, vector construct, swizzle etc. - getElementPtr, getElement, fieldAddress, extractField. - SPIRV target intrinsics with SPIRV asm code in stdlib. - RWStructuredBuffer and StructuredBuffer. - Pointer storage class propagation. - Control flow. * Fix.
* Improvements in -X support (#1852)jsmall-nvidia2021-05-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * #include an absolute path didn't work - because paths were taken to always be relative. * Added SourceLoc handling for command line parsing. * Fix typo in debug. * Fix issue around the DiagnosticSink used in options parsing not having a writer available - by having DiagnosticSink parenting. * Small rename for clarity. * WIP extracting command line args for downstream tools. * Unit tests/bug fixes around extracting args. * Use DownstreamArgs in the EndToEndCompileRequest * Passing downstream compiler options downstream. * Fix issue with endToEndReq being nullptr. * Fix issue with diagnostics number change. * Small improvements to how the source line is displayed if it's too long. Default to 120, as suggested in previous review. * Make render test use x-args parsing and CommandArgReader. * Added missing diagnostics. * More DownstreamArgs to linkage so can be seen by 'components'. Added dxc-x-arg test. * Used combination of name and args instead of two Lists, which whilst equivalent was perhaps a little confusing. * Added documentation for -X support. * Added test for x-args parsing diagnostic. Improved diagnostic with list of known names. * Fix issues from merge. * Fix lookup for -matrix-layout-column-major in render test. * Remove commented out line.
* Add a streamlined syntax for TEST_INPUT lines (#1768)Tim Foley2021-03-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change allows the `TEST_INPUT` syntax used by `render-test` to support aggregate values with a single input line more easily. The test writer can now use a syntax like: ``` //TEST_INPUT:set someVar = 3.0 ``` Input lines that start with the `set` keyword will now use a simpler `dst = src` format (instead of `dst:name=src` as the existing syntax used). The right-hand side expression can include: * Numeric literals, both integer and floating-point (currently only supporting 32-bit scalar types; we could fix this later) * Arrays, consisting of zero or more comma-separated expressions inside `[]` * Aggregates, consisting of zero or more comma-separated "fields" inside `{}`. A field can either be `name: <expr>` or just `<expr>` * Objects, which can be written as either `new SomeType{ <fields> }` or `new{ <fields> }` in the case where the type is know-able from context With this approach is should be possible to support almost arbitrary-type inputs on a single line. For now, I have used this support to re-enable an existing test that had been disabled due to lack of support for setting up arrays of objects. Major things left to do: * The new syntax doesn't support the existing cases we had for `Texture2D`, etc. Those should probably be supported but I'd like to find a way to do it without duplicating the parsing logic (ideally the value cases from the existing code should Just Work in the new model) * There is no support right now for non-32-bit scalar types * It would be good if this support (and the shader cursor system) supported treating vectors like aggregates * The actual value-setting logic doesn't currently handle aggregates without field names, so `{ a:0, b:1 }` will work but `{ 0, 1 }` will parse but fail when it comes time to set values * While this approach lets complicated values be set with a single line, that isn't always what a user will want to do: in the future we should provide a way to break up an aggregate value over multiple lines that is consistent with this approach * Once we port all of the relvant tests over, it would be great to drop the `set` prefix and have these lines look as simple and conventional as possible
* Clean up render-test handling of input (#1766)Tim Foley2021-03-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The original goal of this change was to streamline the `TEST_INPUT` system by eliminating options that are no longer relevant once we have eliminated the non-shader-object execution paths. The result is more or less a re-implementation/refactor of the logic around how input is parsed and represented, that tries to set things up for a more general sytem going forward. The main changes isthat the `ShaderInputLayout` no longer tracks a simple flat list of `ShaderInputLayoutEntry` (that is a kind of pseudo-union of the various buffer/texture/value cases), and it instead uses a hierarchical representation composed of `RefObject`-derived classes to represent "values." There are several "simple" cases of values * Textures * Samplers * Uniform/ordinary data (`uniform`) * Buffers composed of uniform/ordinary data (`ubuffer`) Then there are composed/aggregate values that nest other values: * An *aggregate* value is a set of *fields* which are name/value pairs. It can be used to fill in a structure, for example. * An *array* value is a list of values for the elements of an array. It can be used to fill out an array-of-textures parameter, for example. * A combined texture/sampler value is a pair of a texture value and a sampler value (easy enough) * An *object* holds an optional type name for a shader object to allocate (it defaults to the type that is "under" the current shader cursor when binding), and a nested value that describes how to fill in the contents of that object Finally there are cases of values that are just syntactic sugar: * A `cbuffer` is just shorthand for creating an object value with a nested uniform/ordinary data value The big idea with this recursive structure is that it gives us a way to handle more arbitrary data types with name-based binding. Supporting this new capability requires changes to both how input layouts get parsed, and also how they get bound into shader objects. On the parsing side, things have been refactored a bit so that parsing isn't a single monolithic routine. The refactor also tries to make it so that the various options on an input item (e.g., the `size=...` option for textures) are only supported on the relevant type of entry (so you can't specify as many useless options that will be ignored). The bigger change to parsing is that it now supports a hierarchical structure, where certain input elements like `begin_array` can push a new "parent" value onto a stack, and subsequent `TEST_INPUT` lines will be parsed as children of that item until a matching `end` item. This approach means that we can now in principle describe arbitrary hierarchical structures as part of test input without endlessly increasing the complexity of invididual `TEST_INPUT` lines. On the binding side, we now have a central recursive operation called `assign(ShaderCursor, ShaderInputLayout::ValPtr)` that assigns from a parsed `ShaderInputLayout` value to a particular cursor. That operation can then recurse on the fields/elements/contents of whatever the cursor points to. Major open directions: * With this change it is still necessary to use `uniform` entries to set things like individual integers or `float`s and that is a little silly. It would be good to have some streamlines cases for setting individual scalar values. * Further, once we have a hierarchical representation of the values for `TEST_INPUT` lines, it becomes clear that we really ought to move to a format more like `TEST_INPUT: dstLocation = srcValue;` where `srcValue` is some kind of hierarchial expression grammar. Refactoring things in this way should make the binding logic even more clear and easy to understand. The refactored parser should make parsing hierarchical expressions easier to do in the future (even if it uses the push/pop model for now) * One detailed note is that the representation of buffers in this change is kind of a compromise. Just as an "object" value is a thin wrapper around a recursively-contained value for its "content" it seems clear that a buffer could be represented as a wrapper around a content value that could include hierarchical aggregates/objects instead of just flat binary data (this would be important for things like a buffer over a structure type that lays out different on different targets). The main problem right now with changing the representation is actually needing to compute the size of a buffer based on its content, so that can/should be addressed in a subsequent change. Details: * The base `RenderTestApp` class and the `ShaderObjectRenderTestApp` classes have been merged, since the hierarchy no longer serves any purpose. * Disabled the tess that rely on `StructuredBuffer<IWhatever>` because they aren't really supported by our current shader object implementation * Replaced used of `Uniform` and `root_constants` in `TEST_INPUT` lines with just `uniform` * Removed a bunch of uses of `stride` from `cbuffer` inputs, where it wasn't really correct/meaningful * Added the `copyBuffer()` operation to VK/D3D renderers, along with some missing `Usage` cases to support it. * Made `ShaderCursor` handle the logic to look up a name in the entry points of a root shader object, rather than just having that logic in `render-test`. (We probably need to make a clear design choice on this issue)
* Standard library save/loadable (#1592)jsmall-nvidia2020-11-05
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * #include an absolute path didn't work - because paths were taken to always be relative. * Fix handling of access modifiers inside type definition. * Fix access problem for AST node. Make dumping produce a single function with switch, to potentially make available without Dump specific access. * WIP on serialization design doc. * Remove project references to previously generated files. * More docs on serialization design. * Improve serialization documentation. Remove unused function from IRSerialReader. * Small fixes around naming. Remove long comment from slang-serialize.h - as covered in serialization.md * Remove long comment in slang-serialize.h as covered in serialization.md * More information about doing replacements on read for AST and problems surrounding. * Typo fix. * Spelling fixes. * Value serialize. * Value types with inheritence. * Use value reflection serial conversion for more AST types * Use automatic serialization on more of AST. * Get the types via decltype, simplifies what the extractor has to do. * Update the serialization.md for the value serialization. * Small doc improvements. * Update project. * Remove ImportExternalDecl type Added addImportSymbol and ImportSymbol type Fixed bug in container which meant it wouldn't read back AST module * Because of change of how imports and handled, store objects as SerialPointers. * First pass symbol lookup from mangled names. * Cache current module looked up from mangled name. * Fix SourceLoc bug. Improve comments. * Added diagnostic on mangled symbol not being found * Fix typo. * WIP serializing stdlib. * WIP serializing stdlib in. * Fix problem serializing arrays that hold data that is already serialized. * Remove clash of names in MagicTypeModifier. * Make conversion from char to String explicit. Fix reference count issue with SerialReader. * Add code to save/load stdlib. * Use return code to avoid warning - SerialContainerUtil::write(module, options, &stream)) * Make all String numeric ctors explicit. Added isChar to UnownedStringSlice. Added operator== for UnownedStringSlice to String to avoid need to convert to String and allocate. * Add error check to readAllText. * tabs -> spaces on String.h * tab -> spaces String.cpp * Remove msg for StringBuilder, just build inplace for exceptions. * Check SerialClasses - for name clashes. Renamed Modifier::name as Modifier::keywordName * Handling of extensions when deserializing AST - updating the moduleDecl->mapTypeToCandidateExtensions Co-authored-by: Tim Foley <tim.foley.is@gmail.com>
* WIP: CPU compute coverage (#1030)jsmall-nvidia2019-08-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add support for '=' when defining a name in test. * Add support for double intrinsics. * Add support for asdouble Add findOrAddInst - used instead of findOrEmitHoistableInst, for nominal instructions. Support cloning of string literals. C++ working on more compute tests. * Constant buffer support in reflection. Fixed debugging into source for generated C++. buffer-layout.slang works. * Added cpu test result. * Remove some commented out code. Comment on next fixes. * Improvements to reflection CPU code. * C++ working with ByteAddressBuffer. * Enabled more compute tests for CPU. * Enabled more compute tests on CPU. Added support for [] style access to a vector. * Enabled more CPU compute tests. * Handling of buffer-type-splitting.slang Named buffers can be paths to resources * Fix some warnings, remove some dead code. * Fix problem with verification of number of operands for asuint/asint as they can have 1 or 3 operands. asdouble takes 2. * Fix handling in MemoryArena around aligned allocations. That _allocateAlignedFromNewBlock assumed the block allocated has the aligment that was requested and so did not correct the start address.
* Use slang- prefix on slang compiler and core source (#973)jsmall-nvidia2019-05-31
* Prefixing source files in source/slang with slang- * Prefix source in source/slang with slang- prefix. * Rename core source files with slang- prefix. * Update project files. * Fix problems from automatic merge.