| Age | Commit message (Collapse) | Author |
|---|
|
* Rename existing ParameterBlock to ParameterGroup
We are planning to add a new `ParameterBlock<T>` type, which maps to the notion of a "parameter block" as used in the Spire research work.
Unfortunately, the compiler codebase already uses the term `ParameterBlock` as catch-all to encompass all of HLSL `cbuffer`/`tbuffer` and GLSL `uniform`/`buffer`/`in`/`out` blocks (all of which are lexical `{}`-enclosed blocks that define parameters...).
This change instead renames all of the existing concepts over to `ParameterGroup`, which isn't an ideal name, but at least doesn't directly overlap the new terminology or any existing terminology.
The new `ParameterBlockType` case will probably be a subclass of `ParameterGroupType`, since it is a logical extension of the underlying concept.
* Add Shader Model 5.1 profiles
The HLSL `register(..., space0)` syntax is only allowed on "SM5.1" and later profiles (which is supported by the newer version of `d3dcompiler_47.dll` that comes with the Win10 SDK, but not the older version of `d3dcompiler_47.dll` - good luck figuring out which you have!).
This change adds those profiles to our master list of profiles, and nothing else.
* First pass at support for `ParameterBlock<T>`
- Add the type declaration in stdlib
- Add a special case of `ParameterGroupType` for parameter blocks
- Handle parameter blocks in type layout (currently handling them identically to constant buffers for now, which isn't going to be right in the long term)
- Add an IR pass that basically replaces `ParameterBlock<T>` with `T`
- Eventually this should replace it with either `T` or `ConstantBuffer<T>`, depending on whether the layout that was computed required a constant buffer to hold any "free" uniforms
- Add first stab at an IR pass to "scalarize" global variables using aggregate types with resources inside.
- This currently only applies to global variables, so it won't handle things passed through functions, or used as local variables
- It also only supports cases where the references to the original variable are always references to its fields, and not the whole value itself
- Add a single test case that technically passes with this level of support, but probably isn't very representative of what we need from the feature
* Fold parameter-block desugaring into a more complete "type legalization" pass
The basic problem that was arising is that once you desugar `ParameterBlock<T>` into `T`, you then need todeal with splitting `T` into its constituent fields if it contains any resource types.
Handling those transformations by following the usual use-def chains wasn't really helping, because you might need systematic rewriting that can really only be handled bottom-up.
This change adds a new pass that is intended to perform multiple kinds of type "legalization" at once:
- It will turn `ParameterBlock<T>` into `T`
- It may at some point also convert `ConstantBuffer<T>` into `T` as well
- It will turn an value of an aggregate type that contains resources into N different values (one per field)
- As a result of this, it will also deal with AOS-to-SOA conversion of these types
Legalization is applied to *every* function/instruction/value, so that it can make large-scale changes that would be tough to manage with a work list.
This pass needs to be run *after* generics have been fully specialized, so that we know we are always dealing with fully concrete types, so that their legalization for a given target is completely known.
This is still work in progress; there's more to be done to get this working with all our test cases, and finish the remaining `ParameterBlock<T>` work.
* Improve binding/layout information when using parameter blocks
- When doing type layout for a parameter block, don't include the resources consumed by the element type in the resource usage for the parameter block
- Note that this is pretty much identical to how a `ConstantBuffer<T>` does not report any `LayoutResourceKind::Uniform` usage, except that `ParameterBlock<T>` is *also* going to hide underlying texture/sampler reigster usage
- The one exception here is that any nested items that use up entire `space`s or `set`s those need to be exposed in the resource usage of the parent (I don't have a test for this)
- When type legalization needs to scalarize things, it must propagate layout information down to the new leaf variables. In general, the register/index for a new leaf parameter should be the sum of the offsets for all of the parent variables along the "chain" from the original variable down to the leaf (we aren't dealing with arrays here just yet).
- When type legalization decides to eliminate a pointer(-like) type (e.g., desugar `ParameterBlock<T>` over to `T`), actually deal with that in terms of the `LegalVal`s created, so that we can know to turn a `load` into a no-op when applied to a value that got indirection removed.
- Hack up the "complex" parameter-block test so that it actually passes (the big hack here is that the HLSL baseline is using names that are generated by the IR, and are unlikely to be stable as we add/remove transformations).
- Note: I can't make these be compute tests right now, because regsiter spaces/sets are a feature of D3D12/Vulkan, and our test runner isn't using those APIs.
|
|
- Add shader model 6.0, 6.1, and 6.2 targets
- Add DXIL and DXIL assembly as output formats
- Add header for DXC API to `external/`
- Add `dxc-support.cpp` that wraps usage of the API
- Add `-pass-through dxc` option, equivalent to what we have for `fxc`
Notes:
* This does *not* include any logic to add `dxcompiler.dll` to our build process; that is way out of scope for the build complexity I'm ready to deal with
* For right now, the use of `dxcompiler.dll` is hard-coded, and it must be discoverable in the current executable's search path; options to customize can come later
* The `-pass-through` option is kind of silly because the code doesn't actually pay attention to the value (just whether it is set). If you set it to `fxc` but ask for DXIL, we pass through `dxc` anyway.
|
|
None of these changes are made "live" at the moment. I'm just trying to get them checked in to avoid divering too far from `master` at any point during development.
- Add basic emit logic to produce GLSL from the IR in a few cases (the existing IR emit logic was ad hoc and HLSL-specific)
- When lowering a function declaration, walk up its chain of parent declarations to collect additional parameters as needed
- When lowering a call, make sure to add generic arguments that come from the declaration reference being called
- Attach a "mangled name" to symbols when lowering, so that we can eventually use that name to resolve things for linkage.
- After the above work, I had to apply some fixups to make sure that generic arguments *don't* get added when the user is calling an `__intrinsic_op` function, since those should map 1-to-1 down to instructions with just their ordinary parameter list.
A big open question right now is whether I should continue to represent the generic arguments as just part of the ordinary argument list for a function, or split them out into separate `applyGeneric` and `apply` steps.
A strongly related question is whether a declaration with generic parameters should lower into a single declaration, or one declaration nested inside an outer generic declaration.
A good future step at this point would be to eliminate a lot of the `__intrinsic_op` stuff in favor of having the builtin functions include their own definitions, which might be in terms of a new expression-level construct for writing inline IR operations. This can't be done until the existing AST-to-AST path is no longer needed for cross-compilation purposes.
More immediate next steps here:
- We need a way to round-trip calls to external declaration that get handled by this mangled-name logic. Basically, if we are asked to output HLSL and we see a call to `_S...GetDimensions...(float4, t, a, ...)` we need to be able to walk the mangled name and get back to `t.getDimensions(a, ...)` without a whole lot of manual definitions to make things round-trip.
- In the other case, where a declaration isn't built-in for the chosen target, we need to be able to load a module of target-specific definitions (which will somehow map back to symbols with certain mangled names) and then look these up (by mangled name) and then load/link/inline them into the user's IR to satisfy requirements in their code.
|
|
At a high level, this commit adds two things:
1. A "bytecode" format for serializing Slang IR instructions and related structure (functions, "registers")
2. A virtual machine that can load and then execute code in that bytecode format.
The reason for kicking off this work right now is that we *need* a way to run tests on Slang code generation that doesn't rely on having a GPU present (given that our CI runs on VM instances without GPUs), nor on textual comparison to the output of other compilers. With these features I've implemented a slapdash `slang-eval-test` test fixture that can run a (trivial) compute shader to very our compilation flow through to bytecode.
Some key design constraints/challenges:
- The bytecode format should be "position independent" so that a user can just load a blob of data and then inspect it without having to deserialize into another format, allocate memory, etc. Eventually the bytecode format might be a replacement for out current reflection API (we used to base reflection off a similar format, but the cost/benefit wasn't there at the time and we switched to just using the AST).
- The VM should be able to execute bytecode functions without doing any per-operation translation, JIT, etc. (translation of more coarse-grained symbols is okay). For now the VM is just being used to run tests, but eventually I'd like it to be viable for:
- Running Slang-based code in the context of the compiler itself. This starts with stuff like constant-folding in the front-end, but could expand to more general metaprogramming features.
- Running Slang-based ocde within a runtime application (e.g., a game engine) that wants to be able to run things like "parameter shader" code, or even just evaluate compute-like code on CPU (e.g., when supporting particles on both CPU and GPU).
- Finally, the bytecode format should ideally be able to round-trip back to the IR without unacceptable loss of information. This requirement and the previous one play off of each other, because things like a traditional SSA phi operation is ugly when you have to actually *execute* it. This doesn't matter right now when we don't have SSA yet, but it might be part of the decision-making here.
The actual implementation is centralized in `bytecode.{h,cpp}` and `vm.{h.cpp}`.
Big picture notes:
- The space of opcodes is shared between IR and bytecode (BC), with the hope that this makes translation of operations between the two easy.
- The actual bytecode instruction stream relies on a variable-length encoding for integer values, including opcodes and operand numbers, so that the common case is single-byte encoding.
- In the long term I intend to have a rule that if you use a single-byte encoding for an opcode, then all operands are required to use single-byte encodings too. Operations that need multi-byte operands would then be forced to use a multi-byte encoding of the op, and would be sent down a slower path in the interpeter.
- The "bytecode"'s outer structure is based on ordinary data structures linked with pointers, but they are "relative pointers" so the actual structure is position-independent.
- There are two main kinds of operands: registers and "constants." An operand is a signed integer where non-negatie values indicate registers (with `index == operandVal`) and negative values indicate constants (with `index == ~operandVal`).
- Registers are stored in the "stack frame" for a VM function call, and each has a fixed offset based on the size of the type and those that come before it. Conceptually, registers are allowed to overlap if they aren't live at the same time, and we manage this with a simple stack model: every register is supposed to identify the register that comes directly before it (this isn't implemented yet).
- "Constants" are more realistically a representation of "captured" values, but they are currently also how constants come in. Basically we can use a compact range of indices in the bytecode for a function, and each of these indices indirectly refers to some value in the next outer scope.
- The actual encoding of bytecode instructions right now is largely ad-hoc and very wasteful (we encode the type on everything, and we also encode everything as if it had varargs).
- In some cases, an instruction needs to know the types of the values involved (e.g., because it needs to load an array element, which means copying a number of bytes based on the size). The way the VM works we have types attached to our registers, so we currently get sneaky and look at those types in some ops. Longer term is makes sense to encode the required type info directly in the BC.
- There's a whole lot of hand-waving going on with how the actual top-level bytecode module gets loaded, because of the way we currently treat the top-level module as an instruction stream in the IR. This means that we try to represent the loaded module as a "stack frame" for a call to the module as a function, but that approach as serious problems, and isn't realistically what we want to do.
|
|
* IR: handle control flow constructs
This change includes a bunch of fixes and additions to the IR path:
- `slang-ir-assembly` is now a valid output target (so we can use it for testing)
- This uses what used to be the IR "dumping" logic, revamped to support much prettier output.
- A future change will need to add back support for less prettified output to use when actually debugging
- IR generation for `for` loops and `if` statements is supported
- HLSL output from the above control flow constructs is implemented
- Revamped the handling of l-values, and in particular work on compound ops like `+=`
- Add basic IR support for `groupshared` variables
- Add basic IR support for storing compute thread-group size
- Output semantics on entry point parameters
- This uses the AST structures to find semantics, so its still needs work
- Pass through loop unroll flags
- This is required to match `fxc` output, at least until we implement
unrolling ourselves.
* Fixup: 64-bit build issues.
* fixup for merge
|
|
The goal here is to get the Slang "standard library" code out of string literals and into something a bit more like an actual code file.
This is handled by having a `slang-generate` tool that can translate a "template" file that mixes raw Slang code (or any language we want to generate...) with generation logic that is implemented in C++ (currently).
This work isn't final by any stretch of the imagination, but it moves a lot of code and not merging it ASAP will complicate other changes.
My expectation is that the generator tool will be beefed up on an as-needed basis, to get our stdlib code working.
Similarly, the stdlib code does not really take advantage of the new approach as much as it could. That is something we can clean up along the way as we do modifications of the stdlib.
|
|
The code previously had an enumerated type for "intrinsic" operations, and allowed functions to be marked `__intrinsic_op(...)` to indicate the operation they map to.
The nature of the IR meant that each of these intrinsic ops had to have a corresponding IR opcode, but the `enum` types weren't the same.
This change cleans things up a bit by deciding that the `__intrinsic_op(...)` modifier names an actual IR opcode, and so the `IntrinsicOp` enum is gone.
The biggest source of complexity here is that there are certain operations that need to be "intrinsic"-ish for the purposes of the current AST-based translation path, because we need them to round-trip from source to AST and back.
Right now this is being handled by defining a bunch of "pseudo-ops" which can be used in the `__intrinsic_op` modifier, but which are *not* meant to be represented in the IR.
Currently I don't actually handle this during IR generation.
In the long run, once we are using IR for everything that needs cross-compilation, we should be able to eliminate the pseudo-ops in favor of just having these be ordinary (inline) functions defined in the stdlib (e.g., the `+=` operator can just have a direct definition).
There was a second category of modifier that gets a little caught up in this, which is the `__intrinsic` modifier, which got used in two ways:
1. A function marked `__intrinsic(glsl, ...)` had what I call a "target intrinsic" modifier, which specified how to lower it for a specific target (e.g., GLSL).
2. A function just marked `__intrinsic` was supposed to be a marker for "this function shouldn't be emitted in the output, because the implementation is expected to be provided"
The latter category of function should really be an `__intrinsic_op`, so I translated all those uses. I added a tiny bit of sugar so that `__intrinsic_op` without an explicit opcode will look up an opcode based on the name of the function being called, so that an operation like `sin` can automatically be plumbed through to an equivalent IR op. (The first category is a stopgap for the AST-based cross-compilation, and will hopefully be replaced by something better as we get the IR-based path working).
Getting the switch from `__intrinsic` to `__intrinsic_op` working required shuffling around some code in `emit.cpp` that handles looking up those modifiers and emitting builtin operations appropriately during cross-compilation.
Depending on where we go with things, a possible extension of this approach is to allow multiple operands to `__intrinsic_op` so that the first specifies the opcode, and then the rest are literal arguments to specify "sub-ops." This could help us handle stuff like texture-fetch operations without an explosion in the number of opcodes. I still need to think about whether this is a good idea or not.
|
|
- Change IR instructions to just hold an integer opcode instead of a pointer to the "info" structure
- Externalize definition of IR instructions to a header file, and use the "X macro" approach to allow generating different definitions
- Add notion of function types to the IR, so that we can easily query the result type of a function
- Add some convenience accesors to allow walking the IR in a strongly-typed manner (e.g., iterate over the parameters of a function)
- TODO: these should really be changed to assert the type of things, as least in debug builds
- Add very basic logic to `emit.cpp` so that it can walk the generated IR and start printing it back as HLSL
- This isn't meant to be usable as-is, but it is a step toward where we need to go
|
|
Right now none of this is hooked up, but I want to get things checked in incrementally rather than have along long-lived branches.
- Added placeholder declarations for IR representation of instructions, basic blocks, etc.
- Start adding a `lower-to-ir` pass to translate from AST representation to IR
Again: none of this is functional, so it shouldn't mess with existing users of the compiler.
|
|
Fixes #23
Up to this point, the compiler has used the ordinary `String` type to represent declaration names, which means a bunch of lookup structures throughout the compiler were string-to-whatever maps, which can reduce efficiency.
It also means that things like the `Token` type end up carying a `String` by value and paying for things like reference-counting.
This change adds a `Name` type that is used to represent names of variables, types, macros, etc.
Names are cached and unique'd globally for a session, and the string-to-name mapping gets done during lexing.
From that point on, most mapping is from pointers, which should make all the various table lookups faster.
More importantly (possibly), this brings us one step closer to being able to pool-allocate the AST nodes.
|
|
Fixes #24
So far the code has used a representation for source locations that is heavy-weight, but typical of research or hobby compilers: a `struct` type containing a line number and a (heap-allocated) string.
This is actually very convenient for debugging, but it means that any data structure that might contain a source location needs careful memory management (because of those strings) and has a tendency to bloat.
The new represnetation is that a source location is just a pointer-sized integer.
In the simplest mental model, you can think of this as just counting every byte of source text that is passed in, and using those to name locations.
Finding the path and line number that corresponds to a location involves a lookup step, but we can arrange to store all the files in an array sorted by their start locations, and do a binary search.
Finding line numbers inside a file is similarly fast (one you pay a one-time cost to build an array of starting offsets for lines).
More advanced compilers like clang actually go further and create a unique range of source locations to represent a file each time it gets included, so that they can track the include stack and reproduce it in diagnostic messages.
I'm not doing anything that clever here.
|
|
There were two main places where global variables were used in the Slang implementation:
1. The "standard library" code was generated as a string at run-time, and stored in a global variable so that it could be amortized across compiles.
2. The representation of types uses some globals (well, class `static` members) to store common types (e.g., `void`) and to deal with memory lifetime for things like canonicalized types.
In each case the "simple" fix is to move the relevant state into the `Session` type that controlled their lifetime already (the `Session` destructor was already cleaning up these globals to avoid leaks).
For the standard library stuff this really was easy, but for the types it required threading through the `Session` a bit carefully.
One more case that I found: there was a function-`static` variable used to generate a unique ID for files output when dumping of intermediates is enabled (this is almost strictly a debugging option).
Rather than make this counter per-session (which would lead to different sessions on different threads clobbering the same few files), I went ahead and used an atomic in this case.
Note that the remaining case I had been worried about was any function-`static` counter that might be used in generating unique names.
It turns out that right now the parser doesn't use such a counter (even in cases where it probably should), and the lowering pass already uses a counter local to the pass (again, whether or not this is a good idea).
This change should be a major step toward allowing an application to use Slang in multiple threads, so long as each thread uses a distinct `SlangSession`. The case of using a single session across multiple threads is harder to support, and will require more careful implementation work.
|
|
- Change the `slang` project from a static library to a dynamic one
- Add some details around `slang.h` to make sure DLL export stuff is working
- Make the `slangc` executable use the dynamic library
- Rename the `glslang` sub-project to `slang-glslang` and move it into the main source hierarchy
- This reflects the fact that it isn't a stand-alone tool, and isn't in any way a standard binary of glslang, but rather just an artifact of how Slang uses glslang
|
|
The tricky bit here was that the `reflection-json` output format isn't really a code generation target like the others, and we need to be able to have multiple "targets" active to make sense of it. This needs cleaning-up.
|
|
- The big change here is the introduction of a "lowering" pass that takes an input AST from the semantic checker, and produces an output AST suitable for emitting. The intention is that he lowering pass is responsible for:
- Stripping out unused code (when we have enough information to do so), by only outputting declarations that are transitively references from an entry point
- When cross-compiling to GLSL, generating a suitable `void main()` entry point to wrap the user-written entry-point function
- (Eventually) legalizing types in the program, by scalarizing aggregate types that mix uniform and resource types
- (Eventually) instantiating generic declarations so that the resulting code only deals with fully specialized declarations
- (Eventually) de-sugaring OOP constructs into basic "structs and functions" form
- (Eventually) instantiating code that depends on interface types at the concrete types chosen
- It is clear that there is still a lot of work to be done there, to this change is really about getting infrastructure in place without breaking the existing test cases.
- One cleanup here is that we get rid of the idea of whole-translation-unit output, since that was specific to HLSL output, and there is really no strong reason for keeping it. Users should now just ask for the output for each entry point that they wanted to generate.
- The biggest source of complexity for the lowering process is that it needs to produce the same AST structure as the input, to deal with the complexity of the rewriter case. That is, we need the output to be able to reproduce the input exactly in the case where we are rewriting and nothing needs to change, so the output format needs at least the degrees of freedom of the input.
- As a result, we end up having to distinguish "rewriter" and "full" modes in both lowering and code-emit steps, so that we can react appropriately.
- Generating a GLSL `main()` also adds a lot of complexity. Right now I'm using the simplest approach, where we always output the Slang/HLSL entry point as an ordinary function (as written) and then emit a simple GLSL `main()` to call it. I generate globals for all the shader inputs/outputs (these need to be scalarized and have explicit `location`s attached), and then collect these into the `struct` types of the original parameters as needed.
- This approach will start to have some major down-sides once we have to deal with "arrayed" input/output
- A long-term question here is how to replace entry-point parameter types with scalarized and/or "transposed" versions, while still letting the original code work as written (including copying those inputs to temporary arrays)
- Split `BlockStatementSyntaxNode` into:
- `BlockStmt` which just provides a scope around a `body` statement
- `SeqStmt` which just allows multiple statements to be treated as one
- Change how we emit `for` loops, to deal with the case where the initialization part might expand into multiple statements
- Basically `for(A;B;C) {D}` becomes `{A; for(;B;C) {D}}`, so we can handle arbitrary statements for `A`
- As an additional wrinkle, when we are rewriting HLSL, we just generate `A; for(;B;C) {D}` to deal with the broken scoping there
- This change is needed because the lowering pass was sometimes expanding the original initialization statement `A` into a block `{A}`. Certainly if it declared multiple variables we'd need to handle it, and this seemed the easiest way
- A more significant challenge for lowering would come if/when we ever wanted to support true short-circuiting behavior for `&&` and `||`
- For right now I'm not changing the behavior of the "rewriter" mode, so we still have `UnparsedStmt` instances being generated, but it is clear that eventually we need to parse *all* input, even if we can't type-check 100% of it. This is required so that we can rewrite user code that might refer to a shader input with interface type.
|
|
- The big change here is that all the definitions for syntax-node classes have been macro-ized, to that we can do light metaprogramming over them
- The use of macros for this has big down-sides, but I'm not quite ready to do anything more heavy-weight right now
- The macro-ized definitions can be included multiple times, to generate different declarations/code as needed
- The first example of using this meta-programming facility is a new visitor system
- The actual visitor base classes and the dispatch logic are all generated from the meta-files
- There was only one visitor left in the code: the semantics checker, so that was ported to the new system.
- All current test cases pass, so *of course* that means all is well.
|
|
That is, even if hte user specified the `-no-checking` option (or the equivalent via API), we still want/need to apply full semantic checks to Slang code, so that cross-compilation will be possible.
|
|
The basic idea of this change is that user code can just write:
#include "foo.h"
and then if `foo.h` gets found in a list of registered directories for "auto-import," then it actually gets interpreted as if the user had writte, more or less:
__import foo;
That is, the code in `foo.h` will be treated as Slang, and will be fully parsed and checked (no matter what the source language had been), and the scoping rules will be those of `__import` instead of `#include`.
This is a really big hammer, and I could imagine it smashing fingers if used poorly.
I'm not sure this feature will pan out, but we need to try things to know.
One big piece of that that I'll likely keep in either case is an overhaul of command-line options parsing for `slangc`. In particular, this logic has been moved into the core `slang` library (so that users can just pass options in via the API), and it is all done on UTF-8 strings rather than wide strings (which was always going to be Windows-specific).
|
|
All of this is just related to cruft left over from the old project setup.
|
|
|
