Making it easier to work with shaders https://git.yummers.dev/slang.git/atom?h=master 2019-05-31T21:20:37+00:00 2019-05-31T21:20:37+00:00 jsmall-nvidia jsmall@nvidia.com 2019-05-31T21:20:37+00:00 urn:sha1:6cbc3929a54d37bd23cb5efa8e3320ba02f78b2f * Prefixing source files in source/slang with slang- * Prefix source in source/slang with slang- prefix. * Rename core source files with slang- prefix. * Update project files. * Fix problems from automatic merge. 2018-03-16T14:01:35+00:00 Tim Foley tfoleyNV@users.noreply.github.com 2018-03-16T14:01:35+00:00 urn:sha1:93ac152bf8283eff1d1b2ec0f7df897a4e96464f The existing code parsed all of the square-bracket `[attributes]` into `HLSLUncheckedAttribute`, and then went on to hand-convert some of them to specialized subclasses of `HLSLAttribute`. When attributes didn't check, they were left as-is, and no error message was issued, because at the time the compiler was focused on accepting arbitrary input. This change greatly overhauls the handling of `[attributes]`. Attributes are now declared in the stdlib, with declarations like: ```hlsl __attributeTarget(LoopStmt) attribute_syntax [unroll(count: int = 0)] : UnrollAttribute; ``` In this syntax, the `unroll` part is giving the attribute name (the `[]` are just for flavor, to make the declaration look like a use site; we could drop it if we don't like the clutter), the `count` is a parameter of the attribute, which we expect to be of type `int`, and which has a default value of `0` if unspecified. The `: UnrollAttribute` part specifies the meta-level C++ class that will implement this attribute (and corresponds to a class in `modifier-defs.h`). This syntax is similar to our current `syntax` declarations. I'm starting to think we should change it to something like a `__meta_class(UnrollAttribute)` modifier, and then use that uniformly across all cases (e.g., also replacing the curreent `__magic_type(Foo)` syntax). The `__attributeTarget(LoopStmt)` is a modifier that specifies the meta-level C++ class for syntax that this attribute is allowed to attach to. It is legal to have more than one of these. Attributes continue to be parsed in an unchecked form, so that we don't tie up semantic analysis and parsing more than necessary. During checking, we look up the attribute name in the current scope, and then replace the unchecked attribute with a more specific one *if* the checking passes. Checking proceeds in generic and attribute-specific phases. The generic phase includes checking the number of arguments against those specified in the attribute declaration (I don't currently check types, or handle default arguments), and then checking that at least one `__attributeTarget(...)` modifier applies to the syntax node being modified. The attribute-specific phase then applies to the specialized C++ subclass of `Attribute`, and does the actual checking right now (e.g., that step is responsible for actually type-checking things at present). This can obviously be improved over time. With this support I went ahead and added declarations for all the HLSL attributes I could find documented on MSDN. I also added a provisional declaration for the `[shader(...)]` attribute that has been added to dxc, but which is not yet documented. One important detail here is that lookup of attribute names needs to be done carefully, so that we don't let, e.g., local variables shadow an attribute declaration: ```hlsl int unroll = 5; // This attribute should *not* get confused by the local variable `unroll` [unroll] for(...) { .. } ``` The lookup logic already has a notion of a `LookupMask` that can be used to filter declarations out of the result. In this change I surfaced that mask through the main lookup API (rather than requiring a second pass to "refine" lookup results), and made is so that the default lookup mask does *not* include attributes, while an explicit mask can be used to look up *only* attributes. (An alternatie design we discussed was to follow the approach of C# and have the declaration of an attribute like `[unroll]` actually be `unrollAttribute`, with a suffix. I decided not to follow that approach for now because it seemed like printing good error messages in that case could require us to carefully trim the `Attribute` suffix off of names at times, and using the existing mask behavior seemed simpler.) To verify that the shadowing behavior is indeed correct, I modified the `loop-unroll.slang` test case. Smaller notes: * Removed the `HLSL` prefix from several of the C++ attribute classes * Made sure to actually validate the modifiers on statements * Special-cased checking for `ParamDecl` with a null type, because I'm re-using `ParamDecl` for attribute parameters, but can't give a concrete type to some of them right now * Deleting some old, dead emit-from-AST logic around attributes, rather than try to "fix" code that doesn't run (a more complete scrub of that code is still needed) * Fixed AST inheritance hierarchy so that a `Modifier` is a `SyntaxNode` rather than a `SyntaxNodeBase`. I have *no* idea why we have both of those, and we need to clean that up soon. 2017-10-27T18:22:11+00:00 Tim Foley tfoleyNV@users.noreply.github.com 2017-10-27T18:22:11+00:00 urn:sha1:4ab545bcd0716cc3f2da432a921c1f53fdce7925 This change includes a lot of infrastructure work, but the main point is to allow code like the following: ``` // define an interface interface Helper { float help(); } // define a generic function that uses the interface float test<T : Helper>( T t ) { return t.help(); } // define a type that implements the interface struct A : Helper { float help() { return 1.0 } } // define an ordinary function that calls the // generic function with a concrete type: float doIt() { A a; return test<A>(a); } ``` Getting this to generate valid code involves a lot of steps. This change includes the initial version of all of these steps, but leaves a lot of gaps where more complete implementation is required. The changes include: - Member lookup on types has been centralized, and now handles the case where the type we are looking for a member in is a generic parameter (e.g., given `t.help()` we can now look up `help` in `Helper` by knowing that `t` is a `T` and `T` conforms to `Helper`). - There is an obvious cleanup still to be done here where the same exact logic should be used to look up available "constructor" declarations inside a type when the type is used like a function. - Add a notion of subtype constraint "wittnesses" to the type system. When a generic is declared as taking `<T : Helper>` it really takes two generic parameters: the type `T` and a proof that `T` conforms to `Helper`. The actual arguments to a generic will then include both the type argument and a suitable witness argument (both type-level values). - As it stands right now, a witness wraps a `DeclRef` to the declaration that represents the appropriate subtype relationship. So if we have `struct A : Helper`, that `: Helper` part turns into an `InheritanceDecl` member, and a reference to that member can serve as a witness to the fact that `A` conforms to `Helper`. - Make explicit generic application `G<A,B>` synthesize the additional arguments that represent conformances required by the generic. - This does *not* yet deal with the case where a generic is implicitly specialized as part of an ordinary call `G(a,b)` - A bug fix to not auto-specialize generics during lookup. The problem here was related to an attempted fix of an earlier issue. During checking of a method nested in a generic type, we were running into problems where `DeclRefType::create()` was getting called on an un-specialized reference to `vector`, and this was leading to a crash when the code looked for the arguments for the generic. This was worked around by having name lookup automatically specialize any generics it runs into while going through lookup contexts. That choice creates the problem that in a generic method like this: ``` void test<T>(T val) { ... } ``` any reference to `val` inside the body of `test` will end up getting specialized so that it is effectively `test<T>::val`, when that isn't really needed. - Add front-end logic to check that when a type claims to conform to an interface it actually must provide the methods required by the interface. The checking process goes ahead and builds a front-end "witness table" that maps declarations in the interface being conformed to over to their concrete implementations for the type. - At the moment the checking is completely broken and bad: it assumes that *any* member with the right name is an appropriate declaration to satisfy a requirement. That obviously needs to be fixed. - Add an explicit operation to the IR for lookup of methods: `lookup_interface_method(w, r)` where `w` is a reference to the "witness" value and `r` is an `IRDeclRef` for the member we want to look up. - Add an explicit notion of witness tables to the IR. These end up being the IR representation of an `InheritanceDecl` in a type, and they are generated by enumerating the members that satisfy the interface requirements (which were handily already enumerated by the front-end checking). The witness table is an explicit IR value, and so it will be referenced/used at the site where conformance is being exploited (e.g., as part of a `specialize` call), so it should be safe to eliminate witness tables that are unused (since they represent conformances that aren't actually exploited). Similarly, the entries in a witness table are uses of the functions that implement interface methods, and so keep those live. - In order to implement the above, I did a bit of a cleanup pass on the IR representation so that there is an `IRUser` base that `IRInst` inherits from, so that we can have users of values that aren't instructions. - One annoying thing is that because of how types and generics are handled in the IR, we needed a way to have a type-level `Val` that wraps an IR-level value: e.g., to allow an IR-level witness table to be used as one of the arguments for specialization of a generic. The design I chose here is to have a "proxy" `Val` subclass (`IRProxyVal`) that wraps an `IRValue*`. These should only ever appear as part of types and `DeclRef`s that are used by the IR. - One annoying bit here is that an IR value might then have a use that is not manifest in the set of IR instructions, and instead only appears as part of a type somewhere. - I'm not 100% happy with this design, but it seems like we'd have to tackle similar issues if/when we eventually allow functions to have `constexpr` or `@Constant` parameters - Make generic specialization also propagate witness table arguments through to their use sites (this is mostly just the existing substitution machinery, once we have `IRProxyVal`), and then include logic to specialize `lookup_interface_method` instructions when their first operand is a concrete witness table. All of this work allows a single limited test using generics with constraints to pass, but more work is needed to make the solution robust. 2017-09-27T18:17:39+00:00 Tim Foley tfoleyNV@users.noreply.github.com 2017-09-27T18:17:39+00:00 urn:sha1:74f2f47cb63b02638270beecd20acea1a0f5665e * First attempt at a Linux build - Fix up places where C++ idioms were written assuming lenient behavior of Microsoft's compiler - Add a few more alternatives for platform-specific behavior where Windows was the only platform accounted for. - Add a basic Makefile that can at least invoke our build, even if it isn't going good dependency tracking, etc. - Build `libslang.so` and `slangc` that depends on it, using a relative `RPATH` to make the binary portable (I hope) - Add an initial `.travis.yml` to see if we can trigger their build process. * Fixup: const bug in `List::Sort` I'm not clear why this gets picked up by the gcc *and* clang that Travis uses, but not the (newer) gcc I'm using on Ubuntu here, but I'm hoping it is just some missing `const` qualifiers. * Fixup: reorder specialization of "class info" Clang complains about things being specialized after being instantiated (implicilty), and I hope it is just the fact that I generate the class info for the roots of the hierarchy after the other cases. We'll see. * Fixup: add `platform.cpp` to unified/lumped build * Fixup: Windows uses `FreeLibrary` and not `UnloadLibrary` * Fixup: fix Windows project file to include new source file This obviously points to the fact that we are going to need to be generating these files sooner or later. 2017-08-14T21:48:37+00:00 Tim Foley tfoley@nvidia.com 2017-08-14T21:48:37+00:00 urn:sha1:9885c972a6bfa6f856e505cdd90d9b71fdbdadaf Fixes #23 Up to this point, the compiler has used the ordinary `String` type to represent declaration names, which means a bunch of lookup structures throughout the compiler were string-to-whatever maps, which can reduce efficiency. It also means that things like the `Token` type end up carying a `String` by value and paying for things like reference-counting. This change adds a `Name` type that is used to represent names of variables, types, macros, etc. Names are cached and unique'd globally for a session, and the string-to-name mapping gets done during lexing. From that point on, most mapping is from pointers, which should make all the various table lookups faster. More importantly (possibly), this brings us one step closer to being able to pool-allocate the AST nodes. 2017-08-09T17:23:09+00:00 Tim Foley tfoley@nvidia.com 2017-08-09T17:13:40+00:00 urn:sha1:695c2700de54a5fec72ce7214c137a1dc3a02d7b - `ExpressionSyntaxNode` becomes `Expr` - `StatementSyntaxNode` becomes `Stmt` - `StructSyntaxNode` becomes `StructDecl` - `ProgramSyntaxNode` becomes `ModuleDecl` - `ExpressionType` becomes `Type` - Existing fields names `Type` become `type` - There might be some collateral damage here if there were, e.g., `enum`s named `Type`, but I can live with that for now and fix those up as a I see them 2017-08-07T22:16:54+00:00 Tim Foley tfoley@nvidia.com 2017-08-07T21:54:55+00:00 urn:sha1:7b54f43fb1b123f451460edb0add218a0428fe95 There were two main places where global variables were used in the Slang implementation: 1. The "standard library" code was generated as a string at run-time, and stored in a global variable so that it could be amortized across compiles. 2. The representation of types uses some globals (well, class `static` members) to store common types (e.g., `void`) and to deal with memory lifetime for things like canonicalized types. In each case the "simple" fix is to move the relevant state into the `Session` type that controlled their lifetime already (the `Session` destructor was already cleaning up these globals to avoid leaks). For the standard library stuff this really was easy, but for the types it required threading through the `Session` a bit carefully. One more case that I found: there was a function-`static` variable used to generate a unique ID for files output when dumping of intermediates is enabled (this is almost strictly a debugging option). Rather than make this counter per-session (which would lead to different sessions on different threads clobbering the same few files), I went ahead and used an atomic in this case. Note that the remaining case I had been worried about was any function-`static` counter that might be used in generating unique names. It turns out that right now the parser doesn't use such a counter (even in cases where it probably should), and the lowering pass already uses a counter local to the pass (again, whether or not this is a good idea). This change should be a major step toward allowing an application to use Slang in multiple threads, so long as each thread uses a distinct `SlangSession`. The case of using a single session across multiple threads is harder to support, and will require more careful implementation work. 2017-06-30T19:22:33+00:00 Tim Foley tfoley@nvidia.com 2017-06-29T15:55:09+00:00 urn:sha1:cab694dcead92a554654d7fa3f08909d519425f0 - The big change here is that all the definitions for syntax-node classes have been macro-ized, to that we can do light metaprogramming over them - The use of macros for this has big down-sides, but I'm not quite ready to do anything more heavy-weight right now - The macro-ized definitions can be included multiple times, to generate different declarations/code as needed - The first example of using this meta-programming facility is a new visitor system - The actual visitor base classes and the dispatch logic are all generated from the meta-files - There was only one visitor left in the code: the semantics checker, so that was ported to the new system. - All current test cases pass, so *of course* that means all is well. 2017-06-21T18:59:51+00:00 Tim Foley tfoley@nvidia.com 2017-06-21T17:43:25+00:00 urn:sha1:7f7864e80e8b5631ba5ec1aa9657fdaf1b4adb06 The catch with these operations is that they return a vector based on the scalar of the element type of the texture. That is, given `Texture2D<float> t` the operation `t.GatherRed(...)` should return a `float4`. The ideal way to solve this would use associated types, but we aren't there yet, so I am using extension declarations. An extension can "capture" the identity of the element type, like so: __generic<T, let N : int> __extension Texture2D<vector<T,N> > { ... } That extension will match `Texture2D<float3>` and correctly capture `T == float`, so that we can use it in other operations. Getting this working required a bunch of changes: - Actually emit the relevant extension declarations in the stdlib - Fix the parser to be able to parse `Texture2D<vector<T,N> >` (that is, a nested generic app). - I actually went ahead and significantly overhauled the expression parser while I was there, because I just couldn't deal with the existing code any longer. - Added support for general-case lookup to look through `__extension` declarations. I had logic in place to special-case this for looking up "constructors" but hadn't done anything for general member lookup yet. - This required some annoying holes to be punched through the layers, because lookup might need to invoke semantic analysis to ensure that an extension has been checked. - There is some first-pass code trying to support looking up a `typedef` nested inside the `vector` type. This is a nice idea in principle, but the problem is that the `Texture2D<T>` definition would be looking up `T.Element` and not `float4.Element`, and that means we'd need machinery for doing lookup *through* interface conformances for a type parameter like `T` The big gotcha here is that none of this logic applies to `Texture2D<float>` (the original case I mentioned) because I am matching vector types and not scalars. Matching scalars *should* be as easy as: __generic<T : __BuitlinScalarType> __extension Texture2D<T> { ... } But I'd need to confirm that interface constraints like that actually work, or else that extension would *also* apply to `Texture2D<float4>` and break everything. 2017-06-15T23:35:10+00:00 Tim Foley tfoley@nvidia.com 2017-06-15T23:35:10+00:00 urn:sha1:e0389f5a1f32cb611e5a595a5974ee1d5c15f43d For context: a `DeclRef` is supposed to capture both a pointer to a particualr declaration, and also any information needed to specialize that declaration for a context (e.g., generic parameter substitutions). The existing approach had a hiearchy of specialized decl-ref types that mirrored the AST hierarchy, but that led to a lot of boilerplate where you had to recapitulate the exact same hierarchy. The new appraoch basically treats `DeclRef<T>` as a sort of "smart pointer" in that it wraps a pointer to a `T` (the declaration), plus a side field for the specialization info, and then allows it to be cast as needed to other types (where the pointer cast would be allowed), while carrying along the side info. To enable this, all the things that used to be member functions of declaration-reference types are now free functions that take a `DeclRef<T>` for some specific `T` as a parameter.