slang.git/source/slang/type-layout.h, branch master

Use slang- prefix on slang compiler and core source (#973)

2019-05-31T21:20:37+00:00

* Prefixing source files in source/slang with slang- * Prefix source in source/slang with slang- prefix. * Rename core source files with slang- prefix. * Update project files. * Fix problems from automatic merge.

Basic layout and reflection for specialized types (#970)

2019-05-22T22:01:13+00:00

* Basic layout and reflection for specialized types Suppose I have an interface, and a simple implementation of it: ```hlsl interface IModifier { float modify(float value); } struct Doubler : IModifier { float modify(float value) { return 2 * value; } } ``` SAnd now suppose I want to define an implementation that recursively uses the same interface: ```hlsl struct MultiModifier : IModifier { IModifier first; IModifier second; float modify(float value) { value = first.modify(value); value = second.modify(value); return value; } } ``` And now consider that I might have a generic entry point that uses the interface: ```hlsl void myShader( uniform M modifier, ... ) { ... } ``` I can easily specialize `myShader` for `M = Doubler`, but in order to specialze it for `M = MultiModifier` I need a way to specify what the types of `MultiModifier.first` and `.second` should be. That is what the `spReflection_specializeType` function is used to do: take a type like `MultiModifier` and specialize it for, say, `first : Doubler` and `second : Doubler`. That function creates an `ExistentialSpecializedType` that records the base type (`MultiModifier`) and the specialization arguments (the concrete types plus the witness tables that prove they implement the required interfaces). The change that introduced that logic neglected to include an implementation of type layout for `ExistentialSpecializedType`, and also didn't add any support for the new kind of type through the reflection API. That is what this change seeks to rectify. When it comes to layout, a specialized type neeeds to apply layout to its base type (e.g., `MultiModifier`) with the appropriate existential type "slot" arguments bound, which luckily is stuff that type layout already supporst (to handle specialization of interface-type shader parameters). Unlike the case for interface-type shader parameters where the "primary" and "pending" data for a type get propagated up the chain and allocated to different places, a specialized type should be allocated contiguously (e.g., `myShader` is going to assume that the type `M` occupies a contiguous range in memory). The type layout for a specialized type thus computes a layout that is more-or-less a structure type consisting of the "primary" data followed by the "pending" data. This gets wrapped up in a new `ExistentialSpecializedTypeLayout` class. The reflection API then needs to expose an `ExistentialSpecializedTypeLayout` as a new kind of type, and then also provide access to the relevant pieces. For the "base" type, I went ahead and re-used the `getElementType` entry point, just for simplicity (we can debate whether that or a new entry point is more appropriate/convenient). For the actual layout, all that was needed was a way to query the offset for where the "pending" data gets placed, and that is already conveniently encoded as a `VarLayout` field in the `ExistentialSpecializedTypeLayout`. With this change, specialized types are closer to being truly usable, although there is still missing logic in IR lowering because we need to make sure that explicitly specialized types are represented differently from types that are specialized based on global shader parameters. * fixup: review feedback

String/List closer to conventions, and use Index type (#959)

2019-04-29T21:03:46+00:00

* List made members m_ Tweaked types to closer match conventions. * Use asserts for checking conditions on List. Other small improvements. * List.Count() -> getSize() * List Add -> add First -> getFirst Last -> getLast RemoveLast -> removeLast ReleaseBuffer -> detachBuffer GetArrayView -> getArrayView * List:: AddRange -> addRange Capacity -> getCapacity Insert -> insert InsertRange -> insertRange AddRange -> addRange RemoveRange -> removeRange RemoveAt -> removeAt Remove -> remove Reverse -> reverse FastRemove -> fastRemove FastRemoveAt -> fastRemoveAt Clear -> clear * List FreeBuffer -> _deallocateBuffer Free -> clearAndDeallocate SwapWith -> swapWith * List SetSize -> setSize Reserve -> reserve GrowToSize growToSize * UnsafeShrinkToSize -> unsafeShrinkToSize Compress -> compress FindLast -> findLastIndex FindLast -> findLastIndex Simplify Contains * List Removed m_allocator (wasn't used) Swap -> swapElements Sort -> sort Contains -> contains ForEach -> forEach QuickSort -> quickSort InsertionSort -> insertionSort BinarySearch -> binarySearch Max -> calcMax Min -> calcMin * Initializer::Initialize -> initialize List:: Allocate -> _allocate Init -> _init IndexOf -> indexOf * * Put #include in common.h, and remove unneeded inclusions * Small refactor of ArrayView - remove stride as not used * getSize -> getCount setSize -> setCount unsafeShrinkToSize->unsafeShrinkToCount growToSize -> growToCount m_size -> m_count * Some tidy up around Allocator. * Use Index type on List. * Refactor of IntSet. First tentative look at using Index. * Made Index an Int Did preliminary fixes. Made String use Index. * Partial refactor of String. * String::Buffer -> getBuffer ToWString -> toWString * Small improvements to String. String:: Buffer() -> getBuffer() Equals() -> equals * Try to use Index where appropriate. * Fix warnings on windows x86 builds.

Feature/uint int definition (#954)

2019-04-25T19:00:36+00:00

* * Moved CPU determination macros to slang.h * Determine SlangUInt/SlangInt from the pointer width (determined from CPU macros) * Removed the UnambiguousInt and UnambigousUInt types - as a previous fragile work around * Removed UInt/Int definition from smart-pointer.h as now in common.h * * Remove ambiguity for PrettyWriter and ints * Improve comment around SlangInt/UInt * More fixes around ambiguity with PrettyWriter and integral types. * Disable VK on OSX. * Force CI to rebuild as spurious error.

Allow plugging in types with resources for interface parameters (#913)

2019-03-26T19:49:02+00:00

* Allow plugging in types with resources for interface parameters The key feature enabled by this change is that you can take a shader declared with interface-type parameters: ```hlsl ConstantBuffer gLight; float4 myShader(IMaterial material, ...) { ... } ``` and specialize its interface-type parameters to concrete type that can contain resources like textures, samplers, etc. The hard part of doing this layout is that we need to support signatures that include a mix of interface and non-interface types. Imagine this contrived example: ```hlsl float4 myShader( Texture2D diffuseMap, ILight light, Texture2D specularMap) { ... } ``` We end up wanting `diffuseMap` to get `register(t0)` and `specularMap` to get `register(t1)`, so that they have the same location no matter what we plug in for `light`. But if we plug in a concrete type for `light` that needs a texture register, we need to allocate it *somewhere*. We handle this by having the `TypeLayout` for `light` come back with a "primary" type layout that doesn't have any texture registers, but with a "pending" type layout that includes the texture register requirements of whatever concrete type we plug in. This split between "primary" and "pending" layout then needs to work its way up the hierarchy, so that an aggregate `struct` type with a mix of interface and non-interface fields (recursively), needs to compute an aggregate "primary type layout" and an aggregate "pending type layout," and then each field needs to be able to compute its offset in the primary/pending layout of the aggregate. A large chunk of the work in this PR is then just implementing the split between primary and pending data, and ensuring that layouts are computed appropriately. The next catch is that when a "parameter group" (either a parameter block or constant buffer) contains one or more values of interface type, then we can allow the parameter group to "mask" some of the resource usage of the concrete types we plug in, but others "bleed through." For example, if we have: ```hlsl struct MyStuff { float3 color; ILight light; } ConstantBuffer myStuff; struct SpotLight { float3 position; Texture2D shadowMap; } `` If we plug in the `SpotLight` type for `myStuff.light`, then the `float3` data for the light can be "masked" by the fact that we have a constant buffer (we can just allocate the `float3` `position` right after `color`), but the `Texture2D` needed for `shadowMap` needs to "bleed through" and become "pending" data for the `myStuff` shader parameter. Adding support for that detail more or less required a full rewrite of the logic for allocating parameter group type layouts. The next detail is that when we go to legalize a declaration like the `myStuff` buffer, we will end up with something like: ```hlsl struct MyStuff_stripped { float3 color; } struct Wrapped { MyStuff_stripped primary; SpotLight pending; } ConstantBuffer myStuff; ``` This "wrapped" version of the buffer type more accurately reflects the layout we need/want for the uniform/ordinary data, but in order to further legalize it and pull out the resource-type fields like `shadowMap` we need to have accurate layout information, and the problem is that layout information for the original buffer can't apply to this new "wrapped" buffer. The last major piece of this change is logic that runs during existential type legalization to compute new layouts for "wrapped" buffers like these that embeds correct offset/binding/register information for any resources nested inside them. A key challenge in that code is that existential legalization needs to erase any "pending" data from the program entirely, so that offset information that used to be relatie to the "pending" part of a surrounding type now needs to be relative to the primary part. The work here may not be 100% complete for all scenarios, but it does well enough on the new and existing tests that I want to checkpoint it. Note that a few other tests have had their output changed, but in all cases I've reviewed the diffs and determined that the change in observable behavior is consistent with what we intened Slang's behavior to be. Note that there is still one major piece of support for interface-type parameters that is missing here, and which might force us to revisit some of the decisions in this code: we don't properly support user-defined `struct` types with interface-type fields. * fixup: typos

Improve support for interfaces as shader parameters (#886)

2019-03-09T00:24:02+00:00

* Improve support for interfaces as shader parameters This change adds two main things over the existing support: 1. It is now possible to plug in concrete types that actually contain (uniform/ordinary) fields for the existential type parameters introduced by interface-type shader parameters. The `interface-shader-param2.slang` test shows that this works. 2. There is a limited amount of support for doing correct layout computation and generating output code that matches that layout, so that interface and ordinary-type fields can be interleaved to a limited extent. The `interface-shader-param3.slang` test confirms this behavior. There are several moving pieces in the change. * When it comes to terminology, we try to draw a more clear distinction between existial type parameters/arguments and existential/object value parametes/arguments. A simple way to look at it is that an `IFoo[3]` shader parameter introduces a single existential type parameter (so that a concrete type argument like `SomeThing` can be plugged in for the `IFoo`) but introduces three existential object/value parameters (to represent the concrete values for the array elements). * At the IR level, we support a few new operations. A `BindExistentialsType` can take a type that is not itself an interface/existential type but which depends on interfaces/existentials (e.g., `ConstantBuffer`) and plug in the concrete types to be used for its existential type slots. * Then a `wrapExistentials` instruction can take a type with all the existentials plugged in (possibly by `BindExistentialsType`) and wrap it into a value of the existential-using type (e.g., turn `ConstantBuffer` into a `ConstantBuffer`). * The IR passes for doing generic/existential specialization have been updated to be able to desugar uses of these new operations just enough so that a `ConstantBuffer` can be used. * When we specialize an IR parameter of an interface type like `IFoo` based on a concrete type `SomeThing`, we turn the parameter into an `ExistentialBox` to reflect the fact that we are conceptually referring to `SomeThing` indirectly (it shouldn't be factored into the layout of its surrounding type). * Parameter binding was updated so that it passes along the bound existential type arguments in a `Program` or `EntryPoint` to type layout, so that we can take them into account. The type layout code needs to do a little work to pass the appropriate range of arguments along to sub-fields when computing layout for aggregate types. * Type layout was updated to have a notion of "pending" items, which represent the concrete types of data that are logically being referenced by existential value slots. The basic idea is that these values aren't included in the layout of a type by default, but then they get "flushed" to come after all the non-existential-related data in a constant buffer, parameter block, etc. * The logic for computing a parameter group (`ConstantBuffer` or `ParameterBlock`) layout was updated to always "flush" the pending items on the element type of the group, so that the resource usage of specialized existential slots would be taken into account. * The type legalization pass has been adapted so that we can derive two different passes from it. One does resource-type legalization (which is all that the original pass did). The new pass uses the same basic machinery to legalize `ExistentialBox` types by moving them out of their containing type(s), and then turning them into ordinary variables/parameters of type `T`. Big things missing from this change include: - Nothing is making sure that "pending" items at the global or entry-point level will get proper registers/bindings allocated to them. For the uniform case, all that matters in the current compiler is that we declare them in the right order in the output HLSL/GLSL, but for resources to be supported we will need to compute this layout information and start associating it with the existential/interface-type fields. - Nothing is being done to support `BindExistentials` where `S` is a `struct` type that might have existential-type fields (or nested fields...). Eventually we need to desugar a type like this into a fresh `struct` type that has the same field keys as `S`, but with fields replaced by suitable `BindExistentials` as needed. (The hard part of this would seem to be computing which slots go to which fields). As a practial matter, this missing feature means that interface-type members of `cbuffer` declarations won't work. The current tests carefully avoid both of these problems. They don't declare any buffer/texture fields in the concrete types, and they don't make use of `cbuffer` declarations or `ConstantBuffer`s over structure types with interface-type fields. * fixup: add override to methods * fixup: typos

Move enumeration of shader parameters to Program/EntryPoint (#870)

2019-03-01T17:43:47+00:00

There's a certain amount of logic in `parameter-binding.cpp` that just has to do with the basic problem of enumerating the shader parameters of a `Program`. The main source of complexity is that for legacy/compatibility reasons we need to consider two shader parameters with the same name as being the "same" parameter for layout purposes, and then we need to do a bunch of validation to ensure that these parameters have compatible types. The biggest part of this change is moving that logic to `Program`, so that it builds up a list of its shader parameters during the front-end work, so that any errors related to bad redeclarations will now come up even if we aren't generated target-specific layouts/code. All of the code for `getReflectionName`, `StructuralTypeMatchStack`, etc. is pretty much copy-pasted from `parameter-binding.cpp` over to `check.cpp`, with the `ParameterBindingContext` replaced with a `DiagnosticSink`. The `Program::_collectShaderParameters` function (renamed from `_collectExistentialParams`) then deals with the enumeration and deduplication logic that used to happen in `collectGlobalScopeParameters()`. The new declarations in `compiler.h` reveal the underlying reason for this change: by letting `Program` and `EntryPoint` handle the canonical enumeration of parameters, we can associate each parameter with the range of existential type slots it uses, which will simplify certain work around interfaces (not in this change...). Moving the code out of parameter binding and into `check.cpp` revealed some unused GLSL-related code that I removed while I was at it. I also found that the `IsDeclaration` case of `VarLayoutFlag` wasn't actually being used, so I went ahead and removed it (we can easily re-add it if we ever find a need for it). Overall this isn't a big cleanup (mostly just code moving, rather than being eliminated), but it will facilitate other changes, and it seems cleaner overall to do this work once in target-independent logic, rather than per-target.

A small refactor to how implicit constant buffers are getting created. (#871)

2019-03-01T16:20:09+00:00

This affects layout computation for both the global and entry-point scopes, where multiple discrete shader parameters can be declared, but for layout purposes they must be treated as if they lived in the same `struct` type. If that `struct` type ends up consuming any "ordinary" data (`LayoutResourceKind::Uniform`) then an implicit constant buffer will be needed for that scope (e.g., the way fxc produces a `$Globals` constant buffer for the global scope). The logic for computing those scope layouts had a bug in it, in that the struct type was not being updated to have the right size for uniform data at the scope. That bug hasn't bitten anybody yet because no Slang users are relying on entry-point uniforms, and global-scope uniforms aren't fully implemented (and get diagnosed as an error elsewhere in the compiler). This change fixes that bug. This change also refactors things so that the logic for creating a constant buffer layout if and only if needed is moved into `type-layout.cpp` instead of relying on `parameter-binding.cpp` to compute whether or not it needs a block on its own. This is anticipating the rules for deciding whether or not a constant buffer is needed being slightly more thorny once interface types are in the mix.

Simplify type layout (#867)

2019-02-27T22:58:02+00:00

* Make vector/matrix type layouts include element type layouts Previously the `MatrixTypeLayout` class was a leaf node in the layout hierarchy, and vector types just used `TypeLayout` with no further refinement. This change adds a `VectorTypeLayout`, and makes all of vector, matrix, and array types inherit form a common base class for `SequenceTypeLayout`s. The actual layout computation logic was updated to compute layouts for the element types of vectors, and for the row (and element) types of matrices. Notes: * Because of the way varying input/output parameters are being handled, their type layouts won't include this new information, and they will just use `TypeLayout`. This was true even for the matrix case before. * I made the design choice in this change to have a matrix type always treat rows as the logical element type (since that is what is surfaced to ther user in the HLSL syntax). We could potentially make our lives easier during layout computation if we made the element type of a `MatrixTypeLayout` depend on the row-/column-major layout choice, but that would then require any algorithm that uses the new layout information to take row-vs-column-major into account. * No code is actually *using* this new information yet, although the work in `ir-union.cpp` could probably benefit from it. The main expected use case going forward is representing constant buffers as a "bag of bits." * Add a specialized type layout approach for varying parameters There is a lot of complexity in `GetLayoutImpl` because it needs to service both the "normal" case, which always wants a `TypeLayout` object to be returned, and the varying parameter case, where we currently don't care about getting back a `TypeLayout` object. Confusingly, the varying parameter layout logic actually *does* construct `TypeLayout` objects, and it just does it inside of `parameter-binding.cpp` rather than in `type-layout.cpp`. That logic cannot (easily) be shared with the `GetLayoutImpl` path because: * The varying case needs to deal with system-value semantics and also parameters that may be inputs, outputs, or both (so that they need to combine resource usage computed for inputs and outputs). * The varying case needs to special-case vectors (and to a lesser extent matrices) because of the quirks of uniform layout (e.g., four `float` varying inputs consume four `locations`, but a `float4` consumes only one location). This change introduces a customized layout function just for varying parameters, that only handles the scalar, vector, and matrix cases (since `parameter-binding.cpp` will have recursed through any strucures/arrays, and should error out on any other types that are illegal in varying parameter lists). In the long run we could consider trying to deduplicate code and share more of this logic with `GetLayoutImpl`, but that would require a more significant refactoring of type layout, which should probably wait until we are doing layout on IR types. * Rename CreateTypeLayout to createTypeLayout This is just a fixup to better reflect our established naming conventions. * Simplify type layout so that it always returns a layout object The core `GetLayoutImpl` routine included a fair bit of complexity to deal with the fact that its `outTypeLayout` parameter was optional. The caller could pass in null to say that it doesn't want a `TypeLayout` object to be constructed, and the routine would conditionalize a lot of its logic to deal with this case. This change simplifies the logic so that a `TypeLayout` is always constructed and returned. Instead of using a combination of a function result (for the `SimpleLayoutInfo`) and an output parameter (for the `TypeLayout`) we use a new `TypeLayoutResult` that acts as a tuple over the two. I had hoped for a more significant cleanup by also eliminating the need to return the `SimpleLayoutInfo` separately from the `TypeLayout`, but the simple layout info is what the underlying per-API/-context "rules" implementations use (so that they can avoid all the complexity of `TypeLayout`), and refactoring to derive the simple layout infor from a computed `TypeLayout` would be a bigger refactoring than I was ready for. * fixup: typos

Split front- and back-ends (#846)

2019-02-15T17:08:19+00:00

* Split front- and back-ends This change is a major refactor of several of the types that provide the behind-the-scenes implementation of the public C API. The goal of this refactor is primarily to allow for future API services that let the user operate both the front- and back-ends of the compiler in a more complex fashion. For example, as user should be able to compile a bunch of source code into modules, look up types, functions, etc. in those modules, specialize generic types/functions to the types they've looked up, and then finally request target code to be gernerated for specialized entry points. The back-end code generation they trigger should re-use the front-end compilation work (parsing, semantic checking, IR generation) that was already performed. The most visible change is that `CompileRequest` has been split up into several smaller types that take responsibility for parts of what it did: * The `Linkage` type owns the storage for `import`ed modules, and well as the `TargetRequest`s that represent code-generation targets. The intention is that an application could use a single `Linkage` for the duration of its runtime (so long as it was okay with the memory usage), so that each `import`ed module only gets loaded once. For now, this type needs to manage the search paths, file system, and source manager, because of its responsibility for loading files. * A `FrontEndCompileRequest` owns the stuff related to parsing, semantic checking, and initial IR generation. This most notably includes the `TranslationUnitRequest`s and the `FrontEndEntryPointRequest`s (which used to be just `EntryPointRequest`s). It's main job is to produce AST and IR modules for each translation unit, and to find and validate the entry points. The front-end request does *not* interact with generic arguments for global or entry-point generic parameters. * The main output of both `import` operations and front-end translation units is the `Module` type, which is just a simple container for both the AST module (to service the reflection/layout APIs, and also for semantic checking of code that `import`s the module) and the IR module (for linking and code generation). This type captures the commonalities between the old `LoadedModule` (which is now just an alias for `Module`) and `TranslationUnitRequest` (which now owns a `Module`). * The secondary output of front-end compilation is a `Program`, which comprises a list of referenced `Module`s and validated `EntryPoint`s that will be used together. Layout and code generation both need a `Program` to tell them what modules and entry points will be used together (we don't want to just code-gen everythin that has ever been loaded into the linakge). The `Program`s created by the front-end do not include generic arguments, so they may provide incomplete layout information and/or be unsuitable for code generation. * A `BackEndCompileRequest` owns stuff related to turning a `Program` into output kernels for the targets of a `Linkage`. Most of the data it owns beyond the `Program` to be compiled is minor, so this is a good candidate for demotion from a heap-allocated object to just a `struct` of options that gets passed around. * The `CompileRequestBase` type is an attempt to wrap up the common functionality of both front-end and back-end compile requests. Most of it is just exposing the availability of a linkage and `DiagnosticSink`, so this type is a good candidate for subsequent removal. The main interesting thing it has is the flags related to dumping and validation of IR, so there is probably a good refactoring still to be made around deciding how options should be handled going forward. * Behind the scenes, the `Program` type is set up to handle some level of on-line compilation and layout work. The `Program` knows the `Linkage` it belongs to, and allows for a `TargetProgram` to be looked up based on a specific `TargetRequest`. A `TargetProgram` then allows layout information and compiled kernel code to be asked for on-demand, in order to support eventual "live" compilation scenarios. * The `EndToEndCompileRequest` type is a composition/coordination type that replaces the old `CompileRequest` in a way that uses the services of the various other types. It owns a few pieces of state that only make sense in the context of an end-to-end compile (e.g., there is really no way to "pass through" code when the front- and back-ends are run separately) or a command-line compile (everything to do with specifying output paths for files is really just for the benefit of `slangc`, and might even be moved there over time). * One important detail is that the `EndToEndCompilRequest` owns all of the string-based generic arguments for both global and entry-point generic parameters. The logic in `check.cpp` for dealing with those arguments has been heavily refactored to separate out the parsings steps that are specific to end-to-end compilation with string-based type arguments, and the semantic checking steps that result in a specialized `Program` (which can be exposed through new APIs that aren't tied to end-to-end compilation). It is perhaps not surprising that this change had a lot of consequences, so I'll briefly run over some of the main categories of changes required: * I changed the way that global generic arguments are passed via API (use `spSetGlobalGenericArgs` instead of the generic arguments for `spAddEntryPointEx`, which are not just for entry-point generics), which has been a change that we've needed for a long time. This is technically a breaking API change, although we should have very few client applications that care about it. * A bunch of places that used to take "big" objects like `CompileRequest` now just take the sub-pieces they care about (e.g., a function might have only needed a `Linkage` and a `DiagnosticSink`). This makes many subroutines or "context" struct types more generally useful, at the cost of taking more parameters. * In a few cases the conceptually clean separation of the layers breaks down (often for edge-case or compatibility features), and so we may pass along additional objects that are allowed to be null, but are used when present. A big example of this is how the back-end code generation routines accept an `EndToEndCompileRequest` that is optional, and only used to check whether "pass through" compilation is needed. We should probably look into cleaning this kind of logic up over time so that we don't need to violate the apparent separation of phases of compilation. * In cases where separation of layers was being broken for the sake of GLSL features, I went ahead and ripped them out, since all of that should be dead code anyway. * In many cases I increased the encapsulation of data in the core types to help track down use sites and make sure they are following invariants better. * In cases where code was doing, e.g., `context->shared->compileRequest->session->getThing()` I have tried to introduce convenience routines so that the usage site is just `context->getThing()` to improve encapsulation and allow changes to be made more easily going forward. * The `noteInternalErrorLoc` functionality was moved off of the compile request and into `DiagnosticSink`, since that is the one type you can rely on having around when you want to note an internal error. We may consider going forward if (and how) it should reset the counter used for noting locations on internal errors. * A few APIs now take `DiagnosticSink*` arguments where they didn't before, and as a result some public APIs need to create `DiagnosticSink`s to pass in, before going ahead and ignoring the messages. In the future there should be variations of these APIs that accept an `ISlangBlob**` parameter for the output. * fixup: missing include for compilers with accurate template checking (non-VS) * fixup: review feedback