From 5f1ba7b614b64d24fb45a3815a9867c68dace466 Mon Sep 17 00:00:00 2001 From: Yong He Date: Thu, 5 Sep 2024 15:12:52 -0700 Subject: Various documentation improvements. (#5017) --- docs/proposals/001-basic-interfaces.md | 254 ----- docs/proposals/001-where-clauses.md | 144 ++- docs/proposals/002-api-headers.md | 952 ----------------- docs/proposals/003-error-handling.md | 296 ------ docs/proposals/004-com-support.md | 240 ----- docs/proposals/005-components.md | 507 --------- docs/proposals/006-artifact-container-format.md | 1119 -------------------- docs/proposals/legacy/001-basic-interfaces.md | 254 +++++ docs/proposals/legacy/002-api-headers.md | 952 +++++++++++++++++ docs/proposals/legacy/003-error-handling.md | 296 ++++++ docs/proposals/legacy/004-com-support.md | 240 +++++ docs/proposals/legacy/005-components.md | 507 +++++++++ .../legacy/006-artifact-container-format.md | 1119 ++++++++++++++++++++ docs/user-guide/06-interfaces-generics.md | 71 ++ docs/user-guide/09-targets.md | 6 +- docs/user-guide/a2-01-spirv-target-specific.md | 30 +- docs/user-guide/toc.html | 4 + 17 files changed, 3536 insertions(+), 3455 deletions(-) delete mode 100644 docs/proposals/001-basic-interfaces.md delete mode 100644 docs/proposals/002-api-headers.md delete mode 100644 docs/proposals/003-error-handling.md delete mode 100644 docs/proposals/004-com-support.md delete mode 100644 docs/proposals/005-components.md delete mode 100644 docs/proposals/006-artifact-container-format.md create mode 100644 docs/proposals/legacy/001-basic-interfaces.md create mode 100644 docs/proposals/legacy/002-api-headers.md create mode 100644 docs/proposals/legacy/003-error-handling.md create mode 100644 docs/proposals/legacy/004-com-support.md create mode 100644 docs/proposals/legacy/005-components.md create mode 100644 docs/proposals/legacy/006-artifact-container-format.md (limited to 'docs') diff --git a/docs/proposals/001-basic-interfaces.md b/docs/proposals/001-basic-interfaces.md deleted file mode 100644 index 4d04cbe04..000000000 --- a/docs/proposals/001-basic-interfaces.md +++ /dev/null @@ -1,254 +0,0 @@ -Basic Interfaces -================ - -The Slang standard library is in need of basic interfaces that allow generic code to be written that abstracts over built-in types. -This document sketches what the relevant interfaces and their operations might be. - -Status ------- - -In discussion. - -Background ----------- - -One of the first things that a user who comes from C++ might try to do with generics in Slang is write an operation that works across `float`s, `double`s, and `half`s: - -``` -T horizontalSum( vector v ) -{ - return v.x + v.y + v.z + v.w; -} -``` - -A function like `horizontalSum` does not compile because without a constraint on the type parameter `T`, the compiler has no reason to assume that `T` supports the `+` operator. -A new user is often stymied at this point, because no appropriate `interface` seems to exist, and there does not appear to be a way to *define* an appropriate interface. - -As a user gets more experienced with Slang, they may learn how to use `extension`s to define something nearly suitable: - -``` -interface IMyAddable { This add(This rhs); } - -extension float : IMyAddable { float add(float rhs) { return this + rhs; } } -// ... - -T horizontalSum( vector v ) -{ - return v.x.add(v.y).add(v.z).add(v.w); -} -``` - -While that approach works (or should work), it requires a user to know how to use `extension`s and the `This` type, which are complicated even for experienced users. The resulting code is also less readable because it uses `.add(...)` instead of the ordinary `+` operator. - -Many more users end up finding out about the `__BuiltinFloatingPointType` interface, and write something like: - -``` -T horizontalSum( vector v ) -{ - return v.x + v.y + v.z + v.w; -} -``` - -This alternative is much more palatable to users, but it results in them using a double-underscored interface (which we consider to mean "implementation details that are subject to change"). Users often get tripped up when they find out that certain operations that make sense to be available through `__BuiltinFloatingPointType` are not available (because those operations were not needed in the definition of the stdlib, which is what the `__` interfaces were created to support). - -Related Work ------------- - -There are several languages that have constructs similar to our `interface`s, and which provide built-in interfaces for simple math operations that are suitable for use with the built-in types provided by the language. - -Existing solutions can be broadly categorized based on whether their built-in interfaces are related to semantic/mathematical structures, or are purely about specific classes of operators. - -Haskell and Swift are both examples of languages where the built-in interfaces are intended to be semantic. Haskell provides type classes such as `Additive`, `Ring`, `Algebraic`, `RealTranscendental`, etc. -Swift is similar (although it provides a less complete hierarchy of algebraic structures than Haskell), but also includes more detail amount machine number representations, so that it has `BinaryFloatingPoint`, `FixedWidthInteger`, etc. - -Rust is in the other camp, where it has a built-in interface to correspond to each of its overloadable operators. The `Add` and `Sub` traits allow the built-in `+` and `-` operators to be overloaded for a user-defined type, but impose no implicit or explicit semantic expectations on those operations. - -It may help to describe a concrete example of how the difference between the two camps affects design. The Rust `Add` trait is implemented by the left-hand-side type, and does not constrain the right-hand side or result type of an addition. A Rust programmer may implement `Add` for a type `X` so that `x + ...` expects a right-hand-side operand of some other type `Y` and produces a result of yet *another* type `Z`. Knowing that `X` supports the `Add` trait does *not* mean that it is possible to take the sum of a list of `X`s, because there is no guarantee that `x0 + x1` is valid, or that `X` has a logical "zero" value that could be used as the sum of an empty list. - -In contrast, in Swift a type `X` that conforms to `AdditiveArithmetic` must provide a `+` operation that takes two `X` values and yields an `X`. It also requires that `X` provide a `static` property `zero` of type `X`, to represent its zero value. As a result, it is possible to write a generic function in switch that can compute the sum of a list of `T` values, provided `T` conforms to `AdditiveArithmetic`. - -Proposed Approach ------------------ - -Slang supports operators as ordinary overloadable functions, so the rationale behind the Rust operator traits does not seem to apply. We propose to implement a modest hierarchy of numeric interfaces in the style of Haskell/Swift. - -### Changes to Operator Lookup - -Currently, when Slang encounters an operator invocation like `a + b`, it treats this as more or less equivalent to a function call `+( a, b )`. The compiler looks up `+` in the current lexical environment, and then applies overload resolution to the result of lookup. - -We propose that the rules in that case should be changed so that lookup *also* perform lookup of the operator (`+` in this case) in the context of the static types of `a` and `b`. That change would in theory allow "operator overloads" to be defined as `static` functions within a type they apply to (whether on the left-hand or right-hand side). As a consequence, such a change would also mean that `interface`s could conveniently include operator overloads as requirements. - -### IAdditive - -The `IAdditive` interface is for types where addition, subtraction, and zero have meaning. - -``` -interface IAdditive -{ - // The zero value for this type - static property zero : This { get; } - - // Add two values of this type - static func +(left: This, right: This) -> This; - - // Subtract two values of this type - static func -(left: This, right: This) -> This; -} -``` - -### INumeric - -The `INumeric` interface is for types that are more properly number-like. -Note that this interface does not define division, because the division operations on integers and floating-point numbers are sufficiently different in semantics. - -``` -interface INumeric : IAdditive -{ - // Initialize from an integer - __init< T : IInteger >( T value ); - - // Multiply two values of this type - static func *(left: This, right: This) -> This; -} -``` - -### ISignedNumeric - -Only signed numbers logically support negation (although we all know it also gets applied to unsigned numbers, where it has meaningful and use semantics). - -``` -interface ISignedNumeric : INumeric -{ - // Negate a value of this type - static prefix func -(value: This) -> This; -} -``` - - -### IInteger - -The `IInteger` interface codifies the basic things that a generic wants to be able to access for any integer type. - -``` -interface IInteger : INumeric -{ - // Smallest representable value - static property minValue : This { get; } - - // Largest representable value - static property maxValue : This { get; } - - // Initialize from a floating-point value - // (what rounding mode? round-to-nearest-even?) - __init< T : IFloatingPoint >( T value ); - - - // Integer quotient - static func /(left: This, right: This) -> This; - - // Integer remainder (or is it modulus? or is it undefined which?) - static func %(left: This, right: This) -> This; -} -``` -### IUnsignedInteger - -``` -interface IUnsignedInteger : IInteger -{ - -} -``` - -### ISignedInteger - -The main interesting thing we'd want from a signed integer type is to be able to convert it to the same-size unsigned integer type. - -``` -interface ISignedInteger : IInteger, ISignedNumeric -{ - // Equivalent unsigned type (can always hold magnitude) - associatedtype Unsigned : IUnsignedInteger; - - // Get the magnitude of this value (may not be representable - // as `This` type, if it is `minValue`) - property magnitude : Unsigned { get; }; -} -``` - -### IFloatingPoint - -The `IFloatingPoint` interface provides the minimum of what users expect a floating-point type to support. -It includes the ability to check for special values (not-a-number, infinities), as well as the value of various standard constants. - -``` -interface IFloatingPoint : INumeric, ISignedNumeric -{ - property isFinite : bool { get; } - property isInfinite : bool { get; } - property isNaN : bool { get; } - property isNormal : bool { get; } - property isDenormal : bool { get; } - - // TODO: breaking into magnitude/exponent - - static property infinity : This { get; } - static property nan : This { get; } - static property pi : This { get; } - - // TODO: min/max finite values, smallest non-zero value, etc. - - // Initialize from another floating-point value. - __init< T : IFloatingPoint >( T value ); - - // Floating-point division - static func /(left: This, right: This) -> This; -} -``` - -### ISpecialFunctions - -The `ISpecialFunctions` interface is for floating-point types that also have full support for the standard suite of special functions provided by something like ``. -It is pulled out as a distinct interface from `IFloatingPoint` because many platforms support floating-point types like `double` without also having full support for special functions on those types. - -``` -interface ISpecialFunctions : IFloatingPoint -{ - static This cos(This value); - static This sin(This value); - // TODO: fill this out -} -``` - -Questions ---------- - -### Should these all be `IBuiltin*`? Should we have separate interfaces for built-in and user types? - -The main reason for the current `__Builtin` interfaces is that it allows us to define built-in functions that are generic over those interfaces, but which map to a single instruction in the Slang IR. The relevant operations are not currently defined as - -### What should the naming convention be for `interface`s in Slang? - -These would be the first `interface`s officially exposed by the standard library. -While most of our existing code written in Slang uses an `I` prefix as the naming convention for `interface`s (e.g., `IThing`), we have never really discussed that choice in detail. -Whatever we decide to expose for this stuff is likely to become the de facto convention for Slang code. - -The `I` prefix is precedented in COM and C#/.net/CLR, which are likely to be familiar to many devleopers using Slang. -Because of COM, it is also the convention used in the C++ API headers for Slang and GFX. - -The Rust/Swift languages do not distinguish between traits/protocols and other types. -This choice is intentional, and it might be good to understand the motivation behind it. -At least one potential benefit to not distinguishing such types is that beginning programmers can write code that is "more generic" than they might otherwise write. - -Alternatives Considered ------------------------ - -One important alternative is to follow the precedent of Rust and avoid basing these interfaces on semantic structures. -That choice is important in Rust in part because there is no way for a type to support an operator other than by implementing the built-in operator traits. -If the operator traits had prescriptive semantics, they might cause problems for types that want to support the operators but cannot fit within the semantic constraints. -In contrast, Slang allows operator overloads to be defined independent of interfaces (they are orthogonal features), so there is no risk of developers being "locked in" by our attempts to provide richer interfaces. - -Conversely, one could worry that our interfaces do not provide *enough* semantics. We may find that users need additional interfaces that sit "in between" these ones, or that carve up the same operations into smaller units. -This proposal contends that we need to have *something* in this space, and that it doesn't make sense to try to get these interfaces 100% perfect until we've had some lived experience with them. -Fortunately, the Slang language is not yet at a point of trying to guarantee perfect source stability of these interfaces, nor anything like strong binary compatibility guarantees. -If we make mistakes here, we have time to fix them. - diff --git a/docs/proposals/001-where-clauses.md b/docs/proposals/001-where-clauses.md index c16d283b5..086248386 100644 --- a/docs/proposals/001-where-clauses.md +++ b/docs/proposals/001-where-clauses.md @@ -6,7 +6,12 @@ We propose to allow generic declarations in Slang to move the constraints on gen Status ------ -Unimplemented. +Status: Partially implemented. The only unimplemented case is the canonicalization of generic constraints. + +Implementation: [PR 4986](https://github.com/shader-slang/slang/pull/4986) + +Reviewed by: Theresa Foley, Yong He + Background ---------- @@ -24,18 +29,17 @@ Introducing `where` clauses allows a programmer to state the constraints *after* void resolve(ResolutionContext context, List stuffToResolve, out V destination) where T : IResolvable, - U : IResolver, - V : IResolveDestination + where U : IResolver, + where V : IResolveDestination { ... } This latter form makes it easier to quickly glean the overall shape of the function signature. -A second important benefit of `where` clauses is that they open the door to expressing more complicated constraints on and between type parameters. -While this document does not propose to allow any new forms of constraints right away, we can imagine things like allowing constraints on *associated types*, e.g.: +A second important benefit of `where` clauses is that they open the door to expressing more complicated constraints on and between type parameters, such as allowing constraints on *associated types*, e.g.: void writePackedData(T src, out U dst) where T : IPackable, - T.Packed : IWritable + where T.Packed : IWritable { .. } Related Work @@ -92,12 +96,13 @@ Proposed Approach For any kind of declaration that Slang allows to have generic parameters, we will allow a `where` clause to appear after the *header* of that declaration. A `where` clause consists of the (contextual) keyword `where`, following by a comma-separated list of *constraints*: - - struct MyStuff : Base, IFoo - where T : IFoo, - T : IBar +```csharp + struct MyStuff : IFoo + where T : IFoo, IBar + where T : IBaz + where U : IArray { ... } - +``` A `where` clause is only allowed after the header of a declaration that has one or more generic parameters. Each constraint must take the form of one of the type parameters from the immediately enclosing generic parameter list, followed by a colon (`:`), and then followed by a type expression that names an interface or a conjunction of interfaces. @@ -105,33 +110,49 @@ Multiple constraints can be defined for the same parameter. We haven't previously defined what the header of a declaration is, so we briefly illustrate what we mean by showing where the split between the header and the *body* of a declaration is for each of the major kinds of declarations that are supported. In each case a comment `/****/` is placed between the header and body: - // variables: - let v : Int /****/ = 99; - var v : Int /****/ = 99; - Int v /****/ = 99; - - // simple type declarations: - typealias X : IFoo /****/ = Y; - associatedtype X : IFoo /****/; - - // functions and other callables: - Int f(Float y) /****/ { ... } - func f(Float y) -> Int /****/ { ... } - init(Float y) /****/ { ... } - subscript(Int idx) -> Float /****/ { ... } - - // properties - property p : Int /****/ { ... } - - // aggregates - extension Int : IFoo /****/ { ... } - struct Thing : Base /****/ { ... } - class Thing : Base /****/ { ... } - interface IThing : IBase /****/ { ... } - enum Stuff : Int /****/ { ... } - +```csharp +// variables: +let v : Int /****/ = 99; +var v : Int /****/ = 99; +Int v /****/ = 99; + +// simple type declarations: +typealias X : IFoo /****/ = Y; +associatedtype X : IFoo /****/; + +// functions and other callables: +Int f(Float y) /****/ { ... } +func f(Float y) -> Int /****/ { ... } +init(Float y) /****/ { ... } +subscript(Int idx) -> Float /****/ { ... } + +// properties +property p : Int /****/ { ... } + +// aggregates +extension Int : IFoo /****/ { ... } +struct Thing : Base /****/ { ... } +class Thing : Base /****/ { ... } +interface IThing : IBase /****/ { ... } +enum Stuff : Int /****/ { ... } +``` In practice, the body of a declaration starts at the `=` for declarations with an initial-value expression, at the opening `{` for declarations with a `{}`-enclosed body, or at the closing `;` for any other declarations. +With introduction of `where` clauses, we can extend type system to allow more kinds of type constraints. In this proposal, +we allow type constraints followed by `where` to be one of: +- Type conformance constraint, in the form of `T : IBase` +- Type equality constraint, in the form of `T == X` + +In both cases, the left hand side of a constraint can be a simple generic type parameter, or any types that are dependent on some +generic type parameter. For example, the following is allowed: +```csharp +interface IFoo { associatedtype A; } +struct S + where T : IFoo + where T.A == U +{} +``` + Detailed Explanation -------------------- @@ -241,37 +262,18 @@ Alternatives Considered There really aren't any compelling alternatives to `where` clauses among the languages that Slang takes design influence from. We could try to design something to solve the same problems from first principles, but the hypothetical benefits of doing so are unclear. -When it comes to the syntactic details, we could consider allowing for *multiple* `where` clauses (matching the C# syntax) as an alternative to the comma-separated list: +When it comes to the syntactic details, we could consider disallow type lists in the right hand side of a conformance constraint, and return allow multiple constraints to be separated with comma and sharing with one `where` keyword: struct MyStuff : Base, IFoo - where T : IFoo - where T : IBar + where T : IFoo, + T : IBar { ... } -This alternative form may result in more compact and tidy diffs when editing the constraints on declarations, at the cost of repeating the `where` keyword many times. +This alternative form may result in more compact code without needing duplicated `where` clause, but may be harder to achieve tidy diffs when editing the constraints on declarations. Future Directions ----------------- -### Allow more general types on the left-hand side of `:` - -There are many cases where it would be helpful to be able to introduce constraints on associated types. -As a contrived example: - - interface IPrintable { ... } - interface ISequence - { - associatedtype Element; - ... - } - extention T : IPrintable - where T : ISequence, - T.Element : IPrintable - { ... } - -In this example, an `extension` is used to declare that sequences are printable if their elements are printable. - - ### Allow more general types on the right-hand side of `:` Currently, the only constraints allowed using `:` have a concrete (non-`interface`) type on the left-hand side, and an `interface` (or conjunction of interfaces) on the right-hand side. @@ -284,30 +286,6 @@ In the context of `class`-based hierarchies, we can also consider having constra where T : Base { ... } -### Equality Constraints - -One future direction that we already intend to pursue is allowing exact equality constraints. -The primary use case envisioned for equality constraints is to express restrictions on associated types of type parameters. -As a contrived example: - - interface IProducer - { - associatedtype Element; - // ... - } - interface IConsumer - { - associatedtype Element; - // ... - } - void runPipeline(P producer, C consumer) - where P : IProducer, - C : IConsumer, - P.Element == C.Element - { ... } - -An equality constraint could either constrain an associated type to be equal to some concrete type, or to some other associated type. - ### Allow `where` clauses on non-generic declarations We could consider allowing `where` clauses to appear on any declaration nested under a generic, such that those declarations are only usable when certain additinal constraints are met. diff --git a/docs/proposals/002-api-headers.md b/docs/proposals/002-api-headers.md deleted file mode 100644 index 66b649228..000000000 --- a/docs/proposals/002-api-headers.md +++ /dev/null @@ -1,952 +0,0 @@ - -Revise Slang/GFX API Headers -============================ - -The public C/C++ API headers for Slang (and GFX) are in need of cleanup and refactoring for us to reach a "1.0" API. -This document attempts to document the guidelines that we will follow in such a refactor. - -Status ------- - -In discussion. - -Background ----------- - -The Slang API header (`slang.h`) has evolved over many years, going back as far as the "Spire" research project, which predates Slang (Spire is the reason for the `sp` prefix on functions in the C API). - -At some point, we made a conscious decision to move toward a COM-based API, both because it would simplify our story around binary compatibility and because it would allow us to provide more convenient API models in cases where subtyping/inheritance is fundamental to the domain. -Unfortunately, the net result has been that we have two different APIs cluttering up the same header file (the old C one, and the newer C++/COM one), and the one that is presented *first* is actually the one we would rather users didn't reach for. -The two APIs are sometimes out of sync, with one providing services the other doesn't. - -While the GFX project started later and was thus able to start using COM interfaces and a C++ API from the start, it still faces some challenges around API evolution and binary compatibility. -As support for GPU features (whether pre-existing or new) gets added, we find that the various interfaces want to grow and the various `Desc` structures want to add new fields. -Without care, neither of those is a binary-compatible change for user code. - -A concern across both Slang and GFX is that we have tended to design our APIs around the *most complicated* use cases we intend to support, at the expense of the *simplest* cases. -We know that we cannot remove support for difficult cases, but it would be good to support concise code for simple use cases, and to support a "progressive disclosure" style that allows users to gradually adopt more involved API constructs as they become necessary. - -Related Work ------------- - -There are obviously far too many C/C++ APIs and approachs to design for C/C++ APIs for us to review them all. -We will simply note a few key examples that can be relevant for comparison. - -The gold standard for C/C++ APIs is ultimately plain C. Plain C is easy for most systems programmers to understand and benefits from having a well-defined ABI on almost every interesting platform. FFI systems for other languages tend to work with plain C APIs. Clarity around ABI makes it easy to know what changes/additions to a plain C API will and will not break binary compatibility. The Cg compiler API and the Vulkan GPU API are good examples of C-based APIs in the same domains as Slang and GFX, respectively. These APIs reveal some of the challenges of using plain C for large and complicated APIs: - -* The lack of subtype polymorphism is a problem when a domain fundamentally has subtyping. The Cg reflection API uses a single `CGtype` type to represent all types, so that the an operation like `cgGetMatrixSize` is applicable to any type, not just matrix types. The API cannot guide a programmer toward correct usage, and must define what happens in all possible incorrect cases. - -* C has no built-in model for error signalling or handling. Error codes and out-of-band values (`NULL`, `-1`) are the norm, and there are a multitude of API-specific conventions for how they are applied. - -* C has no built-in model for memory and lifetime management. Most APIs either expose create/delete pairs or some kind of per-type reference-counting retain/release operations. In either case, the application developer is left to ensure that the operations are correctly invoked, often by writing their own C++ wrapper around the raw C API. - -Some developers opt for a "Modern C++" philosophy where the public API of a system makes direct use of standard C++ library types where possible. -For example, strings are passed as `std::string`s, cases that need polymoprhism expose `class` hierarchies, types that benefit from reference-counted lifetime management, explicitly uses `std::shared_ptr<...>`, and errors are signaled by `throw`ing exceptions. -The Falcor API ascribes to aspects of this approach. -The biggest challenges with Modern C++ APIs are: - -* Source compatibility can usually be achieved, but binary compatibility is hard to achieve even *within* a version of a system, must less across versions. The central problem is that C++ ABIs are often compiler-specific (rather than standard on a platform), and even for a single compiler like gcc or Visual Studio, the binary interface to the C++ standard library can and does break between versions. - -* While C++ exceptions are a built-in error-handling scheme, they are almost universally disliked among the kinds of developers who use APIs like Slang. Enabling exceptions in most compilers adds overhead, and actually using exceptions for their intended purpose (catching and handling errors) tends to be onerous. - -* Reference-counted lifetime management in Modern C++ relies on standard library types like `std::shared_ptr` - a type that is both inefficient and inconvenient. Most developers in our target demographic end up using "intrusive" reference counts (when they use reference-counting at all) because they are more efficient and convenient. - -COM is first and foremost an idiomatic way of using C++ to define APIs that are reasonably convenient while also dealing with the recurring problems of typical C and C++ approaches. -COM defines rules for error handling, memory management, and interface versioning that all compatible APIs can use. -While code using COM-based APIs is often verbose, it is largely consistent across all such APIs. - -A key place where COM does *not* provide a complete answer is around fine-grained "extensibility" of APIs, of the kind that commonly occurs with GPU APIs like OpenGL, D3D, and Vulkan. -Across such APIs, we see a wide variety of strategies to dealing with extensibility: - -* OpenGL uses an approach where objects are typically opaque but mutable, and a large number of fine-grained operations are used to massage an object into the correct state for use. Often the fine-grained state-setting operations are all able to use a single API entry point for key-value parameter setting (e.g., `glTexParameteri`), and a new feature can be exposed simply by defining constants for new keys and/or values. When new operations are required, they need to be queried using string-based lookup of functions. - -* D3D11 uses COM interfaces and "desc" structures (called "descriptors" at the time). For example, a mutable `D3D11_RASTERIZER_DESC` structure is filled in and used to create an *immutable* `ID3D11RasterizerState`. If extended features are required, a new interface like `ID3D11RasterizerState1` and/or a new descriptor type like `D3D11_RASTERIZER_DESC1` is defined. In all cases, the "desc" structure holds the union of all state that a given type supports. - -* Vulkan uses "desc" structures (usually called "info" or "create info" structures), which contain a baseline set of state/fields, along with a linked list of dynamically-typed/-tagged extension structures. New functionality that only requires changes to "desc" structures can be added by defining a new tag and extension structure. New operations are added in a manner similar to OpenGL. - -* D3D12 also uses COM interfaces and "desc" structures (although now officialy called "descriptions" to not overload the use of "descriptor" in descriptor tables), much like D3D11, and sometimes uses the same approach to extensibility (e.g., there are currently `ID3D12Device`, `ID3D12Device`, ... `ID2D12Device9`). In addition, D3D12 has also added two variations on Vulkan-like models for creating pipeline state (`ID3D12Device2::CreatePipelineState` and `ID3D12Device5::CreateStateObject`), using a notion of more fine-grained "subojects" that are dynamically-typed/-tagged and each have their own "desc". - -It is important to note that even with the nominal flexibility that COM provides around versioning, D3D12 has opted for a more fine-grained approach when dealing with something as complicated as GPU pipeline state. - -Proposed Approach ------------------ - -The long/short of the proposal is: - -* The primary API interface to Slang (and GFX) should be COM-based and use C++. Convenient C++ features like `enum class` are usable when they do not constrain binary compatibility. - -* Extensibility and versioning (where appropriate) should make use of "desc"-style tagged structures. Each of Slang and GFX will define its own `enum class` for the space of tags, rather than us trying to coordinate across the APIs. - -* We will focus on providing "shortcut" operations in the public API that allow developers to optimize for common cases and reduce the amount of boilerplate. - -* We will expose a C API that wraps the COM using `inline` functions. We will attempt to make the C API idiomatic when/as possible. - -* We can eventually/graduatelly provide a set of C++ wrapper/utility types that can further streamline the experience of using Slang, by hiding some of the details of COM reference counting, and "desc" structs". The utility code could also translate COM-style result codes into C++ exceptions, if we find that this is desired by some users. - -Detailed Explanation --------------------- - -At the end of this document there is a lengthy code block that sketches a possible outline for what the `slang.h` header *could* look like. - -Questions ---------- - -### Will we generate all or some of the API header? If so, what will be the "ground truth" verison? - -Note that Vulkan and SPIR-V benefit from having ground-truth computer-readable definitions, allowing both header files and tooling code to be generated. - -### Can we actually make a reasonably idiomatic C API that wraps a COM one, or should we admit defeat and have everything look like `slang__(...)`? - -Alternatives Considered ------------------------ - -We haven't seriously considered many alternatives in detail, other than the possibility of a plain C API (which we have tried, but not been able to make work). - -Appendix: A Header of Notes ---------------------------- - -The following code represents an sketch of a header that tries to match this proposal (and includes a lot of its own discussion/comentary). - -``` -/* Slang API Header (Proposed) - -This file is an attempt to sketch how the API headers for both -Slang and gfx could be organized in order to provide a nice -experience for developers who belong to different camps in terms -of what they want to see. - -Goals: - -* Support both C and C++ access to the API, with matching types, etc. - -* Able to use COM interfaces including use of inheritance/polymorphism - -* When compiling as C++, it should be possible to mix-and-match both C and C++ APIs - -Constraints: - - -Questions: - -* Do we actually need to restrict to block comments for pedantic compatibility - with old C versions? Are line comments close enough to universally-supported? - -*/ - -#ifdef __cplusplus - -/* Because of our goals above, we will actually end up with what amounts to -two copies of the C API, depend on whether or not we are compiling as C++. - -We start with the C++ COM-lite API, since that is the baseline. Note that -in this proposal, everything is being defined in the `slang` namespace, -rather than first declaring many things as C types and then mapping that -over to C++. -*/ - -namespace slang -{ - /* Note that in this proposal, everything is being declared in the `slang` - namespace first, rather than the old model of declaring various things - in C and then importing them into the namespace. - */ - -/* Basic Types */ - - typedef int32_t Int32; - /* typedefs for basic types, as needed ... */ - - -/* Enumerations and Constants */ - - /* Non-flag enumerations will use `enum class`. If we need to support clients - using older C++ versions/compilers, we can discuss macro-based ways to try - to work around this. - - Except in cases where there is an *extremely* compelling reason to do something - different, all enumerations use the `int` tag type that is the default for - `enum class`. - */ - - - /** Severity of a diagnostic generated by the compiler. - ... - */ - enum class Severity - { - Note, /**< An informative message. */ - /* ... */ - }; - - /* TODO: We need a clearly-defined policy for how to handle "flags" enumerations. - - My strong opinion is that we should generally avoid flags in public API just - because of the tendency to run out of bits sooner or later, but I also understand - the appeal... - */ - - struct SlangTargetFlag - { - enum : UInt32 - { - DumpIR = 1 << 0, - /* ... */ - }; - }; - typedef UInt32 SlangTargetFlags; - - /* I'm note sure about whether the `Result` type ought to be declared with `enum class`. - It would be nice to have the extra level of type safety, but it would also make our - `Result` incompatible with macros and template types that are intended to work with - `HRESULT`s. - */ - enum Result : Int32 - { - /* I *do* think we should go ahead and define all the cases of `Result`s - that we expect our API to traffic in right here in the `enum`, so that users - can easily inspect result codes in the debugger. - */ - - OK = 0, - /* ... */ - }; - -/* Forward Declarations */ - - /* Simple Types */ - struct UUID; - /* ... */ - - /* "Desc" Types */ - struct SessionDesc; - /* ... */ - - /* Interface Types */ - struct ISession; - - -/* Structure Types */ - - /* Theres's not much to say for the easy case... */ - - struct UUID - { - uint32_t data1; - uint16_t data2; - uint16_t data3; - uint8_t data4[8]; - }; - /* ... */ - - /* The more interesting bit is "descriptor" sorts of structures, which - we've done a lot of back-and-forth on how best to support. - - I'm going to write up something here while also acknowledging that picking - a good policy for how to handle this stuff is an orthogonal design choice. - */ - - enum class DescType : UInt32 - { - None, - SessionDesc, - /* ... */ - }; - enum class DescTag : UInt64; - - #define SLANG_MAKE_DESC_TAG(TYPE) DescTag(UInt64(slang::DescType::TYPE) << 32 | sizeof(slang::##TYPE)) - - struct SessionDesc - { - DescTag tag = SLANG_MAKE_DESC_TAG(SessionDesc); - - TargetDesc const* targets = nullptr; - Int targetCount = 0; - /* ... */ - - /* Note: There is some subtlety here if we use default member - initializers here, but also want to expose these types directly - via the C API (in cases where somebody is using the C API but - a C++ compiler). - - The tag approach here is intended to support something akin to - the Vulkan style, when using the C API: - - SlangSessionDesc sessionDesc = { SLANG_DESC_TAG_SESSION_DESC }; - ... - - That code will not compile under C++11, because of the default - members initializers in `slang::SessionDesc`, but it *will* compile - under C++14 and later. - - If we want to deal with C++11 compatiblity in that case, we can, but - it would slightly clutter up the way we declare these things. Realistically, - we'd just split the two types: - - struct _SessionDesc { ... data but no initialization ... }; - struct SessionDec : _SessionDesc { ... put a default constructor here ... }; - - I'm not a fan of that option if we can avoid it. - */ - }; - - /* ... */ - - /* *After* all the "desc" types are defined, we can actually define the enum - for their tags (just to make life easier for users looking at things in their - debugger. - */ - enum class DescTag : UInt64 - { - None = 0, - SessionDesc = SLANG_MAKE_DESC_TAG(SessionDesc), - /* ... */ - }; - - /* Versioning: If we are in a situation where we'd like to change a type that - has already been tagged, we should first consider just creating an additional - "extension" desc type, to be used together with the original. By adding - suitable convenience APIs, we can make this easy to work with. - - If we really do decide that we want a new version of a specific desc, we should - start by doing the thing D3D does, and make a new numbered type: - - struct SessionDesc { ... the original ... }; - struct SessionDesc1 { ... the new one ... }; - - When possible, the new type should use matching field names/types and ordering. - Even if we are just adding fields, we should not try using inheritance (just - because the C++ spec doesn't guarantee enough about how inheritance is implemented). - - The new structure types should get its own desc type/tag, distinct from the original. - - If we decide that we want clients to compile against the latest version of these - types by default, we can shuffle around the names, but we need to be careful to - *also* shuffle the `DescType` cases (so that the binary values stay the same): - - struct SessionDesc0 { ... the original ... }; - struct SessionDesc1 { ... the new one ... }; - typedef SessionDesc SesssionDesc1; - - At the point where we introduce a second version, it is probably the right time - to enable developers to lock in to any version they choose. In the code above - the user can always just use `SessionDesc0` or `SessionDesc1` explicitly, or they - can just stick with `SessionDesc` in the case where they always want the latest - at the point they compile. - - (If we wanted to get really "future-proof" we'd define every struct with the `0` - prefix right out of the gate, and always have the `typedef` in place. I'm not conviced - that would ever pay off.) - - I expect most of this to be a non-issue if we are zealous about using fine-grained - rather than catch-all descriptors at this level of the API. - - (There's more I could talk about here, but this isn't supposed to be the topic at hand) - */ - - -/* Interfaces */ - - /* There's an open question of how to name the `IUnknown` equivalent - once things are all namespaced. We could use `slang::IUnknown`, but I fear - that could lead to complications or confusion for codebases that also use - MS-provided COM-ish APIs. - */ - - struct ISession : public ISlangUnknown - { - SLANG_COM_INTERFACE(...); - - /* In order to maximize our ability to evolve the API while maintaining - binary compatibility, I'm going to recommend the somewhat bold step - of defaulting to making interface entry points that are "implementation - details" rather than intended for direct use in most cases. - */ - - virtual SLANG_NO_THROW Result SLANG_MCALL _createCompileRequest( - void const* const* descs, - Int descCount, - UUID const& uuid, - void** outObject) = 0; - - /* Instead, most users will direclty call the operations only through - wrappers that provide conveniently type-safe behavior: - */ - inline Result createCompileRequest( - CompileRequestDesc const& desc, - ICompileRequest** outCompileRequest) - { - return _createCompileRequest( - &desc, 1, SLANG_UUID_OF(ICompileRequest), - (void**)outCompileRequest); - } - - /* An important property of this design is that we can easily define - convenience overloads that take direct parameters for common cases: - */ - inline Result createCompileRequestForPath( - const char* path, - ICompileRequest** outCompileRequest) - { - ...; - } - - /* Versioning: Note that we can easily define convenience overloads - for multiple versions of descriptor types (`CompileRequest` and - `CompileRequest`), or for *combinations* of descriptor types: - */ - inline Result createCompileRequest( - CompileRequestDesc const& requestDesc, - ExtraFeatureDesc const*& extraFeatureDesc, - ICompileRequest** outCompileRequest) - { - void* descs[] = { &requestDec, &extraFeatureDesc }; - return _createCompileRequest - descs, 2, SLANG_UUID_OF(ICompileRequest), - (void**)outCompileRequest); - } - - /* As a final detail, we should consider whether to support overloads - that work with either our `slang::ComPtr` *or* an application's own - smart pointer type. - - The user could override the smart pointer type by defining macros. - The defaults would be: - - #define SLANG_SMART_PTR(TYPE) slang::ComPtr - #define SLANG_SMART_PTR_WRITE_REF(PTR) ((PTR).writeRef()) - */ -#ifndef SLANG_DISABLE_SMART_POINTER_OVERLOADS - inline Result createCompileRequest( - CompileRequestDesc const& desc, - SLANG_SMART_PTR(ICompileRequest)* outCompileRequest) - { - return _createCompileRequest( - &desc, 1, SLANG_UUID_OF(ICompileRequest), - SLANG_SMART_PTR_WRITE_REF(*outCompileRequest)); - ) -#endif - - /* If we ever have cases where we want to support utility/wrapper operations - of higher complexity than what we feel comfortbale making `inline` (that is, - stuff that might be best off in a `slang-utils` static library) we could conceivably - handle those via judicious use of `extern`: - */ - inline Result createCompileRequestFromJSON( - char const* jsonBlob, //< a serialized form of the compilation state - ICompileRequest** outCompileRequest, - { - extern Result slang_ISession_createCompileRequestFromJSON( - char const*, ICompileRequest**); - return slang_ISession_createCompileRequestFromJSON(jsonBlob, outCompileRequest); - } - /* I doubt we'd ever really need that kind of approach, and would always decide - that functionality either belongs in core Slang (perhaps as a new derived interface) - or can go as global (non-member) functions in a utility library. - */ - - /* ... */ - } - - /* Note: I'm assuming here that we continue our implicit contract in terms - of versioning of COM interfaces: - - * Every interface is thought of as having a contract about who can *provide* - and who can *consume* it. For many (like `ISession`) only the Slang implementation - is supposed to provide it and only users consume it. Some callback interfaces - go the other way, and a few (like `slang::IBlob`) need to go both directions. - - * For interfaces that Slang provides and the user consumes, we can append new - `virtual` methods onto the end. This realistically needs a check somewhere, such - that we fail creation of the `IGlobalSession` if the user compiled against a - header that exposes the new method but is linking a DLL that doesn't. This is - what the `SLANG_API_VERSION` in the original header is supposed to be for: we - should increment it when we expand the API contract, and the global-session - creation should fail if the application is asking for too new of a version. - - * For interfaces that Slang consumes, we cannot realistically add/remove anything. - Theoretically, we could delete some of the `virtual` methods if we no longer - expect to call them, but that could still break client code that uses `override` - on their definitions. - - * We need to try very hard not to change the interface types of parameters to - non-wrapper COM methods, even if the result should be binary-compatible. There - are cases where it might be reasonable and "type-safe," but each and every one - probably needs clear auditing. - - Ideally the rules we use for Slang-provided interfaces can help us avoid the - proliferation of `IThing`, `IThing2`, `IThing3`, etc. We need to be careful about - that in the long run, though, because we may find that it causes problems in the wild - if software needs to interact with Slang in a context where the developer cannot - control the version of the Slang DLL they will be using at runtime. - - In theory, we could solve that problem by letting a user pass `SLANG_API_VERSION` - *in* to the header via a `#define`, and have us skip over any declarations introduced - after the given version. - */ -} - -/* Back outside the namespace (but still in the case where we know C++ is -supported), we can define the C-compatible API by using the C++ API -as its underlying representation. -*/ - -extern "C" -{ - -/* Basic Types */ - - /* The C-API types can just be `typedef`s of the C++ ones */ - - typedef slang::Int32 SlangInt32; - /* ... */ - -/* Enumerations and Constants */ - - /* For the case where we *know* a C++ compiler is being used, we - can actually use the C++ `enum class` declarations to provide - the enumerated types of the C API. - */ - typedef slang::Severity SlangSeverity; - - /* We can use macros to define the C-API enum cases, while still - preserving the type safety of the `enum class` approach. - - Note: We could also use `static const`s here, but that seems like - overkill. - */ - #define SLANG_SEVERITY_NONE (slang::Severity::None) - /* ... */ - -/* Structure Types */ - - /* For the case of providing the C API for a C++ compiler, we can - directly use the C++ structure types in all cases. */ - - typedef slang::UUID SlangUUID; - typedef slang::SessionDesc SlangSessionDesc; - /* ... */ - -/* Interfaces */ - - /* Because we are compiling as C++, defining the types for the interfaces - is easy, and we can easily pass objects between modules/files that are - using the two version of the API without any casting: - */ - typedef slang::ISession SlangSession; - /* ... */ - - /* In order to have a plain-C API, the user of course needs a way to - dispatch into those interfaces. - - Note: There is a big question here of whether the API header should be - trying to define the C API functions `inline` here or not. - - The argument for using `inline` is that it doesn't add any additional - requirements for somebody using the C API from within C/C++, compared to - the C++ API. - - The argument *against* is that for things like binding to other languages, - the user would probably prefer that these operations have linkage. - - Realistically, the right thing is for the header to include both declarations - *and* definitions, but to allow the application to conditionalize the inclusion - of the definitions *and* enable/disable the use of `inline` for declarations/definitions. - A user could use that control to compile their own linkable stub with C-compatible - functions. - */ - - /* We need to provide the fully-general version of the function, for clients - that might need it, but we probably don't want that to be the first one users - reach for. - */ - inline SlangResult _SlangSession_createCompileRequest( - SlangSession* session, - void const* const* descs, - SlangInt descCount, - SlangUUID const* uuid, - void** outObject) - { - return session->_createCompileRequest(descs, descCount, uuid, outObject); - } - - /* The catch here is that the C++ API used overloading as a way - to provide convenient wrappers around the fully-general core operations, - and also to provide versioning support. - - We could define the same set of overloads here, with the same names, for - use by clients who don't actually care about C compatiblity but just like - a C-style API. That is probably worth doing. - - Otherwise, we realistically need to start defining some de-facto naming - scheme and/or versioning for stuff in the C API. At least one wrapper - should be "blessed" as the default one. - */ - inline SlangResult SlangSession_createCompileRequest( - SlangCompileRequestDesc const* desc, - SlangCompileRequest** outCompileRequest) - { - return session->_createCompileRequest(*desc, outCompileRequest); - } - - /* Note that we need/want to provide wrappers for *all* the operations - on each interface, even the ones they inherit. E.g.:*/ - inline uint32_t SlangSession_addRef( - SlangSession* session) - { ... } - /* The reason for this is so that a pure-C user doesn't *have* to rely on - implicit conversion of these types to their bases (which in this path is - made possible via C++ features, but wouldn't be available in a true pure-C - world). - */ - - /* If/when we start to deal with versioning of either the "desc" type or - the interface involved in such an operation, we will need to do the numeric-suffix - thing or similar stuff to distinguish the old and new functions. - - We can probably do some work to always make the latest version (or at least - the one we want users to be using) have the short/clean name. Binary compatibility - shouldn't actually break so long as the signature of the new function can technically - handle calls of the old form (since the COM-level bottleneck function won't care about - the static types of descs - just their tags). - */ - - /* Finally, the C API level is where we should define the core factory entry - point for creating and initializing the Slang global session (just like - in the current header). Here we jsut generalize it for creaitng "any" global - object, based on a UUID and a bunch of descs. - */ - SLANG_API SlangResult slang_createObject( - void const* const* descs, - Int descCount, - UUID const* uuid, - void** outObject); - - /* The actual global session creation is then a wrapper like everything else. - */ - inline SlangResult SlangGlobalSession_create( - SlangGlobalSessionDesc const* desc, - SlangGlobalSession** outGlobalSession) - { - return slang_createObject( - &dec, 1, SLANG_UUID_OF(slang::IGlobalSession), (void**)outGlobalSession); - } -} - -#else - -/* All of the above declarations (even the C-level ones) only work if we are -compiling as C++. Thus we need a distinct strategy to define everything in the -case where we are compiling as pure C. - -The basic strategy isn't that hard: we just do things the raw C way. -There will be a lot of repetition involved, but this proposal assumes we are -generating as much of the API as possible anyway. -*/ - -/* Basic Types */ - - /* We just define the basic types direclty, without the indirection - through the declarations in the `slang::` namespace. - */ - - typedef int32_t SlangInt32; - /* ... */ - -/* Enumerations and Constants */ - - /* Every enum in this case is a `typedef` plus an actual `enum`: - */ - - typedef int SlangSeverity; - enum - { - SLANG_SEVERITY_NONE = 0, - /* ... */ - }; - - /* ... */ - -/* Structure Types */ - - /* The simple case stays simple, just with the gross bit of - duplicating a *lot* of what we already had in the C++ API. - - (There's a big design question here of whether we can/should try - to remove as much duplication as possible in order to reduce - boilerplate, even if it comes at the cost of clarity because of - heavier reliance on macros, etc.) - */ - - struct SlangUUID - { - uint32_t data1; - uint16_t data2; - uint16_t data3; - uint8_t data4[8]; - }; - /* ... */ - - /* The desc-related stuff is really just a translation of the - same basic ideas to plain C: */ - - typedef SlangUInt32 SlangDescType; - enum - { - SLANG_DESC_TYPE_NONE = 0, - SLANG_DESC_TYPE_SessionDesc, - /* ... */ - }; - typedf SlangUInt64 SlangDescTag; - - #define SLANG_MAKE_DESC_TAG(TYPE) SlangDescTag(UInt64(SLANG_DESC_TYPE_##TYPE) << 32 | sizeof(Slang##TYPE)) - - struct SlangSessionDesc - { - SlangDescTag tag; - - SlangTargetDesc const* targets; - SlangInt targetCount; - /* ... */ - }; - /* ... */ - - #define SLANG_DESC_TAG_SESSION_DESC SLANG_MAKE_DESC_TAG(SessionDesc) - /* ... */ - -/* Forward Declarations */ - - typedef struct SlangSession SlangSession; - /* ... */ - -/* Interfaces */ - - /* There's already a lot known about how to define COM interfaces for - consumption from C, so this is actually mostly straightforward. - - Note: these definitions would *only* be needed in the case where we - are compiling the actual implementations of the C API functions. It - is possible that we can/should just not bother with these, under - the assumption that anybody who wants a true pure-C API probably wants - a linkable "stub" library anyway, in which case we can provide that - library ourselves, and compile it as C++. - */ - - /* TODO: The big thing I'm skipping here is setup for the UUIDs. - I think we can provide C-compatible macros for those pretty easily, - but exactly what that should look like is maybe more complicated. */ - - struct SlangSession - { - /* The long/short is that we define a pointer field to a struct - of function pointers, which matches the expected C++ virtual - function table layout. - */ - - struct - { - /* Note: methods from all base interfaces need to go here... */ - - SLANG_NO_THROW SlangResult SLANG_MCALL (*_createCompileRequest)( - SlangSession* session, - void const* const* descs, - SlangInt descCount, - SlangUUID const* uuid, - void** outObject); - - /* ... */ - - } * vtbl; - }; - /* ... */ - - /* With the core type declarations out of the way, the actual functions - that forward to it are easy enough: - */ - inline SlangResult _SlangSession_createCompileRequest( - SlangSession* session, - void const* const* descs, - SlangInt descCount, - SlangUUID const* uuid, - void** outObject) - { - /* The only interesting complications here are the `->vtbl` - and the need to pass `session` explicitly. We could probably - macro away the difference if we don't want to have distinct - C-API-compiled-via-C++ and C-API-compiled-via-C cases. - */ - return session->vtbl->_createCompileRequest( - session, descs, descCount, uuid, outObject); - } - - /* The declarations of the global session creation stuff are almost - identical, so there's no real need to dpulicate it here. - */ - - /* For the true pure-C users, we probably want to provide convenience - functions and/or macros to enable the casts that should be statically - possible.*/ - inline SlangUnknown* SlangSession_asUnknown(SlangSession* session) - { - return (SlangUnknown*) session; - } - /* ... */ - - -/* - -Okay, so that's the basic idea of the proposal for how to expose our API(s). - -I realize this didn't get into the actual details of type hierarchies or what -the actual "desc" types need to be for Slang and gfx. The focus here was much -more on the syntactic side of things, in terms of how we can define our API -so that both C and C++ are usable and can be freely intermixed within a codebase. - -*/ - -/* There's probably an entire additional document that could be written about -utility/wrapper stuff to make the interfaces nicer for C++ users. Some examples -follow: - -We could consider having a hierarchy of wrapper smart-pointer types that codify the -reference-counting policies without the user having to really think about `ComPtr` stuff: - - struct Unknown - { - public: - // typical stuff... - - - protected: - slang::IUnknown* _ptr = nullptr; - } - - struct Session : Unknown - { - public: - ISession* get() const { return (ISession*)_ptr; } - operator ISession*() const { return get(); } - - Result createCompileRequest( - CompileRequestDesc const& desc, - CompileRequest* outCompileRequest) - { ... } - } - -Another thing to consider is whether any of our COM-ish wrappers should allow for -use of exceptions instead of `Result`s: - - struct ISession : ... - { - ... - -#if SLANG_ENABLE_SMART_PTR - ... - - #if SLANG_ENABLE_EXCEPTIONS - SLANG_SMART_PTR(ICompileRequest) createCompileRequest( - CompileRequestDesc const& desc) - { - SLANG_SMART_PTR(ICompileReqest) compileRequest; - SLANG_THROW_IF_FAIL(_createCompileRequest( - &desc, 1, SLANG_UUID_OF(IComileRequest), comileRequest.writeRef())); - return compileRequest; - } - - ... - #endif -#endif - } - -Both for the sake of C API and especialy for gfx (both C and C++), we should consider -defining some coarse-grained aggregate desc types as utilities: - - struct SimpleRasterizationPipelineStateDesc - { - // sub-descs for all the relevant pieces: - // - PipelineProgramDec program; - DepthStencilDesc depthStencil; - MultisampleDesc multisample; - PrimitiveTopologyDesc primitiveTopology; - NVAPIDesc nvapi; - // ... - - // "fluent"-style setters for all the relevant pieces: - - SimpleRasterizationPipelineStateDesc& setEnableDepthTest(bool value) - { - markDepthStencilDescUsed(); - depthStencil.enableDepthTest = value; - return *this; - } - - // ... - - // This is also the logical granularity to provide things like - // List members for attachments, etc. rather than just pointer-and-count: - - private: List colorAttachments; - public: AttachmentDesc& addColorAttachement(); - - // There should also be convenience constructors common cases - // (especially relevant for things like textures). - - // In the simplest implementation strategy, we keep a bitmask for which - // of the sub-descs have actually beem used (either requested by the user, - // or set to non-default values): - // - enum class SubDesc { Program, DepthStencil, ... Count }; - uint32_t usedSubDescs = 0; - void markSubDescUsed(SubDesc d) - { - uint32_t bit = 1 << int(d); - if(usedSubDesc & bit) return; - - usedSubDescs |= bit; - updatePointers(); - } - - // We then maintain a compacted array of all the sub-descriptors needed - // to form the combined state for passing along to the lower-level API. - // - void* subDescs[int(SubDesc::Count)]; - int subDescCount = 0; - - void updatePointers() - { - subDescCount = 0; - if(usedSubDescs & (1 << int(Program))) - { - subDescs[subDescCount++] = &program; - } - /// ... - } - }; - -While the implementation of this monolithic desc types would not necessarily be pretty, -it would enable users who want the benefits of the "one big struct" appraoch to get -what they seem to want. - -The next step down this road is to take these aggregate desc types and turn them into -actual API objects for the purposes of the C API, so that users can more conveniently -create stuff: - - GFXRasterizationPipelineStateBuilder* GFXDevice_beginCreateRasterizationPipelineState( - GFXDevice* device); - - void GFXRasterizationPipelineStateBuilder_setEnableDepthTest( - GFXRasterizationPipelineStateBuilder* builder, - bool enable); - - // Note: frees the given `builder`, so user doesn't have to do it manually - GFXPipelineState* GFXRasterizationPipelineStateBuilder_create( - GFXRasterizationPipelineStateBuilder* builder); - -Obviously the function names are very verbose there, but they could probably be cleaned -up a lot if we want to go down this route. Certainly, if we decide that C API users are -not going to be inclined to use a lot of fine-grained descs, this starts to seem like -an increasingly attractive way to go. -*/ - -#endif -``` \ No newline at end of file diff --git a/docs/proposals/003-error-handling.md b/docs/proposals/003-error-handling.md deleted file mode 100644 index 28652824f..000000000 --- a/docs/proposals/003-error-handling.md +++ /dev/null @@ -1,296 +0,0 @@ -Error Handling -============== - -Slang should support a modern system for functions to signal, propagate, and handle errors. - -Status ------- - -In discussion. - -Background ----------- - -Errors happen. It is impossible for any programming language to statically rule out the possibility of unexpected situations arising at runtime. -There are a wide variety of strategies used in programming, both provided by languages and enforced by idiom in codebases. - -Not all errors are alike, in that some are more expected and reasonable to handle than others. -Most errors can fit into a few broad categories like: - -* Unrecoverable or nearly unrecoverable failures like resource exhaustion (out of memory), or an OS-level signal to terminate the process. - -* Incorrct usage of an API in ways that violate invariants. For example, passing a negative value to a function that says it only accepts positive values. - -* Out-of-range or otherwise invalid data coming from program users. For example, a console program asks the user to type a number, but the user enters some string that does not parse as a number. - -* Failure of operation that will usually succeed, but for which exceptional circumstances can lead to failures. For example, when reading from an open file we typically expect success, but failure is possible for many reasons outside of a programmer's control (like network disruption when accessing a remote file). A robust program often wants to recover from such failures, but often the policy for how recovery should occur is at a higher level than the code that first detects the error. - -These different categories often benefit from different strategies: - -* Typically there is neither a reason nor a desire to do anything about nearly-unrecoverable errors; the program has well and truly crashed. - -* When programmers violate the invariants of an API, they typically want to know about it as early as possible (during development) so they can fix their code. Breaking into the debugger is often the best answer, and in many cases trying to propagate or recover from such failures would be wasted effort. - -* When an operation could fail due to mal-formed data coming from a user, programmers typically want to be forced to handle the failure case at the point where the error may arise. In languages that have an `Optional` or `Maybe` type, it is often easiest to return that. - -* Unpredictable, exceptional, and recoverable errors are among the hardest to deal with, and often benefit from direct language support. - -The Slang language currently doesn't have direct support for *any* form of error handling, but this document focuses on errors in the last of the categories above. - -Related Work ------------- - -In the absence of language support, developers typically signal and propagate errors using *error codes*. The COM `HRESULT` type is a notable example of a well-defined system for using error codes in C/C++ and other languages. -Error codes have the benefit of being easy to implement, and relatively light-weight. -The main drawback of error codes is that developers often forget to check and/or propagate them, and when they do remember to do so it adds a lot of boilerplate. -Additonally, reserving the return value of every function for returning an error code makes code more complex because the *actual* return value must be passed via a function parameter. - -C++ uses *exceptions* for errors in various categories, including unpredictable but recoverable failures. -Propagation of errors up the call stack is entirely automatic, with unwinding of call frames and destruction of their local state occuring as part of the search for a handler. -Neither functions that may throw nor call sites to such functions are syntactically marked. -Exceptions in C++ have often been implemented in ways that add overhead and require complicated support in platform ABIs and intermediate languages to support. - -Java uses exceptions with similar rules to C++, but adds a restriction that functions must be marked with the types of exceptions they may throw or propagate, except for those that inherit from `RuntimeException`, which are intended to represent some of the other categories of error in our taxonomy (like simple invariant violations and nearly unrecoverable errors). -The need to mark every function that might fail (or propagate failure) was seen by most developers at the time as unreasonably onerous. -Developers often smuggled other kinds of exceptions out through `RuntimeException`s, to get them through API layers that were not designed to support exceptions. - -Both Rust and Swift try to strike a balance between error codes and languages with exceptions. -At a high level, each takes an approach where the generated code is comparable to an error-code-based solution (so that no special ABI or IL support is needed), but direct syntactic support makes propagating and/or handling errors more convenient. - -In Rust, a function that returns `std::Result` either returns successfully with a value of type `SomeType`, or fails with an error of type `SomeError`. -The `Result` type is itself just a Rust `enum`, so that results can be handled by pattern-matching with `match`, `if let`, etc. -Direct syntactic support is added so that in the body of a `Result`-returning function, a postfix `?` operator can be applied to an expression of type `Result` to implicitly propagate `E` on any failure, and return the `X` value otherwise. -Some higher-order functions can Just Work with `Result`-returning functions, if their signatures are compatible, but many operations like `map()`, `fold()`, etc. need distinct overloads that support `Result`s. -Functions that return `X` and those that return `Result` are not directly convertible. - -Swift provides more syntactic support for errors than Rust, although the underlying mechanism is similar. -A Swift function may have `throws` added between the parameter list and return type to indicate that a function may yield an error. -All errors in Swift must implement the `Error` protocol, and all functions that can `throw` may produce any `Error` (although there are proposals to extend Swift with "typed `throws`"). -Any call site to a `throws` function must have a prefix `try` (e.g., `try f(a, b)`), which works simiarly to Rust's `?`; any error produced by the called function is propagated, and the ordinary result is returned. -Swift provides an explicit `do { ... } catch ...` construct that allows handlers to be established. -It also provides for conversion between exceptions and an explicit `Result` type, akin to Rust's. -Higher-order functions may be declared as `rethrows` to indicate that whether or not they throw depends on whether or not any of their function-type parameters is actually a `throws` function at a call site. -Any non-`throws` function/closure may be implicitly converted to the equivalent `throws` signature, so that non-throwing functions are subtypes of the throwing ones. - - -The model used in Swift is compatible with the more general notion of *effects* in type theory. -A simple model of function types like `D -> R` can be extended to support zero or more effects `E0`, `E1`, etc. that live "on the arrow": `D -{E0, E1}-> R`. -Purely functional languages like Haskell sometimes use monads as a way to represent effects: a function `D -> IO R` is effectively a function from `D` to `R` with the addition effect that it may perform IO. -Making effects more explicit allows a type system to reason about sub-typing in the presence of effects (a function type without effect `E` is a subtype of a function with that effect), and to express code that is generic over effects. - -Proposed Approach ------------------ - -We propose a modest starting point for error handling in Slang that can be extended over time. -The model borrows heavily from Swift, but also focuses on strongly-typed errors. - -The standard library will provide a built-in interface for errors, initially empty: - -``` -interface IError {} -``` - -User code can define their own types (`struct` or `enum`) that conform to `IError`: - -``` -enum MyError : IError -{ - BadHandle, - TimedOut, - // ... -} -``` - -User-defined functions (in both traditional and "modern" syntax) will support a `throws ...` clause to specify the type of errors that the function may produce: - -``` -float f(int x) throws MyError { ... } - -func g(x: int) throws -> float MyError { ... } -``` - -Call sites to a `throws` function must wrap any potentially-throwing expression with a `try`: - -``` -float g(int y) throws MyError -{ - return 1.0f + try f(y-1); -} -``` - -Code can explicitly raise an error using a `throw` expression: - -``` -throw MyError.TimedOut; -``` - -We will allow `catch` clauses to come at the end of any `{}`-enclosed scope, where they will apply to any errors produced by `throw` or `try` expressions in that scope. - -``` -{ - ... - try f(...); - ... - - catch( e: MyError ) { ... } -} -``` - -We will also want to add `defer` statements, as they are defined in Go, Rust, Swift, etc. -The statements under a `defer` will always be run when exiting a scope, even if exiting as part of error propagation. - -Detailed Explanation --------------------- - -Consider a function that uses most of the facilities we have defined: - -``` -float example(int x) throws MyError -{ - if(someCondition) - { - throw MyError.TimedOut; - } - ... - defer { someCleanup(); } - ... - { - let y : int = 1 + try g(...); - - catch(e : MyError) - { ... } - } - ... - return someValue; -} -``` - -We will show how a function in this form can be transformed via incremental steps into something that can be understood and compiled without specific support for errors. - -### Change Signature - -First, we transform the signature of the function so that it returns something akin to an `Optional` and returns its result via an `out` parameter, and modify any `return` points to write the `out` parameter and return `null` (the not-present case of `Optional`): - -``` -MyError example_modified(int x, out float _result) -{ - ... - - _result = someValue; - return null; -} -``` - -### Desugar `try` Expressions - -Next we can convert any `try` expressions into a more explicit form, to match the transformation of signature. A statement like this: - -``` -let y : int = 1 + try g(...); -``` - -transforms into something like: - -``` -var _tmp : int; -let _err : Optional = g_modified(..., out _tmp); -if( _err != null ) -{ - throw _err.wrappedValue; -} -let y : int = 1 + _tmp; -``` - -### Desugar `throw` Expressions - -For every `throw` site in a function body, there will either be no in-scope `catch` clause that matches the type thrown, or there will be eactly one most-deeply-nested `catch` that statically matches. -Front-end semantic checking should be able to associate each `throw` with the appropriate `catch` if any. - -For `throw` sites with no matching `catch`, the operation simply translates to a `return` of the thrown error (because of the way we transformed the function signature). - -For `throw` sites with a matching `catch`, we treat the operation a a "`goto` with argument" that jumps to the `catch` clause and passes it the error. -Note that our IR structure already has a concept of "`goto` with arguments". - -### Desugar `defer` Statements - -Handling of `defer` statements is actually the hardest part of this proposal, and as such we should probably handle `defer` as a distinct feature that just happens to overlap with what is being proposed here. - -### Subtyping: Front-End - -We should (at some point) add a `Never` type to the Slang type system, which would be an uninhabitable type suitable for use as the return type of functions that never return: - -``` -func exit(code: int) -> Never; // C `exit()` never returns -``` - -`Never` is effectively a subtype of *every* type and, as such, an expression of type `Never` can be implicitly converted to any type. - -A `throw` expression has the type `Never`, allowing a user to write code like: - -``` -// Because `Never` can convert to `int`, this is valid: -int x = value > 0 ? value : throw MyError.OutOfBounds; -``` - -A function without a `throws` clause is semantically equivalent to a function with `throws Never`. -If we make that equivalence concrete at the type-system level, then a higher-order function can be generic over both throwing and non-throwing functions: - -``` -func map( - f: (D) throws E -> R, - l: List) - throws E -> List; -``` - -A function type with `throws X` is a subtype of a function with `throws Y` if `X` is a subtype of `Y`. -That includes the case where `X` is `Never`, so that a non-`throws` function type is a subtype of any `throws` function type with the same parameter/result signature. - -### Subtyping: Low-Level - -The subtyping relationship for `Never` *values* is irrelevant to codegen. Any place in the IR that has a `Never` value available to it represents unreachable code. - -The subtyping relationship for `Never` in function types is more challenging, both for result types and error types. At the most basic, we can inject trampoline/thunk functions at any points where we have a `Never`-yielding function and need a function that returns `X` to pass along. - -If we were doing low-level code generation for a platform where we can define our ABI, it would be possible to have `throws` and non-`throws` functions use distinct calling conventions, such that: - -* The orinary parameters and reuslts are passed in the same registers/locations in both conventions. - -* The error value (if any) in the `throws` convention is passed via registers/locations that are callee-save in the non-`throws` convention. - -Under that model, a call site to a potentially-`throws` function can initialize the registers/locations for the error result to `null`/zero before dispatching to the callee. -If the callee is actually a non-`throws` function it would not touch those registers, and no error would be detected. -In that case, a non-`throws` function/closure could be used directly as a `throws` one with no conversion. -Such calling-convention trickery isn't really possible to implement when emitting code in a high-level language like HLSL/GLSL or C/C++. - -Questions --------------- - -### Should we support the superficially simpler case of "untyped" `throws`? - -Having an `IError` interface allows us to eventually decide that `throws` without an explicit type is equivalent to `throw IError`. -It doesn't seem necessary to implement that convenience for a first pass, especially when there are use cases for `throws` that might not want to get into the mess of existential types. - -### Should the transformations described here be implemented during AST->IR lowering, or at the IR level? - -That's a great question! My guess is that some desugaring will happen during lowering, but we will probably want to keep `throws` functions more explicitly represented in the IR until fairly late, so that we can desugar them differently for different targets (if desired). - -### Do we need `Optional` to be supported to make this work? - -It is unlikely that we'd need it to be a user-visible feature in a first pass, but we might want it at the IR level. -For this feature to work, we really need `sizeof(Optional)` to be the same as `sizeof(X)` for simple cases where `X` is an `enum` or (for suitable targets) a type that is pointer-based. - -A first pass at the feature might only support cases where error types are `enum`s and where the zero value is the "no error" case. - -### Should we have a `Result` type akin to what Rust/Swift have? Should a `throws E` function be equivalent to one that returns `Result`? - -That all sounds nice, but for now it seems like overkill. -Slang doesn't really have any facilities for programming with higher-order functions, pattern matching, etc. so adding types that mostly shine in those cases seems like a stretch. - -Alternatives Considered ------------------------ - -We could decide that Slang shouldn't be in the business of providing error-handling sugar *at all* and make this a problem for users. -That isn't really a reasonable plan for any modern language, but it is the status quo and null hypothesis if we don't start in on a better plan. - -We could try to focus on C++ interop/compatibility and decide that errors in Slang should use exceptions, and only make "proper" language-supported error handling available to platforms that support exceptions at a suitably low level. -Doing so would give us all the disadvantages of C++ exceptions, and also mean that most of our users wouldn't end up using our error-handling tools, because doing so would render code non-portable. diff --git a/docs/proposals/004-com-support.md b/docs/proposals/004-com-support.md deleted file mode 100644 index dc0bd52d3..000000000 --- a/docs/proposals/004-com-support.md +++ /dev/null @@ -1,240 +0,0 @@ -COM Support -=========== - -When Slang is used as a host/CPU programming language, it is likely that users will want to use interact with COM interfaces, either by consuming them or implementing them. -The Slang language and compiler should provide some first-class features to make working with COM interfaces feel lightweight and natural. - -Status ------- - -Implemented. - -Background ----------- - -COM is not perfect, but it is one of the only real solutions for cross-platform portable C++ APIs that care about binary compatibility and versioning. -Developers who use Slang are likely to write code that uses COM, whether to interact with Slang itself (and/or GFX), or with platform APIs like D3D. - -While COM provides idioms for addressing many practical challenges, it is also inconvenient in that it introduces a lot of *boilerplate*: - -* COM types all need to implement the core `IUnknown` operations for casting/querying and reference counting. - -* Code using COM interfaces needs to perform `AddRef` and `Release` operations manually, or use smart pointer types to automate lifetime management. - -* Code that calls into COM interfaces typically needs to use boilerplate code and/or macros to deal with `HRESULT` error codes, handling or propagating them as needed. - -Our in-progress work on supporting CPU programming in Slang emphasizes supporting idiomatic code without a lot of boilerplate. -Our intended path includes things that are compatible with the COM philosophy, like reference-counted lifetime management and idiomatic use of result/error codes, but those features don't currently align with the more explicit style used by COM in C/C++. - -Related Work ------------- - -The .NET platform includes some support for allowing .NET `interface`s and COM interfaces to interoperate. -TODO: Need to study this and learn how it works. - -Proposed Approach ------------------ - -We propose to allow COM interfaces to be declared using the Slang `interface` construct, with an appropriate attribute or modifier: - -``` -[COM] interface IDevice -{ - ITexture createTexture(__read TextureDesc desc) throws HRESULT; - - void setTexture(int index, ITexture texture); -} -``` - -A declaration like the above will translate into output C++ along the lines of: - -``` -struct IDevice : public IUnknown -{ - virtual HRESULT SLANG_MCALL createTexture(TextureDesc const& desc, ITexture** _result) = 0; - - virtual void SLANG_MCALL setTexture(int index, ITexture* texture) = 0; -}; -``` - -Key things to note: - -* The `[COM] interface` becomes a C++ `struct` that inherits from `IUnknown` -* Methods defined in the `[COM] interface` become pure-virtual `SLANG_MCALL` methods in C++ -* Parametes/values of a `[COM] interface` type `IFoo` in Slang translate to `IFoo*` in C++ -* Methods that have a `throws HRESULT` clause are transformed to have an `HRESULT` return type and an output parameter for their result - -A Slang `class` can declare that it implements zero or more `[COM] interface`s. Code like this: - -``` -class MyTexture : ITexture -{ - // ... -} - -class MyDevice : IDevice -{ - ITexture createTexture(__read TextureDesc desc) throws HRESULT - { - return ...; - } - - void setTexture(int index, ITexture texture) - { - ...; - } -} -``` - -translates into output C++ like this: - -``` -class MyTexture : public slang::Object, public ITexture -{ - // ... -}; - -class MyDevice : public slang::Object, public IDevice -{ - virtual HRESULT QueryInterface(REFIID riid, void **ppvObject) { ... } - virtual ULONG AddRef() { ... } - virtual ULONG Release() { ... } - - HRESULT createTexture(TextureDesc const& desc, ITexture** _result) SLANG_OVERRIDE - { - _result = ...; - return S_OK; - } - - void setTexture(int index, ITexture* texture) SLANG_OVERRIDE - { - ... - } -} -``` - -All Slang `class`es translate to C++ classes that inherit from `slang::Object` (equivalent to the `RefObject` type within the current Slang implementation). -A `class` that inherits any `[COM]` interfaces includes an implementation of `IUnknown` plus the methods to override all the interface requirements. - -In ordinary code that makes use of `[COM] interface` types: - -``` -struct Stuff -{ - ITexture t; -} -ITexture getTexture(Stuff stuff) -{ - return stuff.t; -} -ITexture someOperations(IDevice device) -{ - let t = device.createTexture(...); - return t; -} -``` - -the C++ output uses C++ smart-pointer types for local variables, `struct` fields, and function results: - -``` -struct Stuff -{ - ComPtr t; -}; -ComPtr getTexture(Stuff stuff) -{ - return stuff.t; -} -HRESULT someOperations(IDevice* device, ComPtr& _result) -{ - ComPtr t; - HRESULT _err = device->createTexture(&t); - if(_err < 0) return _err; - - _result = t; - return 0; -} -``` - -As a small optimization, an `in` parameter of a `[COM] interface` type translates as a C++ parameter of just the matching pointer type (see `device` above). - -Note that the translation of idiomatic `HRESULT` return codes into `throws HRESULT` functions in Slang allows code working with COM interfaces to benefit from the convenience of the Slang error handling model. - - -Detailed Explanation --------------------- - -This is a case where the simple explanation above covers most of the interesting stuff. - -There are a lot of semantic checks we'd need/want to implement to make sure `[COM]` interfaces are used correctly: - -* Any `interface` that inherits from one or more other `[COM]` interfaces must itself be `[COM]` - -* Any concrete type that implements a `[COM]` interface must be a `class` - -There are also detailed implementation questions to be answered around the in-memory layout of `class` types that implement `[COM]` interfaces. -In particular, we might want to be able to optimize for the case of a single-inheritance `class` hierarchy that mirrors a `[COM] interface` hierarchy, since this comes up often for COM-based APIs: - -``` -interface IBase { void baseFunc(); } -interface IDerived : IBase { void derivedFunc(); } - -class BaseImpl : IBase { ... } -class DerivedImpl : BaseImpl, IDerived { ... } -``` - -Using a naive translation to C++, the `DerivedImpl` type could end up with *three* different virtual function table (`vtbl`) pointers embedded in it: one for `slang::Object`, one for `IBase`, and one for `IDerived`. -Clearly the `vtbl`s for `IBase` and `IDerived` could be shared, but C++ `class`es can't easily express this. -Furthermore, if we are able to tune our strategy for layout, we can set things up so that `[COM] interface`s consume `vtbl` slots starting at index `0` and counting up, while any `virtual` methods in `class`es consume slots starting at index `-1` and counting *down*. -Using such a layout strategy we can actually allow a type like `DerivedImpl` above to use only a *single* `vtbl` pointer. - -Questions --------------- - -### Should we emit COM code that works at the plain C level, or idiomatic C++? - -I honestly don't know. Emitting idomatic C++ (and using things like smart pointers) certainly makes the output code easier to understand. - -### Can we make this work with more advanced features of Slang interfaces? - -Some Slang features don't have perfect analogues. For example, given that `[COM] interface`s can only be conformed to by `class` types, the use of `[mutating]` isn't especially meaningful. - -There is no reason why a `[COM] interface` couldn't make use of `static` methods, but there would be no way to call those from C++ without an instance of the interface type. - -A `[COM] interface` could include `property` declarations, provided that we define the rules for how they translate into getter/setter operations in the generated output. - -One interesting case is that a `[COM] interface` could allow use of `This`, as well as `associatedtype`s that are themselves constrained by a `[COM] interface`. -For example, we could instead device our `IDevice` interface from before as: - -``` -[COM] interface IDevice -{ - associatedtype Texture : ITexture; - - Texture createTexture(__read TextureDesc desc) throws HRESULT; - - void setTexture(int index, Texture texture); -} -``` - -We could set things up so that the `associatedtype` has no impact on the C++ translation of `IDevice`: all parameter/result types that use the `Texture` associated type would translate to `ITexture*` in C++. -As such, a more refined `interface` like this would not disrupt the binary interface of a COM-based API, but could be allowed to express more of the constraints of the underlying API at compile time. -For example, the above use of `associatedtype` would prevent Slang code from mixing up textures across devices: - -``` -void someFunc( IDevice a, IDevice b ) -{ - let x = a.createTexture(...); - let y = b.createTexture(...); - - a.setTexture(0, y); // COMPILE TIME ERROR! -} -``` - -In this example, `y` has type `b.Texture`, while `a.setTexture` expects an argument of type `a.Texture` (a distinct type, even if it also conforms to `ITexture`). -The benefits of this approach are probably purely hypothetical until we make it a lot easier to work with dependent types like `a.Texture` in Slang code. - -Alternatives Considered ------------------------ - -The main alternative would be to have Slang's model for interop with C/C++ focus primarily on C alone, and only allow use of COM-based APIs through C-compatible interfaces. diff --git a/docs/proposals/005-components.md b/docs/proposals/005-components.md deleted file mode 100644 index b257140a7..000000000 --- a/docs/proposals/005-components.md +++ /dev/null @@ -1,507 +0,0 @@ -Components -========== - -We propose to extend Slang with a construct for defining coarse-grained *components* that can be used to assemble shader programs. - -Status ------- - -Under discussion. - -Background ----------- - -First, a bit of terminology. In the context of a specific language like Slang, a term like "component" or "module" will often have a narrow meaning, but when we want to discuss "modularity" broadly we need to have a way to refer to the *things* we want to have be modular: the units of modularity. -In this document we will use the term *unit* to refer abstractly to anything that is a unit of modularity for some context/purpose/system, and try to reserve other terms for cases where we mean something more specific. - -While Slang has many features that address modularity for "small" units, it is still lacking in constructs that adequately address the needs of "large" units. -These are qualitative distinctions, but some examples may clarify the kind of distinction we mean. -An interface `ILight` for light sources, and a `struct` type `OmniLight` that conforms to it are small units. -An entire `LightSampling` module in a path tracer is a much larger unit. - -The main tools that Slang provides for "small" units are: `interface`s, `struct`s, and `ParameterBlock`s. -Interfaces allow a developer to codify the types and operations that clients of a unit may rely on, as well as the requirements that implementations must provide. -Structure types are the main way developers can implement an interface, and it is important for GPU efficiency that Slang `struct`s are *value types*. -By using `ParameterBlock`s, developers can connect the units of modularity used *within* shader code with the parameters passed from *outside* their shaders. - -When we talk about "large" units of modularity, the main thing Slang provides are, well, modules. -A Slang module is basically just a collection of global-scope declarations: types, functions, shader parameters, and entry points. -The `import` construct allows modules to express a dependency on one another, and if/when we add `public`/`internal` visibility qualifiers it will also be able to restrict clients of a module to its defined interface. - -What modules *don't* provide is any of the flexibility that `interface`s provide for "small" units like `struct` types. -There is no first-class way for a Slang programmer to define a common interface that multiple modules implement, and then to express another piece of code that can work with any of those implementations. -Aside from falling back to preprocessor hackery (which negates many of the benefits of Slang in terms of separate compilation), the only way for developers to try to recoup those benefits is to use the tools for "small" units. - -Let's consider a placeholder/pseudocode set of Slang modules that work together: - -``` -// Lighting.slang - -interface ILight { ... } - -StructuredBuffer gLights; - -void doLightingStuff() { ... } -``` - -``` -// Materials.slang -import Lighting; - -interface IMaterial { ... } -StructuredBuffer gMaterials; - -void doMaterialStuff() -{ - doLightingStuff(); -} -``` - -``` -// Integrator.slang -import Lighting; -import Materials; - -ParameterBlock gIntegratorParams; - -void doIntegeratorStuff() -{ - doLightingStuff(); - doMaterialsStuff(); -} -``` - -The details of the module implementations is not the important part here. -The key is that each module defines a collection of types, operations, and shader parameters, and there is a dependency relationship between the modules. -Note that the `Integrator` module depends on both `Lighting` and `Materials`, but it does *not* need to be actively concerned with the fact that `Materials` *also* depends on `Lighting`. - -If we look only at a leaf module like `Lighting` it seems simple enough to translate it into something based on our "small" modularity features: - -``` -// Lighting.slang - -interface ILight { ... } - -interface ILightingSystem -{ - void doLightingStuff(); -} - -struct DefaultLightingSystem : ILightingSystem -{ - StructuredBuffer lights; - - void doLightingStuff() { ... } -} -``` - -Here we were able to move most of the global-scope code in `Lighting.slang` into a `struct` type called `DefaultLightingSystem`. -We were also able to define an explicit `interface` for the system, which makes explicit that we don't consider `gLights` part of the public interface of the system (only `doLighting()`). -By defining the interface, we also create the possibility of plugging in other implementations of `ILightingSystem` - for example, we can imagine a `NullLightingSystem` that actually doesn't perform any lighting (perhaps useful for performane analysis). - -Translating `Materials` in the same way that we did for `Lighting` leads to some immediate questions: - -``` -// Materials.slang -import Lighting; - -interface IMaterial { ... } - -interface IMaterialSystem -{ - void doMaterialStuff(); -} - -struct DefaultMaterialSystem -{ - StructuredBuffer materials; - - void doMaterialStuff() - { - /* ???WHAT GOES HERE??? */.doLightingStuff(); - } -} -``` - -When our `DefaultMaterialSystem` wants to invoke code for lighting, it needs a way to refer to the lighting system. -Beyond that, we want it to be able to work with *any* implementation of `ILightingSystem`. - -A naive first attempt might be to give `DefaultMaterialSystem` field that refers to an `ILightingSystem`: - -``` -struct DefaultMaterialSystem -{ - ILightingSystem lighting; - ... - void doMaterialStuff() - { - lighting.doLightingStuff(); - } -} -``` - -An approach light that (or even one using a `ParameterBlock`) runs into the problem that it is going to force the Slang compiler to use its layout strategy for dynamic dispatch, which cannot handle the resource types in `DefaultLightingSystem` when compiling for most current GPU targets. -There are other problems with directly aggregating an `ILightingSystem` into our material system, but those will need to wait for a bit. - -If we want to allow the code in our `DefaultMaterialSystem` to statically specialize to the type of the lighting system, we end up to use generics, either by making the whole type generic: - -``` -struct DefaultMaterialSystem< L : ILightingSystem > -{ - ParameterBlock lighting; - ... -} -``` - -or by just making the `doMaterialStuff()` operation generic, with the lighting system being passed in as a parameter. - -``` -struct DefaultMaterialSystem -{ - ... - void doMaterialStuff< L : ILightingSystem>( L lighting ) - { - lighting.doLightingStuff(); - } -} -``` - -Each of those options moves the responsibility for managing the lighting system type up a level of abstraction: whatever code works with a material system needs to manage the details. - -When we now step up the next level to the `Integrator` module, the approach using `struct`s really starts to show cracks. -We have the option of making the `DefaultIntegeratorSystem` a generic on the type of *both* subsystems, and reference them via parameter blocks: - -``` -struct DefaultIntegratorSystem< L : ILightingSystem, M : IMaterialSystem > -{ - ParameterBlock lighting; - ParameterBlock material; - ... -} -``` - -or we have to make all the relevant operations on the integrator take both subsystems as pass the relevant subsystem instances in as parameters: - -``` -struct DefaultIntegratorSystem -{ - ... - void doIntegeratorStuff< L : ILightingSystem, M : IMaterialSystem >( - L lighting, - M material) - { - lighting.doLightingStuff(); - material.doMaterialsStuff(lighting); - } -} -``` - -In each case, more and more responsibiity for configuration of implementation details is being punted up to the next higher level of abstraction. -In the first case, somebody else is responsible for instantiating a type like: - -``` -DefaultIntegeratorSystem> -``` - -Also, the application code that works with that messy type needs to make sure to fill in *one* parameter block for the lighting system, but set it into *both* the material system and integrator. - -In the second case, note how the integrator already has to ensure that it passes along the `lighting` subsystem to the `doMaterialStuff` operation, and anybody who invokes `doIntegratorSutff` would have to do the same, but for *two* subsystems. - -The whole thing doesn't scale and becomes intractable with more than a few subsystems. -Trying to work anything like inheritance into the mix just falls flat completely. - -Related Work ------------- - -There is a lot of work in general-purpose programming languages around defining larger-scale modularity units. - -SML (Standard ML) has both modules and *signatures*, which are effectively interfaces for modules. -Modules can be parameterized on other modules based on signatures, and instantiated to use different concrete implementations. - -Beta and gbeta unify both classes and functions into a single construct called a *pattern*, and show that patterns (including pattern inheritance) can be used for things akin to traditional modules. -A variety of techniques for *family polymorphism* in world of Java and similar languages followed on from that tradition. -The Scala language continues in the same vein, with papers and presentations on Scala advocating for using `class`es to model large units of modularity akin to modules. - -In the world of "enterprise" software using Java, C#, JavaScript, etc. there is a large family of techniques and system for "dependency inversion" which is used to automate some or all of the process of "wiring up" the concrete implementations of various subsystems/components based on explicit representations of dependencies (often attached as metadata on the fields of a type). - -Modern general-purpose game engines like Unity and Unreal often use a "component" concept, where a game entity/object is composed of multiple loosely-coupled sub-objects (components). -Often these systems allow dependencies between component types to be stated explicitly, with runtime or tools support for ensuring that objects are not created with unsatisifed dependencies. - -Note that almost all of the approaches enumerated above rely deeply on the fact that a dependency of unit `X` on unit `Y` can be handily represented as a single pointer/reference in most general-purpose programming languages. For example, in C++: - -``` -class Y { ... }; -class X -{ - Y* y; - ... -}; -``` - -In the above, an instance of `X` can always find the `Y` it depends on easily and (relatively) efficiently. -There is no particularly high overhead to having `X` diretly store an indirect reference to `Y` (at least not for coarse-grained units), and it is trivial for multiple units like `X` to all share the same *instance* of `Y` (potentially even including mutable state, for applications that like that sort of thing). - -In general most CPU languages (and especially OOP ones) can express the concepts of "is-a" and "has-a" but they often don't distinguish between when "has-as" means "refers-to-and-depends-on-a" vs. when it means "aggregates-and-owns-a". -This is important when looking at a GPU language like Slang, where "aggergates-and-owns-a" is easy (we have `struct` types), but "refers-to-and-depends-on-a" is harder (not all of our targets can really support pointers). - -Proposed Approach ------------------ - -We propose to introduce a new construct called a *component type* to Slang, which can be used to describe units of modularity larger than what `struct`s are good for, but that is intentionally defined in a way that allows it to be used in cases where fully general `class` types could not be supported. - -To render our earlier examples in terms of component types: - -``` -// Lighting.slang - -interface ILight { ... } - -interface ILightingSystem -{ - void doLightingStuff(); -} - -__component_type DefaultLightingSystem : ILightingSystem -{ - StructuredBuffer lights; - - void doLightingStuff() { ... } -} -``` - -``` -// Materials.slang -import Lighting; - -interface IMaterial { ... } -interface IMaterialSystem -{ - void doMaterialStuff(); -} - -__component_type DefualtMaterialSystem : IMaterialSystem -{ - __require lighting : ILightingSystem; - - StructuredBuffer materials; - - void doMaterialStuff() - { - lighting.doLightingStuff(); - } -} -``` - -``` -// Integrator.slang -import Lighting; -import Materials; - -interface IIntegratorSystem -{ - void doIntegratorStuff(); -} - -__component_type DefaultIntegratorSystem : IIntegratorSystem -{ - __require ILightingSystem; - __require IMaterialSystem; - - ParameterBlock params; - - void doIntegeratorStuff() - { - doLightingStuff(); - doMaterialsStuff(); - } -} -``` - -The `__component_type` keyword is akin to `struct` or `class`, but introduces a component type. -Component types are similar to both structure and class types in that they can: - -* Conform to zero or more interfaces -* Define fields, methods, properties, and nested types -* Eventually: optionally inherit from one (or more) other component types - -The key thing that a `__component_type` can do that a `struct` cannot (but that a `class` might be allowed to) is include nested `__require` declarations. -In the simplest form, a require declaration is of the form: - -``` -__require someName : ISomeInterface; -``` - -Within the scope where the `__require` is visible, `someName` will refer to a value that conforms to `ISomeInterface`, but code need not know what the value is (nor what its type is). -The other form of `__require`: - -``` -__require ISomeInterface; -``` - -Can be seen as shorthand for something like: - -``` -__require _anon : ISomeInterface; -using anon.*; // NOTE: not actual Slang syntax -``` - -One more construct is needed to complete the feature, and it can introduced by illustrating a concerete type that pulls together our default implementations: - -``` -__component_type MyProgram -{ - __aggregate lighting : DefaultLightingSystem; - __aggregate DefaultMaterialSystem; - __aggregate DefaultIntegeratorSystem; -} -``` - -An `__aggregate` declaration is only allowed inside a `__component_type` (or a `class`, if we allow it). -Similar to `__require`, an `__aggregate` can either name the thing being aggregated, or leave it anonymous (and have its members imported into the current scope). -The semantics of `__aggregate SomeType` are similar to just declaring a field of `SomeType`, but the key distinction is that the aggregated sub-object is conceptually allocated and initialized as *part* of the outer object (one alternative name for the keyword would be `__part`). -It is not possible to assign to an `__aggregate` member like it is a field (although if the type is a reference type, *its* fields are visible and might still be mutable). - -At the point where an `__aggregate SomeType` member is declared, the front-end semantic checking must be able to find/infer the identity of a value to use to satisfy each `__require` member in `SomeType`. -For example, because `DefaultIntegratorSystem` declares `__require IMaterialSystem`, the compiler searches in the current context for a value that can provide that interface. -It finds a single suitable value: the value implicitly defined by `__aggregate DefaultMaterialSystem`, and thus "wires up" the input dependency of the `DefaultIntegratorSystem`. - -It is posible for a `__require` in an `__aggregate`d member to be satisfied via another `__require` of its outer type: - -``` -__component_type MyUnit -{ - __require ILightingSystem; - __aggregate DefaultMaterialSystem; -} -``` - -In the above example, the `ILightingSystem` requirement in `DefaultMaterialSystem` will be satisfied using the `ILightingSystem` `__require`d by `MyUnit`. - -In cases where automatic search and connection of dependencies does not work (or yields an ambiguity error), the user will need some mechanism to be able to explicitly specify how dependencies should be satisfied. - -While the above examples do not show it, component types should be allowed to contain shader entry points. - -Detailed Explanation --------------------- - -Component types need to be restricted in where and how they can be used, to avoid creating situations that would give them all the flexiblity of arbitrary `class`es. -The only places where a component type may be used are: - -* `__require` and `__aggregate` declarations -* Function parameters -* Generic arguments (??? Need to double-check how this can go wrong) - -Any given `__component_type` either has no `__require`s and is thus concrete, or it has a nonzero number of `__require`s and is abstract. - -We can ignore the `__require`s in a component type (if any) and form an equivalent `struct` type. -In that `struct` type, `__aggregate`s turn into ordinary fields. -For example, the `MyProgram` (concrete) and `MyUnit` (abstract) types above become: - -``` -struct MyProgram -{ - DefaultLightingSystem lighting; - DefaultMaterialSystem _anon0; - DefaultIntegeratorSystem _anon1; -}; -struct MyUnit -{ - DefaultMaterialSystem _anon2; -}; -``` - -When a component type is used as a function parameter (including an implicit `this` parameter) it effectively maps to a function that takes additional (generic) parameters corresponding to each `__require`. -For example, given: - -``` -void doStuff( MyUnit u ) { ... u.doMaterialStuff(); ... } -``` - -we would generate something like: - -``` -void doStuff< T : ILightingSystem >( - T _anon3, // for the `__require : ILightingSystem` in `MyUnit` - MyUnit u) -{ - ... - DefaultMaterialSystem_doMaterialStuff(_anon3, u._anon2); - ... -} -``` - -Note that when the generated code invokes an operation through one of the `__aggregate`d members of a component type, where the aggregated type had `__require`ments, the compiler must pass along the additional parameters that represent those requirements in the current context. - -Effectively, the compiler generates all of the boilerplate parameter-passing that the programmer would have otherwise had to write by hand. - -It might or might not be obvious that the notion of "component type" being described here has a clear correspondance to the `IComponentType` interface provided by the Slang runtime/compilation API. -It should be possible for us to provide reflection services that allow a programmer to look up a component type by name and get an `IComponentType`. -The existing APIs for composing, specializing, and linking `IComponentType`s should Just Work for explicit `__component_type`s. -Large aggregates akin to `MyProgram` above can be defined entirely via the C++ `IComponentType` API at program runtime. - -Questions ---------- - -### How do we explain to users when to use component types vs. when to use modules? Or `struct`s? - -The basic advice should be something like: - -* If the thing feels subsystem-y, favor modules or component types. If it feels object-y or value-y, use a `struct`. This is loose, but intuition is good here. - -* If the thing wants to depend on other subsystems through `interface`s, to allow mix-and-match flexibility, it should probably be a component type and not a module. - -* If the thing wants to have its own shader parameters, then we encourage users to consider that a component type is likely to be what they want, so that they don't pollute the global scope. - -That last point is important, since a component type allows users to define a collection of global shader parameters and entry points that use them as a unit, without putting those parameters in the global scope, which is something that was not really possible before. - -### Can the `__component_type` construct just be subsumed by either `struct` or `class`? - -Maybe. The key challenge is that component types need to provide the "look and feel" of by-refernece re-use rather than by-value copying. A `__require T` should effectively act like a `T*` and not a bare `T` value, so I am reluctant to say that should map to `struct`. - -### But what about `[mutating]` operations and writing to fields of component types, then? - -Yeah... that's messy. If component types really are by-reference, then they should be implicitly mutable even without passing as `inout`, and should ideally also support aliasing. We need to make sure we get clarity on this. - -### Is `__aggregate` really required? Isn't it basically just a field? - -An `__aggregate X` acts a lot like a field if `X` is a *value* type, but in cases where `X` is a *reference* type, there is a large semantic distinction. - -### How does all this stuff relate to inheritance? - -There are some things that can be done with (multiple) inheritance that can also be expressed via `__require`s. For example, both can represent the "diamond" pattern: - -``` -class A { ... } -class B : A { ... } -class C : A { ... } -class D : B, C { ... } -``` - -``` -__component_type A { ... } -__component_type B { __require A; ... } -__component_type C { __require A; ... } -__component_type D { __require B; __require C; ... } -``` - -The Spark shading language research project used multiple mixin class inheritance to compose units of shader code akin to what are being proposed here as coponent types (hmm... I guess that should go into related work...). - -In general, using inheritance to model something that isn't an "is-a" relationship is poor modeling. -Inheritance as a modelling tool cannot capture some patterns that are possible with `__aggregate` (notably, with mixin inheritance you can't get multiple "copies" of a component). -Most importantly, when inheritance is abused for modeling like this, the resulting code can be confusing. Consider: - -``` -abstract class MyFeature : ISystemA, ISystemB -{ ... } -``` - -From this declaration, it is not possible to tell whether `MyFeature` implements just `ISystemA`, just `ISystemB`, both, or neither. -The distinction between an inheritance clause ("I implement this thing") vs. a `__require` ("I need *somebody else* to implement this thing") is important documentation. - -Alternatives Considered ------------------------ - -I'm not aware of any big design alternatives that don't amount to more or less the same thing with different syntax. -One alternatives is to try to do something like ML-style "signature" for our modules, and allow something like `import ILighting` to allow module-level dependencies on abstracted interfaces. - -Another alternative is to do what this document proposes, but make it work with the existing `struct` keyword (or `class`) instead of adding a new one. diff --git a/docs/proposals/006-artifact-container-format.md b/docs/proposals/006-artifact-container-format.md deleted file mode 100644 index 81910151b..000000000 --- a/docs/proposals/006-artifact-container-format.md +++ /dev/null @@ -1,1119 +0,0 @@ -Shader Container Format -======================= - -This proposal is for a file hierarchy based structure that can be used to represent compile results and more generally a 'shader cache'. Ideally it would feature - -* Does not require an extensive code base to implement -* Flexible and customizable for specific use cases -* Possible to produce a simple fast implementation -* Use simple and open standards where appropriate -* Where possible/appropriate human readable/alterable -* A way to merge, or split out contents that is flexible and easy. Ideally without using Slang tooling. - -Should be able to store - -* Compiled kernels -* Reflection/layout information -* Diagnostic information -* "meta" information detailing user specific, and product specific information -* Source -* Debug information -* Binding meta data -* Customizable, and user specified additional information - -API support needs to allow - -* Interchangeable use of static shader cache/slang compilation/combination of the two - * Implies compilation needs to be initiated in some way that is compatible with shader cache keying -* Ability to store compilations as they are produced - -Needs to be able relate and group products such that with suitable keys, it is relatively fast and easy to find appropriate results. - -It's importance/relevance - -* Provides a way to represent complex compilation results -* Could be used to support an open standard around 'shader cache' -* Provide a standard 'shader cache' system that can be used for Slang tooling and customers -* Supports Slang tooling and language features - -## Use - -There are several kinds of usage scenario - -* A runtime shader cache -* A runtime shader cache with persistence -* A capture of compilations -* A baked persistent cache - must also work without shader source -* A baked persistent cache, that is obfuscated - -A runtime shader cache has the following characteristics: - -* Can works with mechanisms that do not require any user control (such as naming). Ie purely inputs/options can define a 'key'. -* It is okay to have keys/naming that are not human understandable/readable. -* The source is available - such that hashes based on source contents can be produced. -* It does not matter if hashes/keys are static between runs. -* It is not important that a future version would be compatible with a previous version or vice versa. -* Could be all in memory. -* May need mechanism/s to limit the working set -* Generated source can be made to work, because it is possible to hash generated source - -At the other end of the spectrum a baked persistent cache - -* Probably wants user control over naming -* Doesn't have access to source so can't use that as part of a hash -* Probably doesn't have access to dependencies -* Having some indirection between a request and a result is a useful feature -* Ideally can be manipulated and altered without significant tooling -* Generated source may need to be identified in some other way than the source itself - -It should be possible to serialize out a 'runtime shader cache' into the same format as used for persistent cache. It may be harder to use such a cache without Slang tooling, because the mapping from compilation options to keys will probably not be simple. - -Status ------- - -## Gfx - -There is a run time shader cache that is implemented in gfx. - -There is some work around a file system backed shader cache in gfx. - -## Artifact System - -The Artifact provides a mechanism to transport source/compile results through the Slang compiler. It already supports most of the different items that need to be stored. - -Artifact has support for "containers". An artifact container is an artifact that can contain other artifacts. Support for different 'file system' style container formats is also implemented. The currently supported underlying container formats supported are - -* Zip -* Riff - * Riff without compression - * Riff with deflate - * Riff with LZ4 compression - -Additionally the mechanisms already implemented support - -* The OS filesystem -* A virtual file system -* A 'chroot' of the file system (using RelativeFileSystem) - -In order to access a file system via artifact, is as simple as adding a modification to the default handler to load the container, and to implement `expandChildren`, which will allow traversal of the container. In general this works in a 'lazy' manner. Children are not expanded, unless requested, and files not decompressed unless required. The system also provides a caching mechanism such that a representation, such as uncompressed blob, can be associated with the artifact. - -Very little code is needed to support this behavior because the IExtFileArtifactRepresentation and the use of the ISlangFileSystemExt interface, mean it can work using the existing mechanisms. - -It is a desired feature of the container format that it can be represented as 'file system', and have the option of being human readable where appropriate. Doing so allows - -* Third party to develop tools/formats that suit their specific purposes -* Allows different containers to used -* Is generally simple to understand -* Allows editing and manipulating of contents using pre-existing extensive and cross platform tools -* Is a simple basis - -This documents is, at least in part, about how to structure the file system to represent a 'shader cache' like scenario. - -Incorporating the 'shader container' into the Artifact system will require a suitable Payload type. It may be acceptable to use `ArtifactPayload::CompileResults`. The IArtifactHandler will need to know how to interpret the contents. This will need to occur lazily at the `expandChildren` level. This will create IArtifacts for the children that some aspects are lazily evaluated, and others are interpreted at the expansion. For example setting up the ArtifactDesc will need to happen at expansion. - -Background -========== - -The following long section provides background discussion on a variety of topics. Jump to the [Proposed Approach](#proposed-approach) to describe what is actually being suggested in conclusion. - -To enumerate the major challenges - -* How to generate a key for the runtime scenario -* How to produce keys for the persistent scenario - implies user control, and human readability -* How to represent compilation in a composable 'nameable' way -* How to produce options from a named combination - -The mechanism for producing keys in the runtime scenario could be used to check if an entry in the cache is out of date. - -A compilation can be configured in many ways. Including - -* The source including source injection -* Pre-processor defines -* Compile options - optimization, debug information, include paths, libraries -* Specialization types and values -* Target and target specific features API/tools/operating system -* Specific version of slang and/or downstream compilers -* Pipeline aspects -* Slang components - -In general we probably don't want to use the combination of source and/or the above options as a 'key'. Such a key would be hard and slow to produce. It would not be something that could be created and used easily by an application. Moreover it is commonly useful to be able to name results such that the actual products can be changed and have things still work. - -Background: Hashing source -========================== - -Hashing source is something that is needed for runtime cache scenario, as it is necessary to generate a key purely from 'input' which source is part of. It can also be used in the persistent scenario, in order to validate if everything is in sync. That sync checking might perhaps only be performed when source and other resources are available. - -The fastest/simplest way to hash source, is to take the blob and hash that. Unfortunately there are several issues - -* Ignores dependencies - if this source includes another file the hash will also need to depend on that transitively -* Hash changes with line end character encoding -* Hash is sensitivve to white space changes in general - -A way to work around whitespace issues would be to use a tokenizer, or a 'simplified' tokenizer that only handles the necessary special cases. An example special case would be white space in a string is always important. Such a solution does not require an AST or rely on a specific tokenizer. A hash could be made of concatenation of all of the lexemes with white space inserted between. - -Another approach would be to hash each "token" as produced. Doing so doesn't require memory allocation for the concatenation. You could special case short strings or single chars, and hash longer strings. - -## Dependencies - -Its not enough to rely on hashing of input source, because `#include` or other resource references, such as modules or libraries may be involved. - -If we are are relying on dependencies specified at least in part by `#include`, it implies the preprocessor be executed. This could be used for other languages such as C/C++. Some care would need to be taken because *some* includes will probably not be located by our preprocessor, such as system include paths in C++. For the purpose of hashing, an implementation could ignore `#includes` that cannot be resolved. This may work for some scenarios - but doesn't work in general because symbols defined in unfound includes might cause other includes. Thus this could lead to other dependencies not being found, or being assumed when they weren't possible. - -In practice whilst not being perfect it may work well enough to be broadly usable. - -## AST - -A hash could be performed via the AST. This assumes - -1) You can produce an AST for the input source - this is not generally true as source could be CUDA, C++, etc -2) The AST would have to be produced post preprocessing - because prior to preprocessing it may not be valid source -3) If 3rd parties are supposed to be able to produce a hash it requires their implementing a Slang lexer/parser in general -4) Depending on how the AST is used, may not be stable between versions - -Another disadvantage around using the AST is that it requires the extra work and space for parsing. - -Using the AST does allow using pre-existing Slang code. It is probably more resilient to structure changes. It would also provide slang specific information more simply - such as imports. - -## Slang lexer - -If we wanted to use Slang lexer it would imply the hash process would - -1) Load the file -2) Lex the file -3) Preprocess the file (to get dependencies). Throw these tokens away (we want the hash of just the source) -4) Hash the original lex of the files tokens -5) Dependencies would require hashing and combining - -For hashing Slang language and probably HLSL we can use the Slang preprocessor tokenizer, and hash the tokens (actually probably just the token text). - -Using the Slang lexer/preprocessor may work reasonably for other languages such as C++/C/HLSL/GLSL. It does imply a reliance on a fairly large amount of slang source. - -## Simplified Lexer - -We may want to use some simple lexer. A problem with using a lexer at all is that it adds a great amount of complexity to a stand alone implementation. The simplified lexer would - -* Simplify white space - much we can strip -* Honor string representations (we can't strip whitespace) -* Honor identifiers -* We may want some special cases around operators and the like -* Honor `#include` (but ignore preprocessor behavior in general) -* Ignore comments - -We need to handle `#include` such that we have dependencies. This can lead to dependencies that aren't required in actual compilation. - -We need to honor some language specific features - such as say `import` in Slang. - -Such an implementation would be significantly simpler, and more broadly applicable than Slang lexer/parser etc. Writing an implementation would determine how complex - but would seem to be at a minimum 100s of lines of code. - -We can provide source for an implementation. We could also provide a shared library that made the functionality available via COM interface. This may help many usage scenarios, but we would want to limit the complexity as much as possible. - -## Generated Source - -Generated source can be part of a hash if the source is available. As touched on there are scenarios where generated source may not be available. - -We could side step the issues around source generation if we push that problem back onto users. If they are using code generation, the system could require providing a string that uniquely identifies the generation that is being used. This perhaps being a requirement for a persistent cache. For a temporary runtime cache, we can allow hash generation from source. - -Background: Hashing Stability -============================= - -Ideally a hashing mechanism can be resilient to unimportant changes. The previous section described some approaches for changes in source. The other area of significant complexity is around options. If options are defined as JSON (or some other 'bag of values') hashing can be performed relatively easily with a few rules. If the representation is such that if a value is not set, the default is used, it is resilient to changes of options that are explicitly set. - -When the hashing is on some native representation this isn't quite so simple, as a typical has function will include all fields. A field value, default or not will alter the hash. Therefore adding or removing a field will necessarily change the hash. - -One way around this would be to use a hashing regime that only altered the hash if the values are not default. - -```C++ - -Hash calcHash(const Options& options) -{ - const Options defaultOptions; - - Hash = ...; - if (option.someOption != defaultOption.someValue) - { - hash = combineHash(hash, option.someOption.getHash()); - } - // ... -} -``` - -This could perhaps be simplified with some macro magic. - -``` - -// Another - -struct HashCalculator -{ - template - void hashIfDifferent(const field& f, const T& defaultValue) - { - if (value != defaultValue) - { - hash = combineHash(hash, value.getHash()); - } - } - - const T defaultValue; - const T* value; - Hash hash; -}; - -Hash calcHash(const Options& options) -{ - HashCalculator calc; - const Options defaultOptions; - - calc.hashIfDifferent(options.someOption, defaultOptions.someOption); - // ... - Hash = ...; - -} -``` - -This is a little more clumsy, but if we wanted to use a final native representation, it is workable. - -Note that the ordering of hashing is also important for stability. - -Background: Key Naming -====================== - -The container could be seen as a glorified key value store, with the key identifying a kernel and associated data. - -Much of the difficulty here is how to define the key. If it's a combination of the 'inputs' it would be huge and complicated. If it's a hash, then it can be short, but not human readable, and without considerable care not stable to small or irrelevant changes. - -For a runtime cache type scenario, the instability and lack of human readability of the key probably doesn't matter too much. It probably is a consideration how slow and complicated it is to produce the key. - -For any cache that is persistent how naming occurs probably is important. Because - -* Our 'options' aren't going to make much sense with other compilers (if we want the standard to be more broadly applicable) -* The options we have will not remain static -* Having an indirection is useful from an application development and shipping perspective -* That the *name* perhaps doesn't always have to indicate every aspect of a compilation from the point of view of the application - -One idea touched on in this document is to move 'naming' into a user space problem. That compilations are defined by the combination of 'named' options. In order to produce a shader cache name we have a concatination of names. The order can be user specified. Order could also break down into "directory" hierarchy as necessary. - -Some options will need to be part of some order. This is perhaps all a little abstract so as an example - -```JSON -{ - // Configuration - - "configuration" : { - - "debug" : { - "group" : "configuration", - "optimization" : "0", - "debug-info" : true, - "defines" : [ "-DDEBUG=1" ] - }, - "release" : { - "optimization" : "2", - "debug-info" : true, - "defines" : [ "-DRELEASE=1" ] - }, - "full-release" : { - "optimization" : "3", - "debug-info" : false, - "defines" : [ "-DRELEASE=1", "-DFULL_RELEASE=1" ] - } - }, - - // Target - "target" : { - "vk" : { - } - "d3d12" : { - } - - "cpu" : { - } - }, - - // Stage - "stage" : { - "compute" : { - }, - }, - - combinations : [ - { - key : [ "vk", "compute", ["release", "full-release"] ], - options : - { - "optimization" : 1 - } - } - ] -} -``` - -The combination in this manner doesn't quite work, because some combinations may imply different options. The "combinations" section tries to address this by providing a way to 'override' behavior. This could of course be achieved with a call back mechanism. We may also want to have options that don't appear in the key, allowing 'overriding' behavior without needing a 'combinations' section. The implication is that when used in the application when looking up only the 'named' configuration is needed. - -This whole mechanism provides a way of specifying a compilation by a series of names, that can produce a unique human readable key. It is under user control, but the mechanism on how the combination takes place is at least as a default defined within an implementation. - -It may be necessary to define options by tool chain. Doing so would mean the names can group together what might be quite different options on different compilers. Having options defined in JSON means that the mechanisms described here can be used for other tooling. If the desire is to have some more broadly applicable 'shader cache' representation this is desirable. - -If it is necessary obfuscate the contents, it would be possible to put the human readable key though a hash, and then the hash can be used for lookup. - -## Container Location - -We could consider the contents of the container as 'flat' with files for each of the keys. There could be several files related to a key if we use file association mechanism (as opposed to a single manifest). - -Whilst this works it probably isn't particularly great from an organizational point of view. It might be more convenient if we can use the directory hierarchy if at least optionally. For example putting all the kernels for a target together... - -``` -/target/name-entryPoint -``` - -Or - -``` -/target/name/entryPoint-generated-hash -``` - -Where 'generated-hash' was the hash for generated source code. - -Perhaps this information would be configured in a JSON file for the repository. - -What happens if we want to obfuscate? We could make the whole path obfuscated. We could in the configuration describe which parts of the name will be obfuscated, such that it's okay to see there are different 'target' names. - -## Default names - -When originally discussed, the idea was that all options can be named, and thus any combination is just a combination of names. That combination produces the key. - -Whilst this works, it might make sense to allow 'meta' names for common types. The things that will typically change for a fixed set of options would be - -* The input translation unit source -* The target (in the Slang sense) -* The entryPoint/stage (can we say the entry point name implies the stage?) - -We could have pseudo names for these commonly changed values. If there are multiple input source files for a translation unit, we could key on the first. - -We could also have some 'names' that are built in. For example default configuration names such as 'debug' and 'release'. They can be changed in part of configuration but have some default meaning. That options can perhaps override the defaults. - -Using the pseudo name idea might mean it is possible to produce reasonable default names. Moreover we can still use the hashing mechanism to either report a validation issue, or trigger recompilation when everything needed to do as much is available. - -## Target - -A target can be quite a complicated thing to represent. Artifact has - -* 'Kind' - executable, library, object code, shared library -* 'Payload' - SPIR-V, DIXL, DXBC, Host code, Universal, x86_64 etc... - * Version -* 'Style' - Kernel, host, unknown - -This doesn't take into account a specific 'platform', where that could vary a kernel depending on the specific features of the platform. There are different versions of SPIR-V and there are different extensions. - -This doesn't cover the breadth though because for CPU targets there is additionally - -* Operating system - including operating system version -* Tool chain - Compiler - -Making this part of the filename could lead to very long filenames. The more detailed information could be made available in JSON associated files. - -This section doesn't provide a specific plan on how to encapsulate the subtlety around a 'target'. Again how this is named is probably something that is controllable in user space, but there are some reasonable defaults when it is not defined. - -Background: Describing Options -============================== - -We need some way to describe options for compilation. The most 'obvious' way would be something like the IDownstreamCompiler interface and associated types - -```C++ -struct Options -{ - Includes ...; - Optimizations ...; - Miscellaneous ...; - Source* source[]; - EntryPoint ... ; - ... - CompilerSpecific options; -} - -ICompiler -{ - Result compile(const Options* options, IArtifact** outArtifact); -} -``` - -For this to work we need a mapping from 'options' to the cached result (if there is one). There are problem around this, because - -* Items may be specified multiple times -* Ideally the hash would or at least could remain stable with updated to options -* Also ideally the user might want control over what constitutes a new version/key -* Calculating a hash is fairly complicated, and would need to take into account ordering - -Another option might be to split common options from options that are likely to be modified per compilation. For example - -``` -struct Options -{ - const char* name; ///< Human readable name - Includes ...; - Optimizations ...; - Miscellaneous ...; - Source* source[]; - ... - CompilerSpecific options; -} - -struct CompileOptions -{ - Stage stage ...; - SpecializationArgs ...; - EntryPoint ...; -}; - -ICompiler -{ - Result createOptions(Options* options, IOptions* outOptions); - - Result compile(IOptions* options, const CompileOptions* compileOptions, IArtifact** outArtifact); -} -``` - -Having the split greatly simplifies the key production, because we can use the unique human name, and the very much simpler values of CompileOptions to produce a key. - -This specifying of options in this way is tied fairly tightly to the Slang API. We can generalize the named options by allowing more than one named option set. - -## Bag of Named Options - -Perhaps identification is something that is largely in user space for the persistent scenario. You could imagine a bag of 'options', that are typically named. Then the output name is the concatenation of the names. If an option set isn't named it doesn't get included. Perhaps the order of the naming defines the precedence. - -This 'bag of options' would need some way to know the order the names would be combined. This could be achieved with another parameter or option that describes name ordering. Defining the ordering could be achieved if different types of options are grouped, by specifying the group. The ordering would only be significant for named items that will be concatenated. The ordering of the options could define the order of precedence of application. - -Problems: - -How to combine all of these options to compile? -How to define what options are set? Working at the level of a struct doesn't work if you want to override a single option. -The grouping - how does it actually work? It might require specifying what group a set of options is in. - -An advantage to this approach is that policy of how naming works as a user space problem. It is also powerful in that it allows control on compilation that has some independence from the name. - -We could have some options that are named, but do not appear as part of the name/path within the container. The purpose of this is to allow customization of a compilation, without that customization necessarily appearing withing the application code. The container could store group of named options that is used, such that it is possible to recreate the compilation or perhaps to detect there is a difference. - -### JSON options - -One way of dealing with the 'bag of options' issue would be to just make the runtime json options representation, describe options. Merging JSON at a most basic level is straight forward. For certain options it may make sense to have them describe adding, merging or replacing. We could add this control via adding a key prefix. - -```JSON -{ - "includePaths" : ["somePath", "another/path"], - "someValue" : 10, - "someEnum" : enumValue, - "someFlags" : 12 -} -``` - -As an example - -```JSON -{ - "+includePaths" : ["yet/another"], - "intValue" : 20, - "-someValue" : null, - "+someFlags" : 1 -} -``` - -When merged produces - -```JSON -{ - "includePaths" : ["somePath", "another/path", "yet/another"], - "someEnum" : enumValue, - "someFlags" : 13, - "intValue" : 20 -} -``` - -It's perhaps also worth pointing out that using JSON as the representation provides a level of compatibility. Things that are not understood can be ignored. It is human readable and understandable. We only need to convert the final JSON into the options that are then finally processed. - -One nice property of a JSON representation is that it is potentially the same for processing and hashing. - -### Producing a hash from JSON options - -One approach would be to just hash the JSON if that is the representation. We might want a pass to filter out to just known fields and perhaps some other sanity processing. - -* Filtering -* Ordering - the order of fields is generally not the order we want to combine. One option would be to order by key in alphabetical order. -* Handling values that can have multiple representations (if we allow an enum as int or text, we need to hash with one or ther other) -* Duplicate handling - -Alternatively the JSON could be converted into a native representation and that hashed. The problem with this is that without a lot of care, the hash will not be stable with respect to small changes in the native representation. - -Another advantage of using JSON for hash production, is that it is something that could be performed fairly easily in user space. - -Two issues remain significant with this approach - -* Filtering - how? -* Handling multiple representations for values - -Filtering is not trivial - it's not a question of just specifying what fields are valid, because doing so requires context. In essence it is necessary to describe types, and then describe where in a hierarchy a type is used. - -I guess this could be achieved with... JSON. For example - -``` -{ - "types": - { - "SomeType": - { - "kind" : "struct", - "derivesFrom": "..." - fields: { - [ "name", "type", "default"] - } - }, - "MainOptions": - { - "..." - } - }, - "structure" : - { - "MainOptions" - } -} -``` - -When traversing we use the 'structure' to work out where a type is used. - -This is workable, but adds additional significant complexity. - -The issue around different representations could also use the information in the description to convert into some canonical form. - -The structure could potentially generated via reflection information. - -## Native bag of options - -Options could be represented via an internal struct on which a hash can be performed. - -Input can be described as "deltas" to the current options. The final options is the combination of all the deltas - which would produce the final options structure for use. The hash of all of the options is the hash of the final structure. - -How are the deltas described? - -The in memory representation is not trivial in that if we want to add a struct to a list we would need a way to describe this. - -Whilst in the runtime the 'field' could be uniquely identified an offset, within a file format representation it would need to be by something that works across targets, and resistant to change in contents. - -## Slangs Component System - -Slang has a component system that can be used for combining options to produce a compilation. An argument can be made that it should be part of the hashing representation, as it is part of compilation. - -If combination is at the level of components, then as long as components are serializable, we can represent a compilation by a collection of components. Has several derived interfaces... - -* IEntryPoint -* ITypeConformance -* IModule - -Can be constructed into composites, through `createCompositeComponentType`, which describes aspects of the combination takes place. - -If the components were serializable (as say as JSON), we could describe a compilation as combination of components. If components are named, a concatenation of names could name a compilation. - -It doesn't appear as if there is a way to more finely control the application of component types. For example if there was a desire to change the optimization option, it would appear to remain part of the ICompileRequest (it's not part of a component). This implies this mechanism as it stands whilst allowing composition, doesn't provide the more nuanced composition. Additional component types could perhaps be added which would add such control. - -Perhaps having components is not necessary as part of the representation, as 'component' system is a mechanism for achieving a 'bag of options' and so we can get the same effect by using that mechanism without components. - -Background: Describing Options -============================== - -The 'naming' options idea implies that options and ways of combining options can be stored within the configuration for a container. Perhaps there is additionally a runtime API that allows creation of deltas. - -## 'Bag of named options' - -Perhaps identification is something that is largely in user space for the persistent scenario. You could imagine a bag of 'options', that are typically named. Then the output name is the concatenation of the names. If an option set isn't named it doesn't get included. Perhaps the order of the naming defines the precedence. - -This 'bag of options' would need some way to know the order the names would be combined. This could be achieved with another parameter or option that describes name ordering. Defining the ordering could be achieved if different types of options are grouped, by specifying the group. The ordering would only be significant for named items that will be concatenated. The ordering of the options could define the order of precedence of application. - -Problems: - -How to combine all of these options to compile? -How to define what options are set? Working at the level of a struct doesn't work if you want to override a single option. -The grouping - how does it actually work? It might require specifying what group a set of options is in. - -An advantage to this approach is that policy of how naming works as a user space problem. It is also powerful in that it allows control on compilation that has some independence from the name. - -### JSON options - -One way of dealing with the 'bag of options' issue would be to just make the runtime JSON options representation, describe options. Merging JSON at a most basic level is straight forward. For certain options it may make sense to have them describe adding, merging or replacing. We could add this control via adding a key prefix. - -```JSON -{ - "includePaths" : ["somePath", "another/path"], - "someValue" : 10, - "someEnum" : enumValue, - "someFlags" : 12 -} -``` - -As an example - -```JSON -{ - "+includePaths" : ["yet/another"], - "intValue" : 20, - "-someValue" : null, - "+someFlags" : 1 -} -``` - -When merged produces - -```JSON -{ - "includePaths" : ["somePath", "another/path", "yet/another"], - "someEnum" : enumValue, - "someFlags" : 13, - "intValue" : 20 -} -``` - -It's perhaps also worth pointing out that using JSON as the representation provides a level of compatibility. Things that are not understood can be ignored. It is human readable and understandable. We only need to convert the final JSON into the options that are then finally processed. - -One nice property of a JSON representation is that it is potentially the same for processing and hashing. - -### Producing a hash from JSON options - -One approach would be to just hash the JSON if that is the representation. We might want a pass to filter out to just known fields and perhaps some other sanity processing. - -* Filtering -* Ordering - the order of fields is generally not the order we want to combine. One option would be to order by key in alphabetical order. -* Handling values that can have multiple representations (if we allow an enum as int or text, we need to hash with one or ther other) -* Duplicate handling - -Alternatively the JSON could be converted into a native representation and that hashed. The problem with this is that without a lot of care, the hash will not be stable with respect to small changes in the native representation. - -Another advantage of using JSON for hash production, is that it is something that could be performed fairly easily in user space. - -Two issues remain significant with this approach - -* Filtering - how? -* Handling multiple representations for values - -Filtering is not trivial - it's not a question of just specifying what fields are valid, because doing so requires context. In essence it is necessary to describe types, and then describe where in a hierarchy a type is used. - -I guess this could be achieved with... JSON. For example - -``` -{ - "types": { - "SomeType": - { - "kind" : "struct", - "derivesFrom": "..." - fields: { - [ "name", "type", "default"] - } - }, - "MainOptions": - { - "..." - } - }, - "structure" : - { - "MainOptions" - } -} -``` - -When traversing we use the 'structure' to work out where a type is used. - -This is workable, but adds additional significant complexity. - -The issue around different representations could also use the information in the description to convert into some canonical form. - -The structure could potentially generated via reflection information. - -## Native bag of options - -Options could be represented via an internal struct on which a hash can be performed. - -Input can be described as "deltas" to the current options. The final options is the combination of all the deltas - which would produce the final options structure for use. The hash of all of the options is the hash of the final structure. - -How are the deltas described? - -The in memory representation is not trivial in that if we want to add a struct to a list we would need a way to describe this. - -Whilst in the runtime the 'field' could be uniquely identified an offset, within a file format representation it would need to be by something that works across targets, and resistant to change in contents. That implies it should be a name. - -If we ensure that all types involved in options as JSON serializable via reflection, this does provide a way for code to traffic between and manipulate the native types. - -How do we add a structure to a list? - -```JSON -{ - "+listField", { "structField" : 10, "anotherField" : 20 } -} -``` - -The problem perhaps is how to implement this in native code? It looks like it is workable with the functionality already available in RttiUtil. - -## Slangs Component System - -Slang has a component system that can be used for combining options to produce a compilation. An argument can be made that it should be part of the hashing representation, as it is part of compilation. - -If combination is at the level of components, then as long as components are serializable, we can represent a compilation by a collection of components. Has several derived interfaces... - -* IEntryPoint -* ITypeConformance -* IModule - -Can be constructed into composites, through `createCompositeComponentType`, which describes aspects of the combination takes place. - -If the components were serializable (as say as JSON), we could describe a compilation as combination of components. If components are named, a concatenation of names could name a compilation. - -It doesn't appear as if there is a way to more finely control the application of component types. For example if there was a desire to change the optimization option, it would appear to remain part of the ICompileRequest (it's not part of a component). This implies this mechanism as it stands whilst allowing composition, doesn't provide the more nuanced composition. Additional component types could perhaps be added which would add such control. - -Perhaps having components is not necessary as part of the representation, as 'component' system is a mechanism for achieving a 'bag of options' and so we can get the same effect by using that mechanism without components. - -Discussion: Container -===================== - -## Manifest or association - -A typical container will contain kernels - in effect blobs. The blobs themselves, or the blob names are not going to be sufficient to express the amount of information that is necessary to meet the goals laid out at the start of this document. Some extra information may be user supplied. Some extra information might be user based to know how to classify different kernels. Therefore it is necessary to have some system to handle this metadata. - -As previously discussed the underlying container format is a file system. Some limited information could be infered from the filename. For example a .spv extension file is probably SPIR-V blob. For more rich meta data describing a kernel something more is needed. Two possible approaches could be to have a 'manifest' that described the contents of the container. Another approach would to have a file associated with the kernel that describes it's contents. - -Single Manifest Pros - -* Single file describes contents -* Probably faster to load and use -* Reduces the amount of extra files -* Everything describing how the contents is to be interpreted is all in one place - -Single Manifest Cons - -* Not possible to easily add and remove contents - requires editing of the manifest, or tooling - * Extra tooling specialized tooling was deemed undesirable in original problem description -* Manifest could easily get out of sync with the contents - -Associated Files Pros - -* Simple -* Can use normal file system tooling to manipulate -* The contents of the container is implied by the contents of the file system - * Easier to keep in sync - -Associated Files Cons - -* Requires traversal of the container 'file system' to find the contents -* Might mean a more 'loose' association between results - -Another possible way of doing the association is via a directory structure. The directory might contain the 'manifest' for that directory. - -Given that we want the format to represent a file system, and that we would want it to be easy and intuitive how to manipulate the representation, using a single manifest is probably ruled out. It remains to be seen which is preferable in practice, but it seems likely that using 'associated files' is probably the way to go. - -## How to represent data - -As previously discussed, unless there is a very compelling reason not to we want to use representations that are open standards and easy to use. We also need such representations to be resilient to changes. It is important that file formats can be human readable or easily changeable into something that is human readable. For these reasons, JSON seems to be a good option for our main 'meta data' representation. Additionally Slang already has a JSON system. - -If it was necessary to have meta data stored in a more compressed format we could consider also supporting [BSON](https://en.wikipedia.org/wiki/BSON). Conversion between BSON and JSON can be made quickly and simply. BSON is a well known and used standard. - -Discussion: Container Layout -============================ - -We probably want - -* Global configuration information - describe how names map to contents -* Configuration that is compiler specific - * The format could support configuration for different compilers -* Use 'associated' file style for additional information to a result - -``` -config -config/global.json -config/slang.json -config/dxc.json -source/ -source/some-source.slang -source/some-header.h -``` - -The `source` path holds all the unique source used during a compilation. This is the 'deduped' representation. Any include hierarchy is lost. Names are generated such that they remain the same as the original where possible, but are made unique if not. The 'dependency' file for a compilation specifies how the source as included maps to the source held in the source directory. The source held in the repository like this provides a way to repeat compilations from source, but isn't the same as the source hierarchy for compilation and is typically a subset. - -`config/global.json` holds configuration that applies to the whole of the container. In particular how names map to the container contents - say the use of directories, or naming concat order. -`config/slang.json` holds how names map to option configuration that is specific to Slang - -We may want to have some config that applies to all different compilers. - -We may want to use the 'name' mechanism for some options, but commonly changing items such as the translation unit source name, entry point name can be passed directly (and used as part of the name). - -Let's say we have a config that consists of `target`, `configuration`. And we use the source name and entry point directly. We could have a configuration that expressed this as a location - -```JSON -{ - "keyPath" : "$(target)/$(filename)/$(entry-point)-$(configuration)" -} -``` - -Lets say we compile `thing.slang`, with entry point 'computeMain' and options - -``` -target: vk -configuration: release -``` - -We end up with - -``` -vk/thing/computeMain-release.spv -vk/thing/computeMain-release.spv-info.json -vk/thing/computeMain-release.spv-diagnostics.json -vk/thing/computeMain-release.spv-layout.json -vk/thing/computeMain-release.spv-dependency.json -``` - -`-info.json` holds the detailed information about what is in spv to identify the artifact, but also for the 'system' in general. Including - -`-dependency.json` is a mapping of source 'names' as part of compilation to the file - -* Artifact type -* The combination of options (which might be *more* than in the path) -* Hashes/other extra information - -Other items associated with the main 'artifact' - typically stored as 'associated' in an artifact, are optional and could be deleted. - -Having an extension, on the associated types is perhaps not necessary. Doing so makes it more clear what items are from a usability point of view. If this data can be represented in multiple ways - say JSON and BSON it also makes clear which it is. - -Discussion: Interface -===================== - -There are perhaps two ends of the spectrum of how an interface might work. One one end would be the interface is a 'slang like' compiler interface, with the the most extreme version being it *is* the Slang compiler interface. Having this interface like this means - -* There is direct access to all options -* If your application already uses the Slang interface, it can just be switched out for the cache -* Has full knowledge of the compilation - making identification unambiguous, and trivially allowing fallback to actually doing a compilation -* Trivially supports more unusual aspects of the API such as the component system -* There is no (or very little) new API, the shader container API *is* the Slang API - -It also means - -* The API has a very large surface area -* It works at the detail of the API -* Does not provide an application level indirection to some more meaningful naming/identification -* It is tied to the Slang compiler - so can't be seen as an interface to 'shader container's more generally -* Naming will almost certainly need to include a hash -* The hash will be hard to produce independently (it will be hard to calculate just anyway) - -More significantly - -* Higher requirement for source - * Could store hashes of source seen - * If there is source injection does this even make sense? - * Could store modules as Slang IR -* For generated source it requires the source -* All source does in general need to be hashed - as paths do not indicate uniqueness -* How is this obfuscated? The amount of information needed is *all of the settings*. - -At the other end of the spectrum the interface could be akin to passing in a set of user configurable parameter "names", that identify the input 'options'. The most extreme form might look something like.... - -``` -class IShaderContainer -{ - Result getArtifact(const char*const* configNames, Count configNamesCount, const Options& options, IArtifact** outArtifact); - Result getOrCreateArtifact(const char*const* configNames, Count configNamesCount, const Options& options, IArtifact** outArtifact); -}; -``` - -Q: Perhaps we don't return an Artifact, because the IArtifact interface is not simple enough. Maybe it returns a blob and an ArtifactDesc? -Q: We could simplify the IArtifact interface by moving to IArtifactContainer. Perhaps we should do this just for this reason? -Q: Is there a way to access other information - diagnostics for example? With IArtifact that can be returned as associated data. We don't want to create by default probably. -Q: If we wanted to associate 'user data' with a result how do we do that? It could just be JSON stored in the `-info`? -Q: We could have a JSON like interface for arbitrary data? - -The combination of the 'configNames' produce the key/paths within the container. - -It would probably be desirable to be able to create 'configNames' through an API. This would have to be Slang specific, and not part of this interface. The config system could be passed into the construction of the container. Doing so might contain all the information to map names to how to compile something. - -This interface may be a little too abstract, and perhaps should have parameters for common types of controls. - -As previously touched on it may be useful to pass in configuration that is *not* part of the key name to override compilation behavior. - -This style means - -* Naming is trivial -* Hashing is often not necessary -* Issues such as 'generated source' are pushed to user space -* Is user configurable -* Main interface is very simple and small -* Can be used with other compilers - because the interface is not tied to Slang in any way -* Implies the container format itself can be used trivially -* Human/application centered -* Hashing of source/options is still possible, for a variety of purposes, but is not a *requirement* as it doesn't identify a compilation. - * Meaning a simpler/less stable hashing might be fine - -With this style is it implied that the identification of a unique combination is a user space problem. For example that the source is static in general, and if not generated source identification is a user space problem. It's perhaps important to note that mechanisms previously discussed - such as hashing the source can still be useful and used. The hashing of source could be used to identify in a development environment that a recompilation is required. Or an edit of source could be made, and a single command could update all contents that is applicable automatically. These are more advanced features, and are not necessary for a user space implementation, which typically do not require the capability. - -More problematically - -* Doesn't provide a runtime cache that 'just works' for example just using the slang API -* Needs to provide a way given a combination of config names to produce the appropriate settings - * If it is just a delivery mechanism this isn't a requirement -* Probably needs both an API and 'config' mechanisms to describe options -* The indirection may lose some control - -All things considered, based on the goals of the effort it seems to make more sense to have an interface that is in the named config style. Because - -* It allows trivial 3rd party implementation -* It works with other compilers - (important if it's to work as some kind of standard) -* Provides an easy to understand mapping from input to the contents of the cache -* Can use more advanced features (like source hashing) if desired - -How config options are described or combined may be somewhat complicated, but is not necessary to use the system, and allows different compilers to implement however is appropriate. - -Discussion : Deduping source -============================ - -When compiling shaders, typically much of the source is shared. Unfortunately it is not generally possible to just save 'used source', because some source can be generated on demand. One way this is already performed by users is to use a specialized include handler, that will inject the necessary code. - -It is not generally possible therefore to identify source by path, or unique identity (as used by the slang file system interface). - -It is also the case that compilations can be performed where the source is passed by contents, and the name is not set, or not unique. - -The `slang-repro` system already handles these cases, and outputs a map from the input path to the potentially 'uniquified' name within the repro. - -You could imagine a container holding a folder of source that is shared between all the kernels. In general it would additionally require a map of each kernel that would map names to uniqified files. - -In the `slang-repro` mechanism the source is actually stored in a 'flat' manner, with the actual looked up paths stored within a map for the compilation. It would be preferable if the source could be stored in a hierarchy similar to the file system it originates. This would be possible for source that are on a file system, but would in general lead to deeper and more complex hierarchies contained in container. - -Including source, provides a way to distribute a 'compilation' much like the `slang-repro` file. It may also be useful such that a shader could be recompiled on a target. This could be for many reasons - allowing support for future platforms, allowing recompilation to improve performance or allowing compilation to happen on client machines for rare scenarios on demand. - -We may want to have tooling such that directories of source can be specified and are added to the deduplicated source library. - -We may also want to have configuration information that describes how the contents maps to search paths. It might be useful to have only the differences for lookup stored for a compilation, and some or perhaps multiple configuration files that describe the common cases. - -Discussion : Artifact With Runtime Interface -============================================ - -It should be noted *by design* `IArtifactContainer`s children is *not* a mechanism that automatically updates some underlying representation, such as files on the file system. Once a IArtifactContainer has been expanded, it allows for manipulation of the children (for example adding and removing). The typical way to produce a zip from an artifact hierarchy would be to call a function that writes it out as such. This is not something that happens incrementally. - -For an in memory caching scenario this choice works well. We can update the artifact hierarchy as needed and all is good. - -In terms of just saving off the whole container - this is also fine as we can have a function given a hierarchy that saves off the contents into a ISlangMutableFileSystem, such that it's on the file system or compressed. - -If we want the representation to be *synced* to some backing store this presents some problems. It seems this most logically happens as part of the compilation interface implementation. The Artifact system doesn't need to know anything about such issues directly. - -Once a compilation is complete, an implementation could save the result in Artifact hierarchy and write out a representation to disk from that part of the hierarchy. For some file systems doing this on demand is probably not a great idea. For example the Zip file system does not free memory directly when a file is deleted. Perhaps as part of the interface there needs to be a way to 'flush' cached data to backing store. Lastly there could be a mechanism to write out the changes (or the new archive). - -Discussion : Other -================== - -It would be a useful feature to have tooling where it is possible to - -## Generating the container - -* Generate updated kernels automatically offline - * For example when a source file changed - * For example when a config file changed - * Just force rebuilding the whole container -* Specify the combinations that are wanted in some offline manner - * Perhaps compiling in parallel - * Perhaps noticing aspects such that work can be shared - -## Obfuscation - -* Most simply could be using a hash of a 'key'. -* Or perhaps if the desire is to obfuscate at the application level, a hash of the *names* as input could be used - -## Stripping containers - -At a minimum there needs to be mechanisms to be able to strip out information that is not needed for use on a target. - -There probably also additionally needs to be a way to specify items such that names, such as type names, source names, entry point names, compile options and so forth are not trivially contained in the format, as their existence could leak sensitive information about the specifics of a compilation. - -## Indexing - -No optimized indexed scheme is described as part of this proposal. - -Indexing is probably something that happens at the 'runtime interface' level. The index can be built up using the contents of the file system. - -No attempt at an index is made as part of the container, unless later we find scenarios where this is important. Not having an index means that the file system structure itself describes it's contents, and allows manipulation of the containers contents, without manipulation of an index or some other tooling. - -## Slang IR - -It may be useful for a representation to hold `slang-ir` of a compilation. This would allow some future proofing of the representation, because it would allow support for newer versions of Slang and downstream compilers without distributing source. - -Related Work -============ - -* Shader cache system as part of gfx (https://github.com/lucy96chen/slang/tree/shader-cache) -* Lumberyard [shader cache](https://docs.aws.amazon.com/lumberyard/latest/userguide/mat-shaders-custom-dev-cache-intro.html) -* Unreal [FShaderCache](https://docs.unrealengine.com/5.0/en-US/fshadercache-in-unreal-engine/) -* Unreal [Derived Data Cache - DDC](https://docs.unrealengine.com/4.26/en-US/ProductionPipelines/DerivedDataCache/) -* Microsoft [D3DSCache](https://github.com/microsoft/DirectX-Specs/blob/master/d3d/ShaderCache.md) - -Lumberyard uses the zip format for its '.pak' format. - -Microsoft D3DSCache provides a binary keyed key-value store. - -## Gfx - -Gfx has a runtime shader cache based on `PipelineKey`, `ComponentKey` and `ShaderCache`. ShaderCache is a key value store. - -A key for a pipeline is a combination of - -``` - PipelineStateBase* pipeline; - Slang::ShortList specializationArgs; -``` - -`ShaderComponentID` can be created on the ShaderCache, from - -``` - Slang::UnownedStringSlice typeName; - Slang::ShortList specializationArgs; - Slang::HashCode hash; -``` - -For reflected types, a type name is generated if specialized. - -The shader cache can be thought of as being parameterized by the pipeline and associated specialization args. It appears to currently only support specialization types. - -Gfx does not appear to support any serialization/file representation. - - - -Proposed Approach -================= - -Based on the goals described in the introduction, the proposed approach is - -* Use a collection of named options to describe a compilation - * Requires a mechanism to provide combining - * Have some additional symbols (such as + field name prefix as described elsewhere) to describe how options should be applied -* Meaning of names can be described within configuration and through an API - * API might be compiler specific -* Some names contribute to the key, whilst others do not - * Non inclusion in key allows customization for a specific result without a key change -* Some options within a configuration can use standard names, others will require being compiler specific -* Probably easiest to use a native representation for combining - * Using the collection of names approach makes hash stability and hashes in general less important -* Use JSON/BSON as the format for configuration files - * Possible to have some options defined that *aren't* part of the key name - * The actual combination can be stored along with products, such that the combination can be recreated, or an inconsistency detected -* Use JSON to native conversion to produce native types that can then be combined -* Can have some standard ways to generate names for standard scenarios - * Such as using input source name as part of key -* Use associated files (not a global manifest), to allow easy manipulation/tooling -* Source will in general be deduped, with a compilation describing where it's source originated - * This is similar to how repro files work -* Some names will be automatically available by default -* Ideally a 'non configured' (or default configured) cache can work for common usage scenarios. - -For the runtime cache scenario this all still works. If an application wants a runtime cache that is memory based that works transparently (ie just through the use of Slang API), this is of course possible and it's output can be made compatible with the format. It will be fragile to Slang API changes, and probably not usable outside of the Slang ecosystem. - -Alternatives Considered ------------------------ - -Discussed elsewhere. - -## Issues On Github - -* Support low-overhead runtime "shader cache" lookups [#595](https://github.com/shader-slang/slang/issues/595) -* Compilation id/hash [#2050](https://github.com/shader-slang/slang/issues/2050) -* Support a simple zip-based container format [#860](https://github.com/shader-slang/slang/issues/860) - diff --git a/docs/proposals/legacy/001-basic-interfaces.md b/docs/proposals/legacy/001-basic-interfaces.md new file mode 100644 index 000000000..4d04cbe04 --- /dev/null +++ b/docs/proposals/legacy/001-basic-interfaces.md @@ -0,0 +1,254 @@ +Basic Interfaces +================ + +The Slang standard library is in need of basic interfaces that allow generic code to be written that abstracts over built-in types. +This document sketches what the relevant interfaces and their operations might be. + +Status +------ + +In discussion. + +Background +---------- + +One of the first things that a user who comes from C++ might try to do with generics in Slang is write an operation that works across `float`s, `double`s, and `half`s: + +``` +T horizontalSum( vector v ) +{ + return v.x + v.y + v.z + v.w; +} +``` + +A function like `horizontalSum` does not compile because without a constraint on the type parameter `T`, the compiler has no reason to assume that `T` supports the `+` operator. +A new user is often stymied at this point, because no appropriate `interface` seems to exist, and there does not appear to be a way to *define* an appropriate interface. + +As a user gets more experienced with Slang, they may learn how to use `extension`s to define something nearly suitable: + +``` +interface IMyAddable { This add(This rhs); } + +extension float : IMyAddable { float add(float rhs) { return this + rhs; } } +// ... + +T horizontalSum( vector v ) +{ + return v.x.add(v.y).add(v.z).add(v.w); +} +``` + +While that approach works (or should work), it requires a user to know how to use `extension`s and the `This` type, which are complicated even for experienced users. The resulting code is also less readable because it uses `.add(...)` instead of the ordinary `+` operator. + +Many more users end up finding out about the `__BuiltinFloatingPointType` interface, and write something like: + +``` +T horizontalSum( vector v ) +{ + return v.x + v.y + v.z + v.w; +} +``` + +This alternative is much more palatable to users, but it results in them using a double-underscored interface (which we consider to mean "implementation details that are subject to change"). Users often get tripped up when they find out that certain operations that make sense to be available through `__BuiltinFloatingPointType` are not available (because those operations were not needed in the definition of the stdlib, which is what the `__` interfaces were created to support). + +Related Work +------------ + +There are several languages that have constructs similar to our `interface`s, and which provide built-in interfaces for simple math operations that are suitable for use with the built-in types provided by the language. + +Existing solutions can be broadly categorized based on whether their built-in interfaces are related to semantic/mathematical structures, or are purely about specific classes of operators. + +Haskell and Swift are both examples of languages where the built-in interfaces are intended to be semantic. Haskell provides type classes such as `Additive`, `Ring`, `Algebraic`, `RealTranscendental`, etc. +Swift is similar (although it provides a less complete hierarchy of algebraic structures than Haskell), but also includes more detail amount machine number representations, so that it has `BinaryFloatingPoint`, `FixedWidthInteger`, etc. + +Rust is in the other camp, where it has a built-in interface to correspond to each of its overloadable operators. The `Add` and `Sub` traits allow the built-in `+` and `-` operators to be overloaded for a user-defined type, but impose no implicit or explicit semantic expectations on those operations. + +It may help to describe a concrete example of how the difference between the two camps affects design. The Rust `Add` trait is implemented by the left-hand-side type, and does not constrain the right-hand side or result type of an addition. A Rust programmer may implement `Add` for a type `X` so that `x + ...` expects a right-hand-side operand of some other type `Y` and produces a result of yet *another* type `Z`. Knowing that `X` supports the `Add` trait does *not* mean that it is possible to take the sum of a list of `X`s, because there is no guarantee that `x0 + x1` is valid, or that `X` has a logical "zero" value that could be used as the sum of an empty list. + +In contrast, in Swift a type `X` that conforms to `AdditiveArithmetic` must provide a `+` operation that takes two `X` values and yields an `X`. It also requires that `X` provide a `static` property `zero` of type `X`, to represent its zero value. As a result, it is possible to write a generic function in switch that can compute the sum of a list of `T` values, provided `T` conforms to `AdditiveArithmetic`. + +Proposed Approach +----------------- + +Slang supports operators as ordinary overloadable functions, so the rationale behind the Rust operator traits does not seem to apply. We propose to implement a modest hierarchy of numeric interfaces in the style of Haskell/Swift. + +### Changes to Operator Lookup + +Currently, when Slang encounters an operator invocation like `a + b`, it treats this as more or less equivalent to a function call `+( a, b )`. The compiler looks up `+` in the current lexical environment, and then applies overload resolution to the result of lookup. + +We propose that the rules in that case should be changed so that lookup *also* perform lookup of the operator (`+` in this case) in the context of the static types of `a` and `b`. That change would in theory allow "operator overloads" to be defined as `static` functions within a type they apply to (whether on the left-hand or right-hand side). As a consequence, such a change would also mean that `interface`s could conveniently include operator overloads as requirements. + +### IAdditive + +The `IAdditive` interface is for types where addition, subtraction, and zero have meaning. + +``` +interface IAdditive +{ + // The zero value for this type + static property zero : This { get; } + + // Add two values of this type + static func +(left: This, right: This) -> This; + + // Subtract two values of this type + static func -(left: This, right: This) -> This; +} +``` + +### INumeric + +The `INumeric` interface is for types that are more properly number-like. +Note that this interface does not define division, because the division operations on integers and floating-point numbers are sufficiently different in semantics. + +``` +interface INumeric : IAdditive +{ + // Initialize from an integer + __init< T : IInteger >( T value ); + + // Multiply two values of this type + static func *(left: This, right: This) -> This; +} +``` + +### ISignedNumeric + +Only signed numbers logically support negation (although we all know it also gets applied to unsigned numbers, where it has meaningful and use semantics). + +``` +interface ISignedNumeric : INumeric +{ + // Negate a value of this type + static prefix func -(value: This) -> This; +} +``` + + +### IInteger + +The `IInteger` interface codifies the basic things that a generic wants to be able to access for any integer type. + +``` +interface IInteger : INumeric +{ + // Smallest representable value + static property minValue : This { get; } + + // Largest representable value + static property maxValue : This { get; } + + // Initialize from a floating-point value + // (what rounding mode? round-to-nearest-even?) + __init< T : IFloatingPoint >( T value ); + + + // Integer quotient + static func /(left: This, right: This) -> This; + + // Integer remainder (or is it modulus? or is it undefined which?) + static func %(left: This, right: This) -> This; +} +``` +### IUnsignedInteger + +``` +interface IUnsignedInteger : IInteger +{ + +} +``` + +### ISignedInteger + +The main interesting thing we'd want from a signed integer type is to be able to convert it to the same-size unsigned integer type. + +``` +interface ISignedInteger : IInteger, ISignedNumeric +{ + // Equivalent unsigned type (can always hold magnitude) + associatedtype Unsigned : IUnsignedInteger; + + // Get the magnitude of this value (may not be representable + // as `This` type, if it is `minValue`) + property magnitude : Unsigned { get; }; +} +``` + +### IFloatingPoint + +The `IFloatingPoint` interface provides the minimum of what users expect a floating-point type to support. +It includes the ability to check for special values (not-a-number, infinities), as well as the value of various standard constants. + +``` +interface IFloatingPoint : INumeric, ISignedNumeric +{ + property isFinite : bool { get; } + property isInfinite : bool { get; } + property isNaN : bool { get; } + property isNormal : bool { get; } + property isDenormal : bool { get; } + + // TODO: breaking into magnitude/exponent + + static property infinity : This { get; } + static property nan : This { get; } + static property pi : This { get; } + + // TODO: min/max finite values, smallest non-zero value, etc. + + // Initialize from another floating-point value. + __init< T : IFloatingPoint >( T value ); + + // Floating-point division + static func /(left: This, right: This) -> This; +} +``` + +### ISpecialFunctions + +The `ISpecialFunctions` interface is for floating-point types that also have full support for the standard suite of special functions provided by something like ``. +It is pulled out as a distinct interface from `IFloatingPoint` because many platforms support floating-point types like `double` without also having full support for special functions on those types. + +``` +interface ISpecialFunctions : IFloatingPoint +{ + static This cos(This value); + static This sin(This value); + // TODO: fill this out +} +``` + +Questions +--------- + +### Should these all be `IBuiltin*`? Should we have separate interfaces for built-in and user types? + +The main reason for the current `__Builtin` interfaces is that it allows us to define built-in functions that are generic over those interfaces, but which map to a single instruction in the Slang IR. The relevant operations are not currently defined as + +### What should the naming convention be for `interface`s in Slang? + +These would be the first `interface`s officially exposed by the standard library. +While most of our existing code written in Slang uses an `I` prefix as the naming convention for `interface`s (e.g., `IThing`), we have never really discussed that choice in detail. +Whatever we decide to expose for this stuff is likely to become the de facto convention for Slang code. + +The `I` prefix is precedented in COM and C#/.net/CLR, which are likely to be familiar to many devleopers using Slang. +Because of COM, it is also the convention used in the C++ API headers for Slang and GFX. + +The Rust/Swift languages do not distinguish between traits/protocols and other types. +This choice is intentional, and it might be good to understand the motivation behind it. +At least one potential benefit to not distinguishing such types is that beginning programmers can write code that is "more generic" than they might otherwise write. + +Alternatives Considered +----------------------- + +One important alternative is to follow the precedent of Rust and avoid basing these interfaces on semantic structures. +That choice is important in Rust in part because there is no way for a type to support an operator other than by implementing the built-in operator traits. +If the operator traits had prescriptive semantics, they might cause problems for types that want to support the operators but cannot fit within the semantic constraints. +In contrast, Slang allows operator overloads to be defined independent of interfaces (they are orthogonal features), so there is no risk of developers being "locked in" by our attempts to provide richer interfaces. + +Conversely, one could worry that our interfaces do not provide *enough* semantics. We may find that users need additional interfaces that sit "in between" these ones, or that carve up the same operations into smaller units. +This proposal contends that we need to have *something* in this space, and that it doesn't make sense to try to get these interfaces 100% perfect until we've had some lived experience with them. +Fortunately, the Slang language is not yet at a point of trying to guarantee perfect source stability of these interfaces, nor anything like strong binary compatibility guarantees. +If we make mistakes here, we have time to fix them. + diff --git a/docs/proposals/legacy/002-api-headers.md b/docs/proposals/legacy/002-api-headers.md new file mode 100644 index 000000000..66b649228 --- /dev/null +++ b/docs/proposals/legacy/002-api-headers.md @@ -0,0 +1,952 @@ + +Revise Slang/GFX API Headers +============================ + +The public C/C++ API headers for Slang (and GFX) are in need of cleanup and refactoring for us to reach a "1.0" API. +This document attempts to document the guidelines that we will follow in such a refactor. + +Status +------ + +In discussion. + +Background +---------- + +The Slang API header (`slang.h`) has evolved over many years, going back as far as the "Spire" research project, which predates Slang (Spire is the reason for the `sp` prefix on functions in the C API). + +At some point, we made a conscious decision to move toward a COM-based API, both because it would simplify our story around binary compatibility and because it would allow us to provide more convenient API models in cases where subtyping/inheritance is fundamental to the domain. +Unfortunately, the net result has been that we have two different APIs cluttering up the same header file (the old C one, and the newer C++/COM one), and the one that is presented *first* is actually the one we would rather users didn't reach for. +The two APIs are sometimes out of sync, with one providing services the other doesn't. + +While the GFX project started later and was thus able to start using COM interfaces and a C++ API from the start, it still faces some challenges around API evolution and binary compatibility. +As support for GPU features (whether pre-existing or new) gets added, we find that the various interfaces want to grow and the various `Desc` structures want to add new fields. +Without care, neither of those is a binary-compatible change for user code. + +A concern across both Slang and GFX is that we have tended to design our APIs around the *most complicated* use cases we intend to support, at the expense of the *simplest* cases. +We know that we cannot remove support for difficult cases, but it would be good to support concise code for simple use cases, and to support a "progressive disclosure" style that allows users to gradually adopt more involved API constructs as they become necessary. + +Related Work +------------ + +There are obviously far too many C/C++ APIs and approachs to design for C/C++ APIs for us to review them all. +We will simply note a few key examples that can be relevant for comparison. + +The gold standard for C/C++ APIs is ultimately plain C. Plain C is easy for most systems programmers to understand and benefits from having a well-defined ABI on almost every interesting platform. FFI systems for other languages tend to work with plain C APIs. Clarity around ABI makes it easy to know what changes/additions to a plain C API will and will not break binary compatibility. The Cg compiler API and the Vulkan GPU API are good examples of C-based APIs in the same domains as Slang and GFX, respectively. These APIs reveal some of the challenges of using plain C for large and complicated APIs: + +* The lack of subtype polymorphism is a problem when a domain fundamentally has subtyping. The Cg reflection API uses a single `CGtype` type to represent all types, so that the an operation like `cgGetMatrixSize` is applicable to any type, not just matrix types. The API cannot guide a programmer toward correct usage, and must define what happens in all possible incorrect cases. + +* C has no built-in model for error signalling or handling. Error codes and out-of-band values (`NULL`, `-1`) are the norm, and there are a multitude of API-specific conventions for how they are applied. + +* C has no built-in model for memory and lifetime management. Most APIs either expose create/delete pairs or some kind of per-type reference-counting retain/release operations. In either case, the application developer is left to ensure that the operations are correctly invoked, often by writing their own C++ wrapper around the raw C API. + +Some developers opt for a "Modern C++" philosophy where the public API of a system makes direct use of standard C++ library types where possible. +For example, strings are passed as `std::string`s, cases that need polymoprhism expose `class` hierarchies, types that benefit from reference-counted lifetime management, explicitly uses `std::shared_ptr<...>`, and errors are signaled by `throw`ing exceptions. +The Falcor API ascribes to aspects of this approach. +The biggest challenges with Modern C++ APIs are: + +* Source compatibility can usually be achieved, but binary compatibility is hard to achieve even *within* a version of a system, must less across versions. The central problem is that C++ ABIs are often compiler-specific (rather than standard on a platform), and even for a single compiler like gcc or Visual Studio, the binary interface to the C++ standard library can and does break between versions. + +* While C++ exceptions are a built-in error-handling scheme, they are almost universally disliked among the kinds of developers who use APIs like Slang. Enabling exceptions in most compilers adds overhead, and actually using exceptions for their intended purpose (catching and handling errors) tends to be onerous. + +* Reference-counted lifetime management in Modern C++ relies on standard library types like `std::shared_ptr` - a type that is both inefficient and inconvenient. Most developers in our target demographic end up using "intrusive" reference counts (when they use reference-counting at all) because they are more efficient and convenient. + +COM is first and foremost an idiomatic way of using C++ to define APIs that are reasonably convenient while also dealing with the recurring problems of typical C and C++ approaches. +COM defines rules for error handling, memory management, and interface versioning that all compatible APIs can use. +While code using COM-based APIs is often verbose, it is largely consistent across all such APIs. + +A key place where COM does *not* provide a complete answer is around fine-grained "extensibility" of APIs, of the kind that commonly occurs with GPU APIs like OpenGL, D3D, and Vulkan. +Across such APIs, we see a wide variety of strategies to dealing with extensibility: + +* OpenGL uses an approach where objects are typically opaque but mutable, and a large number of fine-grained operations are used to massage an object into the correct state for use. Often the fine-grained state-setting operations are all able to use a single API entry point for key-value parameter setting (e.g., `glTexParameteri`), and a new feature can be exposed simply by defining constants for new keys and/or values. When new operations are required, they need to be queried using string-based lookup of functions. + +* D3D11 uses COM interfaces and "desc" structures (called "descriptors" at the time). For example, a mutable `D3D11_RASTERIZER_DESC` structure is filled in and used to create an *immutable* `ID3D11RasterizerState`. If extended features are required, a new interface like `ID3D11RasterizerState1` and/or a new descriptor type like `D3D11_RASTERIZER_DESC1` is defined. In all cases, the "desc" structure holds the union of all state that a given type supports. + +* Vulkan uses "desc" structures (usually called "info" or "create info" structures), which contain a baseline set of state/fields, along with a linked list of dynamically-typed/-tagged extension structures. New functionality that only requires changes to "desc" structures can be added by defining a new tag and extension structure. New operations are added in a manner similar to OpenGL. + +* D3D12 also uses COM interfaces and "desc" structures (although now officialy called "descriptions" to not overload the use of "descriptor" in descriptor tables), much like D3D11, and sometimes uses the same approach to extensibility (e.g., there are currently `ID3D12Device`, `ID3D12Device`, ... `ID2D12Device9`). In addition, D3D12 has also added two variations on Vulkan-like models for creating pipeline state (`ID3D12Device2::CreatePipelineState` and `ID3D12Device5::CreateStateObject`), using a notion of more fine-grained "subojects" that are dynamically-typed/-tagged and each have their own "desc". + +It is important to note that even with the nominal flexibility that COM provides around versioning, D3D12 has opted for a more fine-grained approach when dealing with something as complicated as GPU pipeline state. + +Proposed Approach +----------------- + +The long/short of the proposal is: + +* The primary API interface to Slang (and GFX) should be COM-based and use C++. Convenient C++ features like `enum class` are usable when they do not constrain binary compatibility. + +* Extensibility and versioning (where appropriate) should make use of "desc"-style tagged structures. Each of Slang and GFX will define its own `enum class` for the space of tags, rather than us trying to coordinate across the APIs. + +* We will focus on providing "shortcut" operations in the public API that allow developers to optimize for common cases and reduce the amount of boilerplate. + +* We will expose a C API that wraps the COM using `inline` functions. We will attempt to make the C API idiomatic when/as possible. + +* We can eventually/graduatelly provide a set of C++ wrapper/utility types that can further streamline the experience of using Slang, by hiding some of the details of COM reference counting, and "desc" structs". The utility code could also translate COM-style result codes into C++ exceptions, if we find that this is desired by some users. + +Detailed Explanation +-------------------- + +At the end of this document there is a lengthy code block that sketches a possible outline for what the `slang.h` header *could* look like. + +Questions +--------- + +### Will we generate all or some of the API header? If so, what will be the "ground truth" verison? + +Note that Vulkan and SPIR-V benefit from having ground-truth computer-readable definitions, allowing both header files and tooling code to be generated. + +### Can we actually make a reasonably idiomatic C API that wraps a COM one, or should we admit defeat and have everything look like `slang__(...)`? + +Alternatives Considered +----------------------- + +We haven't seriously considered many alternatives in detail, other than the possibility of a plain C API (which we have tried, but not been able to make work). + +Appendix: A Header of Notes +--------------------------- + +The following code represents an sketch of a header that tries to match this proposal (and includes a lot of its own discussion/comentary). + +``` +/* Slang API Header (Proposed) + +This file is an attempt to sketch how the API headers for both +Slang and gfx could be organized in order to provide a nice +experience for developers who belong to different camps in terms +of what they want to see. + +Goals: + +* Support both C and C++ access to the API, with matching types, etc. + +* Able to use COM interfaces including use of inheritance/polymorphism + +* When compiling as C++, it should be possible to mix-and-match both C and C++ APIs + +Constraints: + + +Questions: + +* Do we actually need to restrict to block comments for pedantic compatibility + with old C versions? Are line comments close enough to universally-supported? + +*/ + +#ifdef __cplusplus + +/* Because of our goals above, we will actually end up with what amounts to +two copies of the C API, depend on whether or not we are compiling as C++. + +We start with the C++ COM-lite API, since that is the baseline. Note that +in this proposal, everything is being defined in the `slang` namespace, +rather than first declaring many things as C types and then mapping that +over to C++. +*/ + +namespace slang +{ + /* Note that in this proposal, everything is being declared in the `slang` + namespace first, rather than the old model of declaring various things + in C and then importing them into the namespace. + */ + +/* Basic Types */ + + typedef int32_t Int32; + /* typedefs for basic types, as needed ... */ + + +/* Enumerations and Constants */ + + /* Non-flag enumerations will use `enum class`. If we need to support clients + using older C++ versions/compilers, we can discuss macro-based ways to try + to work around this. + + Except in cases where there is an *extremely* compelling reason to do something + different, all enumerations use the `int` tag type that is the default for + `enum class`. + */ + + + /** Severity of a diagnostic generated by the compiler. + ... + */ + enum class Severity + { + Note, /**< An informative message. */ + /* ... */ + }; + + /* TODO: We need a clearly-defined policy for how to handle "flags" enumerations. + + My strong opinion is that we should generally avoid flags in public API just + because of the tendency to run out of bits sooner or later, but I also understand + the appeal... + */ + + struct SlangTargetFlag + { + enum : UInt32 + { + DumpIR = 1 << 0, + /* ... */ + }; + }; + typedef UInt32 SlangTargetFlags; + + /* I'm note sure about whether the `Result` type ought to be declared with `enum class`. + It would be nice to have the extra level of type safety, but it would also make our + `Result` incompatible with macros and template types that are intended to work with + `HRESULT`s. + */ + enum Result : Int32 + { + /* I *do* think we should go ahead and define all the cases of `Result`s + that we expect our API to traffic in right here in the `enum`, so that users + can easily inspect result codes in the debugger. + */ + + OK = 0, + /* ... */ + }; + +/* Forward Declarations */ + + /* Simple Types */ + struct UUID; + /* ... */ + + /* "Desc" Types */ + struct SessionDesc; + /* ... */ + + /* Interface Types */ + struct ISession; + + +/* Structure Types */ + + /* Theres's not much to say for the easy case... */ + + struct UUID + { + uint32_t data1; + uint16_t data2; + uint16_t data3; + uint8_t data4[8]; + }; + /* ... */ + + /* The more interesting bit is "descriptor" sorts of structures, which + we've done a lot of back-and-forth on how best to support. + + I'm going to write up something here while also acknowledging that picking + a good policy for how to handle this stuff is an orthogonal design choice. + */ + + enum class DescType : UInt32 + { + None, + SessionDesc, + /* ... */ + }; + enum class DescTag : UInt64; + + #define SLANG_MAKE_DESC_TAG(TYPE) DescTag(UInt64(slang::DescType::TYPE) << 32 | sizeof(slang::##TYPE)) + + struct SessionDesc + { + DescTag tag = SLANG_MAKE_DESC_TAG(SessionDesc); + + TargetDesc const* targets = nullptr; + Int targetCount = 0; + /* ... */ + + /* Note: There is some subtlety here if we use default member + initializers here, but also want to expose these types directly + via the C API (in cases where somebody is using the C API but + a C++ compiler). + + The tag approach here is intended to support something akin to + the Vulkan style, when using the C API: + + SlangSessionDesc sessionDesc = { SLANG_DESC_TAG_SESSION_DESC }; + ... + + That code will not compile under C++11, because of the default + members initializers in `slang::SessionDesc`, but it *will* compile + under C++14 and later. + + If we want to deal with C++11 compatiblity in that case, we can, but + it would slightly clutter up the way we declare these things. Realistically, + we'd just split the two types: + + struct _SessionDesc { ... data but no initialization ... }; + struct SessionDec : _SessionDesc { ... put a default constructor here ... }; + + I'm not a fan of that option if we can avoid it. + */ + }; + + /* ... */ + + /* *After* all the "desc" types are defined, we can actually define the enum + for their tags (just to make life easier for users looking at things in their + debugger. + */ + enum class DescTag : UInt64 + { + None = 0, + SessionDesc = SLANG_MAKE_DESC_TAG(SessionDesc), + /* ... */ + }; + + /* Versioning: If we are in a situation where we'd like to change a type that + has already been tagged, we should first consider just creating an additional + "extension" desc type, to be used together with the original. By adding + suitable convenience APIs, we can make this easy to work with. + + If we really do decide that we want a new version of a specific desc, we should + start by doing the thing D3D does, and make a new numbered type: + + struct SessionDesc { ... the original ... }; + struct SessionDesc1 { ... the new one ... }; + + When possible, the new type should use matching field names/types and ordering. + Even if we are just adding fields, we should not try using inheritance (just + because the C++ spec doesn't guarantee enough about how inheritance is implemented). + + The new structure types should get its own desc type/tag, distinct from the original. + + If we decide that we want clients to compile against the latest version of these + types by default, we can shuffle around the names, but we need to be careful to + *also* shuffle the `DescType` cases (so that the binary values stay the same): + + struct SessionDesc0 { ... the original ... }; + struct SessionDesc1 { ... the new one ... }; + typedef SessionDesc SesssionDesc1; + + At the point where we introduce a second version, it is probably the right time + to enable developers to lock in to any version they choose. In the code above + the user can always just use `SessionDesc0` or `SessionDesc1` explicitly, or they + can just stick with `SessionDesc` in the case where they always want the latest + at the point they compile. + + (If we wanted to get really "future-proof" we'd define every struct with the `0` + prefix right out of the gate, and always have the `typedef` in place. I'm not conviced + that would ever pay off.) + + I expect most of this to be a non-issue if we are zealous about using fine-grained + rather than catch-all descriptors at this level of the API. + + (There's more I could talk about here, but this isn't supposed to be the topic at hand) + */ + + +/* Interfaces */ + + /* There's an open question of how to name the `IUnknown` equivalent + once things are all namespaced. We could use `slang::IUnknown`, but I fear + that could lead to complications or confusion for codebases that also use + MS-provided COM-ish APIs. + */ + + struct ISession : public ISlangUnknown + { + SLANG_COM_INTERFACE(...); + + /* In order to maximize our ability to evolve the API while maintaining + binary compatibility, I'm going to recommend the somewhat bold step + of defaulting to making interface entry points that are "implementation + details" rather than intended for direct use in most cases. + */ + + virtual SLANG_NO_THROW Result SLANG_MCALL _createCompileRequest( + void const* const* descs, + Int descCount, + UUID const& uuid, + void** outObject) = 0; + + /* Instead, most users will direclty call the operations only through + wrappers that provide conveniently type-safe behavior: + */ + inline Result createCompileRequest( + CompileRequestDesc const& desc, + ICompileRequest** outCompileRequest) + { + return _createCompileRequest( + &desc, 1, SLANG_UUID_OF(ICompileRequest), + (void**)outCompileRequest); + } + + /* An important property of this design is that we can easily define + convenience overloads that take direct parameters for common cases: + */ + inline Result createCompileRequestForPath( + const char* path, + ICompileRequest** outCompileRequest) + { + ...; + } + + /* Versioning: Note that we can easily define convenience overloads + for multiple versions of descriptor types (`CompileRequest` and + `CompileRequest`), or for *combinations* of descriptor types: + */ + inline Result createCompileRequest( + CompileRequestDesc const& requestDesc, + ExtraFeatureDesc const*& extraFeatureDesc, + ICompileRequest** outCompileRequest) + { + void* descs[] = { &requestDec, &extraFeatureDesc }; + return _createCompileRequest + descs, 2, SLANG_UUID_OF(ICompileRequest), + (void**)outCompileRequest); + } + + /* As a final detail, we should consider whether to support overloads + that work with either our `slang::ComPtr` *or* an application's own + smart pointer type. + + The user could override the smart pointer type by defining macros. + The defaults would be: + + #define SLANG_SMART_PTR(TYPE) slang::ComPtr + #define SLANG_SMART_PTR_WRITE_REF(PTR) ((PTR).writeRef()) + */ +#ifndef SLANG_DISABLE_SMART_POINTER_OVERLOADS + inline Result createCompileRequest( + CompileRequestDesc const& desc, + SLANG_SMART_PTR(ICompileRequest)* outCompileRequest) + { + return _createCompileRequest( + &desc, 1, SLANG_UUID_OF(ICompileRequest), + SLANG_SMART_PTR_WRITE_REF(*outCompileRequest)); + ) +#endif + + /* If we ever have cases where we want to support utility/wrapper operations + of higher complexity than what we feel comfortbale making `inline` (that is, + stuff that might be best off in a `slang-utils` static library) we could conceivably + handle those via judicious use of `extern`: + */ + inline Result createCompileRequestFromJSON( + char const* jsonBlob, //< a serialized form of the compilation state + ICompileRequest** outCompileRequest, + { + extern Result slang_ISession_createCompileRequestFromJSON( + char const*, ICompileRequest**); + return slang_ISession_createCompileRequestFromJSON(jsonBlob, outCompileRequest); + } + /* I doubt we'd ever really need that kind of approach, and would always decide + that functionality either belongs in core Slang (perhaps as a new derived interface) + or can go as global (non-member) functions in a utility library. + */ + + /* ... */ + } + + /* Note: I'm assuming here that we continue our implicit contract in terms + of versioning of COM interfaces: + + * Every interface is thought of as having a contract about who can *provide* + and who can *consume* it. For many (like `ISession`) only the Slang implementation + is supposed to provide it and only users consume it. Some callback interfaces + go the other way, and a few (like `slang::IBlob`) need to go both directions. + + * For interfaces that Slang provides and the user consumes, we can append new + `virtual` methods onto the end. This realistically needs a check somewhere, such + that we fail creation of the `IGlobalSession` if the user compiled against a + header that exposes the new method but is linking a DLL that doesn't. This is + what the `SLANG_API_VERSION` in the original header is supposed to be for: we + should increment it when we expand the API contract, and the global-session + creation should fail if the application is asking for too new of a version. + + * For interfaces that Slang consumes, we cannot realistically add/remove anything. + Theoretically, we could delete some of the `virtual` methods if we no longer + expect to call them, but that could still break client code that uses `override` + on their definitions. + + * We need to try very hard not to change the interface types of parameters to + non-wrapper COM methods, even if the result should be binary-compatible. There + are cases where it might be reasonable and "type-safe," but each and every one + probably needs clear auditing. + + Ideally the rules we use for Slang-provided interfaces can help us avoid the + proliferation of `IThing`, `IThing2`, `IThing3`, etc. We need to be careful about + that in the long run, though, because we may find that it causes problems in the wild + if software needs to interact with Slang in a context where the developer cannot + control the version of the Slang DLL they will be using at runtime. + + In theory, we could solve that problem by letting a user pass `SLANG_API_VERSION` + *in* to the header via a `#define`, and have us skip over any declarations introduced + after the given version. + */ +} + +/* Back outside the namespace (but still in the case where we know C++ is +supported), we can define the C-compatible API by using the C++ API +as its underlying representation. +*/ + +extern "C" +{ + +/* Basic Types */ + + /* The C-API types can just be `typedef`s of the C++ ones */ + + typedef slang::Int32 SlangInt32; + /* ... */ + +/* Enumerations and Constants */ + + /* For the case where we *know* a C++ compiler is being used, we + can actually use the C++ `enum class` declarations to provide + the enumerated types of the C API. + */ + typedef slang::Severity SlangSeverity; + + /* We can use macros to define the C-API enum cases, while still + preserving the type safety of the `enum class` approach. + + Note: We could also use `static const`s here, but that seems like + overkill. + */ + #define SLANG_SEVERITY_NONE (slang::Severity::None) + /* ... */ + +/* Structure Types */ + + /* For the case of providing the C API for a C++ compiler, we can + directly use the C++ structure types in all cases. */ + + typedef slang::UUID SlangUUID; + typedef slang::SessionDesc SlangSessionDesc; + /* ... */ + +/* Interfaces */ + + /* Because we are compiling as C++, defining the types for the interfaces + is easy, and we can easily pass objects between modules/files that are + using the two version of the API without any casting: + */ + typedef slang::ISession SlangSession; + /* ... */ + + /* In order to have a plain-C API, the user of course needs a way to + dispatch into those interfaces. + + Note: There is a big question here of whether the API header should be + trying to define the C API functions `inline` here or not. + + The argument for using `inline` is that it doesn't add any additional + requirements for somebody using the C API from within C/C++, compared to + the C++ API. + + The argument *against* is that for things like binding to other languages, + the user would probably prefer that these operations have linkage. + + Realistically, the right thing is for the header to include both declarations + *and* definitions, but to allow the application to conditionalize the inclusion + of the definitions *and* enable/disable the use of `inline` for declarations/definitions. + A user could use that control to compile their own linkable stub with C-compatible + functions. + */ + + /* We need to provide the fully-general version of the function, for clients + that might need it, but we probably don't want that to be the first one users + reach for. + */ + inline SlangResult _SlangSession_createCompileRequest( + SlangSession* session, + void const* const* descs, + SlangInt descCount, + SlangUUID const* uuid, + void** outObject) + { + return session->_createCompileRequest(descs, descCount, uuid, outObject); + } + + /* The catch here is that the C++ API used overloading as a way + to provide convenient wrappers around the fully-general core operations, + and also to provide versioning support. + + We could define the same set of overloads here, with the same names, for + use by clients who don't actually care about C compatiblity but just like + a C-style API. That is probably worth doing. + + Otherwise, we realistically need to start defining some de-facto naming + scheme and/or versioning for stuff in the C API. At least one wrapper + should be "blessed" as the default one. + */ + inline SlangResult SlangSession_createCompileRequest( + SlangCompileRequestDesc const* desc, + SlangCompileRequest** outCompileRequest) + { + return session->_createCompileRequest(*desc, outCompileRequest); + } + + /* Note that we need/want to provide wrappers for *all* the operations + on each interface, even the ones they inherit. E.g.:*/ + inline uint32_t SlangSession_addRef( + SlangSession* session) + { ... } + /* The reason for this is so that a pure-C user doesn't *have* to rely on + implicit conversion of these types to their bases (which in this path is + made possible via C++ features, but wouldn't be available in a true pure-C + world). + */ + + /* If/when we start to deal with versioning of either the "desc" type or + the interface involved in such an operation, we will need to do the numeric-suffix + thing or similar stuff to distinguish the old and new functions. + + We can probably do some work to always make the latest version (or at least + the one we want users to be using) have the short/clean name. Binary compatibility + shouldn't actually break so long as the signature of the new function can technically + handle calls of the old form (since the COM-level bottleneck function won't care about + the static types of descs - just their tags). + */ + + /* Finally, the C API level is where we should define the core factory entry + point for creating and initializing the Slang global session (just like + in the current header). Here we jsut generalize it for creaitng "any" global + object, based on a UUID and a bunch of descs. + */ + SLANG_API SlangResult slang_createObject( + void const* const* descs, + Int descCount, + UUID const* uuid, + void** outObject); + + /* The actual global session creation is then a wrapper like everything else. + */ + inline SlangResult SlangGlobalSession_create( + SlangGlobalSessionDesc const* desc, + SlangGlobalSession** outGlobalSession) + { + return slang_createObject( + &dec, 1, SLANG_UUID_OF(slang::IGlobalSession), (void**)outGlobalSession); + } +} + +#else + +/* All of the above declarations (even the C-level ones) only work if we are +compiling as C++. Thus we need a distinct strategy to define everything in the +case where we are compiling as pure C. + +The basic strategy isn't that hard: we just do things the raw C way. +There will be a lot of repetition involved, but this proposal assumes we are +generating as much of the API as possible anyway. +*/ + +/* Basic Types */ + + /* We just define the basic types direclty, without the indirection + through the declarations in the `slang::` namespace. + */ + + typedef int32_t SlangInt32; + /* ... */ + +/* Enumerations and Constants */ + + /* Every enum in this case is a `typedef` plus an actual `enum`: + */ + + typedef int SlangSeverity; + enum + { + SLANG_SEVERITY_NONE = 0, + /* ... */ + }; + + /* ... */ + +/* Structure Types */ + + /* The simple case stays simple, just with the gross bit of + duplicating a *lot* of what we already had in the C++ API. + + (There's a big design question here of whether we can/should try + to remove as much duplication as possible in order to reduce + boilerplate, even if it comes at the cost of clarity because of + heavier reliance on macros, etc.) + */ + + struct SlangUUID + { + uint32_t data1; + uint16_t data2; + uint16_t data3; + uint8_t data4[8]; + }; + /* ... */ + + /* The desc-related stuff is really just a translation of the + same basic ideas to plain C: */ + + typedef SlangUInt32 SlangDescType; + enum + { + SLANG_DESC_TYPE_NONE = 0, + SLANG_DESC_TYPE_SessionDesc, + /* ... */ + }; + typedf SlangUInt64 SlangDescTag; + + #define SLANG_MAKE_DESC_TAG(TYPE) SlangDescTag(UInt64(SLANG_DESC_TYPE_##TYPE) << 32 | sizeof(Slang##TYPE)) + + struct SlangSessionDesc + { + SlangDescTag tag; + + SlangTargetDesc const* targets; + SlangInt targetCount; + /* ... */ + }; + /* ... */ + + #define SLANG_DESC_TAG_SESSION_DESC SLANG_MAKE_DESC_TAG(SessionDesc) + /* ... */ + +/* Forward Declarations */ + + typedef struct SlangSession SlangSession; + /* ... */ + +/* Interfaces */ + + /* There's already a lot known about how to define COM interfaces for + consumption from C, so this is actually mostly straightforward. + + Note: these definitions would *only* be needed in the case where we + are compiling the actual implementations of the C API functions. It + is possible that we can/should just not bother with these, under + the assumption that anybody who wants a true pure-C API probably wants + a linkable "stub" library anyway, in which case we can provide that + library ourselves, and compile it as C++. + */ + + /* TODO: The big thing I'm skipping here is setup for the UUIDs. + I think we can provide C-compatible macros for those pretty easily, + but exactly what that should look like is maybe more complicated. */ + + struct SlangSession + { + /* The long/short is that we define a pointer field to a struct + of function pointers, which matches the expected C++ virtual + function table layout. + */ + + struct + { + /* Note: methods from all base interfaces need to go here... */ + + SLANG_NO_THROW SlangResult SLANG_MCALL (*_createCompileRequest)( + SlangSession* session, + void const* const* descs, + SlangInt descCount, + SlangUUID const* uuid, + void** outObject); + + /* ... */ + + } * vtbl; + }; + /* ... */ + + /* With the core type declarations out of the way, the actual functions + that forward to it are easy enough: + */ + inline SlangResult _SlangSession_createCompileRequest( + SlangSession* session, + void const* const* descs, + SlangInt descCount, + SlangUUID const* uuid, + void** outObject) + { + /* The only interesting complications here are the `->vtbl` + and the need to pass `session` explicitly. We could probably + macro away the difference if we don't want to have distinct + C-API-compiled-via-C++ and C-API-compiled-via-C cases. + */ + return session->vtbl->_createCompileRequest( + session, descs, descCount, uuid, outObject); + } + + /* The declarations of the global session creation stuff are almost + identical, so there's no real need to dpulicate it here. + */ + + /* For the true pure-C users, we probably want to provide convenience + functions and/or macros to enable the casts that should be statically + possible.*/ + inline SlangUnknown* SlangSession_asUnknown(SlangSession* session) + { + return (SlangUnknown*) session; + } + /* ... */ + + +/* + +Okay, so that's the basic idea of the proposal for how to expose our API(s). + +I realize this didn't get into the actual details of type hierarchies or what +the actual "desc" types need to be for Slang and gfx. The focus here was much +more on the syntactic side of things, in terms of how we can define our API +so that both C and C++ are usable and can be freely intermixed within a codebase. + +*/ + +/* There's probably an entire additional document that could be written about +utility/wrapper stuff to make the interfaces nicer for C++ users. Some examples +follow: + +We could consider having a hierarchy of wrapper smart-pointer types that codify the +reference-counting policies without the user having to really think about `ComPtr` stuff: + + struct Unknown + { + public: + // typical stuff... + + + protected: + slang::IUnknown* _ptr = nullptr; + } + + struct Session : Unknown + { + public: + ISession* get() const { return (ISession*)_ptr; } + operator ISession*() const { return get(); } + + Result createCompileRequest( + CompileRequestDesc const& desc, + CompileRequest* outCompileRequest) + { ... } + } + +Another thing to consider is whether any of our COM-ish wrappers should allow for +use of exceptions instead of `Result`s: + + struct ISession : ... + { + ... + +#if SLANG_ENABLE_SMART_PTR + ... + + #if SLANG_ENABLE_EXCEPTIONS + SLANG_SMART_PTR(ICompileRequest) createCompileRequest( + CompileRequestDesc const& desc) + { + SLANG_SMART_PTR(ICompileReqest) compileRequest; + SLANG_THROW_IF_FAIL(_createCompileRequest( + &desc, 1, SLANG_UUID_OF(IComileRequest), comileRequest.writeRef())); + return compileRequest; + } + + ... + #endif +#endif + } + +Both for the sake of C API and especialy for gfx (both C and C++), we should consider +defining some coarse-grained aggregate desc types as utilities: + + struct SimpleRasterizationPipelineStateDesc + { + // sub-descs for all the relevant pieces: + // + PipelineProgramDec program; + DepthStencilDesc depthStencil; + MultisampleDesc multisample; + PrimitiveTopologyDesc primitiveTopology; + NVAPIDesc nvapi; + // ... + + // "fluent"-style setters for all the relevant pieces: + + SimpleRasterizationPipelineStateDesc& setEnableDepthTest(bool value) + { + markDepthStencilDescUsed(); + depthStencil.enableDepthTest = value; + return *this; + } + + // ... + + // This is also the logical granularity to provide things like + // List members for attachments, etc. rather than just pointer-and-count: + + private: List colorAttachments; + public: AttachmentDesc& addColorAttachement(); + + // There should also be convenience constructors common cases + // (especially relevant for things like textures). + + // In the simplest implementation strategy, we keep a bitmask for which + // of the sub-descs have actually beem used (either requested by the user, + // or set to non-default values): + // + enum class SubDesc { Program, DepthStencil, ... Count }; + uint32_t usedSubDescs = 0; + void markSubDescUsed(SubDesc d) + { + uint32_t bit = 1 << int(d); + if(usedSubDesc & bit) return; + + usedSubDescs |= bit; + updatePointers(); + } + + // We then maintain a compacted array of all the sub-descriptors needed + // to form the combined state for passing along to the lower-level API. + // + void* subDescs[int(SubDesc::Count)]; + int subDescCount = 0; + + void updatePointers() + { + subDescCount = 0; + if(usedSubDescs & (1 << int(Program))) + { + subDescs[subDescCount++] = &program; + } + /// ... + } + }; + +While the implementation of this monolithic desc types would not necessarily be pretty, +it would enable users who want the benefits of the "one big struct" appraoch to get +what they seem to want. + +The next step down this road is to take these aggregate desc types and turn them into +actual API objects for the purposes of the C API, so that users can more conveniently +create stuff: + + GFXRasterizationPipelineStateBuilder* GFXDevice_beginCreateRasterizationPipelineState( + GFXDevice* device); + + void GFXRasterizationPipelineStateBuilder_setEnableDepthTest( + GFXRasterizationPipelineStateBuilder* builder, + bool enable); + + // Note: frees the given `builder`, so user doesn't have to do it manually + GFXPipelineState* GFXRasterizationPipelineStateBuilder_create( + GFXRasterizationPipelineStateBuilder* builder); + +Obviously the function names are very verbose there, but they could probably be cleaned +up a lot if we want to go down this route. Certainly, if we decide that C API users are +not going to be inclined to use a lot of fine-grained descs, this starts to seem like +an increasingly attractive way to go. +*/ + +#endif +``` \ No newline at end of file diff --git a/docs/proposals/legacy/003-error-handling.md b/docs/proposals/legacy/003-error-handling.md new file mode 100644 index 000000000..28652824f --- /dev/null +++ b/docs/proposals/legacy/003-error-handling.md @@ -0,0 +1,296 @@ +Error Handling +============== + +Slang should support a modern system for functions to signal, propagate, and handle errors. + +Status +------ + +In discussion. + +Background +---------- + +Errors happen. It is impossible for any programming language to statically rule out the possibility of unexpected situations arising at runtime. +There are a wide variety of strategies used in programming, both provided by languages and enforced by idiom in codebases. + +Not all errors are alike, in that some are more expected and reasonable to handle than others. +Most errors can fit into a few broad categories like: + +* Unrecoverable or nearly unrecoverable failures like resource exhaustion (out of memory), or an OS-level signal to terminate the process. + +* Incorrct usage of an API in ways that violate invariants. For example, passing a negative value to a function that says it only accepts positive values. + +* Out-of-range or otherwise invalid data coming from program users. For example, a console program asks the user to type a number, but the user enters some string that does not parse as a number. + +* Failure of operation that will usually succeed, but for which exceptional circumstances can lead to failures. For example, when reading from an open file we typically expect success, but failure is possible for many reasons outside of a programmer's control (like network disruption when accessing a remote file). A robust program often wants to recover from such failures, but often the policy for how recovery should occur is at a higher level than the code that first detects the error. + +These different categories often benefit from different strategies: + +* Typically there is neither a reason nor a desire to do anything about nearly-unrecoverable errors; the program has well and truly crashed. + +* When programmers violate the invariants of an API, they typically want to know about it as early as possible (during development) so they can fix their code. Breaking into the debugger is often the best answer, and in many cases trying to propagate or recover from such failures would be wasted effort. + +* When an operation could fail due to mal-formed data coming from a user, programmers typically want to be forced to handle the failure case at the point where the error may arise. In languages that have an `Optional` or `Maybe` type, it is often easiest to return that. + +* Unpredictable, exceptional, and recoverable errors are among the hardest to deal with, and often benefit from direct language support. + +The Slang language currently doesn't have direct support for *any* form of error handling, but this document focuses on errors in the last of the categories above. + +Related Work +------------ + +In the absence of language support, developers typically signal and propagate errors using *error codes*. The COM `HRESULT` type is a notable example of a well-defined system for using error codes in C/C++ and other languages. +Error codes have the benefit of being easy to implement, and relatively light-weight. +The main drawback of error codes is that developers often forget to check and/or propagate them, and when they do remember to do so it adds a lot of boilerplate. +Additonally, reserving the return value of every function for returning an error code makes code more complex because the *actual* return value must be passed via a function parameter. + +C++ uses *exceptions* for errors in various categories, including unpredictable but recoverable failures. +Propagation of errors up the call stack is entirely automatic, with unwinding of call frames and destruction of their local state occuring as part of the search for a handler. +Neither functions that may throw nor call sites to such functions are syntactically marked. +Exceptions in C++ have often been implemented in ways that add overhead and require complicated support in platform ABIs and intermediate languages to support. + +Java uses exceptions with similar rules to C++, but adds a restriction that functions must be marked with the types of exceptions they may throw or propagate, except for those that inherit from `RuntimeException`, which are intended to represent some of the other categories of error in our taxonomy (like simple invariant violations and nearly unrecoverable errors). +The need to mark every function that might fail (or propagate failure) was seen by most developers at the time as unreasonably onerous. +Developers often smuggled other kinds of exceptions out through `RuntimeException`s, to get them through API layers that were not designed to support exceptions. + +Both Rust and Swift try to strike a balance between error codes and languages with exceptions. +At a high level, each takes an approach where the generated code is comparable to an error-code-based solution (so that no special ABI or IL support is needed), but direct syntactic support makes propagating and/or handling errors more convenient. + +In Rust, a function that returns `std::Result` either returns successfully with a value of type `SomeType`, or fails with an error of type `SomeError`. +The `Result` type is itself just a Rust `enum`, so that results can be handled by pattern-matching with `match`, `if let`, etc. +Direct syntactic support is added so that in the body of a `Result`-returning function, a postfix `?` operator can be applied to an expression of type `Result` to implicitly propagate `E` on any failure, and return the `X` value otherwise. +Some higher-order functions can Just Work with `Result`-returning functions, if their signatures are compatible, but many operations like `map()`, `fold()`, etc. need distinct overloads that support `Result`s. +Functions that return `X` and those that return `Result` are not directly convertible. + +Swift provides more syntactic support for errors than Rust, although the underlying mechanism is similar. +A Swift function may have `throws` added between the parameter list and return type to indicate that a function may yield an error. +All errors in Swift must implement the `Error` protocol, and all functions that can `throw` may produce any `Error` (although there are proposals to extend Swift with "typed `throws`"). +Any call site to a `throws` function must have a prefix `try` (e.g., `try f(a, b)`), which works simiarly to Rust's `?`; any error produced by the called function is propagated, and the ordinary result is returned. +Swift provides an explicit `do { ... } catch ...` construct that allows handlers to be established. +It also provides for conversion between exceptions and an explicit `Result` type, akin to Rust's. +Higher-order functions may be declared as `rethrows` to indicate that whether or not they throw depends on whether or not any of their function-type parameters is actually a `throws` function at a call site. +Any non-`throws` function/closure may be implicitly converted to the equivalent `throws` signature, so that non-throwing functions are subtypes of the throwing ones. + + +The model used in Swift is compatible with the more general notion of *effects* in type theory. +A simple model of function types like `D -> R` can be extended to support zero or more effects `E0`, `E1`, etc. that live "on the arrow": `D -{E0, E1}-> R`. +Purely functional languages like Haskell sometimes use monads as a way to represent effects: a function `D -> IO R` is effectively a function from `D` to `R` with the addition effect that it may perform IO. +Making effects more explicit allows a type system to reason about sub-typing in the presence of effects (a function type without effect `E` is a subtype of a function with that effect), and to express code that is generic over effects. + +Proposed Approach +----------------- + +We propose a modest starting point for error handling in Slang that can be extended over time. +The model borrows heavily from Swift, but also focuses on strongly-typed errors. + +The standard library will provide a built-in interface for errors, initially empty: + +``` +interface IError {} +``` + +User code can define their own types (`struct` or `enum`) that conform to `IError`: + +``` +enum MyError : IError +{ + BadHandle, + TimedOut, + // ... +} +``` + +User-defined functions (in both traditional and "modern" syntax) will support a `throws ...` clause to specify the type of errors that the function may produce: + +``` +float f(int x) throws MyError { ... } + +func g(x: int) throws -> float MyError { ... } +``` + +Call sites to a `throws` function must wrap any potentially-throwing expression with a `try`: + +``` +float g(int y) throws MyError +{ + return 1.0f + try f(y-1); +} +``` + +Code can explicitly raise an error using a `throw` expression: + +``` +throw MyError.TimedOut; +``` + +We will allow `catch` clauses to come at the end of any `{}`-enclosed scope, where they will apply to any errors produced by `throw` or `try` expressions in that scope. + +``` +{ + ... + try f(...); + ... + + catch( e: MyError ) { ... } +} +``` + +We will also want to add `defer` statements, as they are defined in Go, Rust, Swift, etc. +The statements under a `defer` will always be run when exiting a scope, even if exiting as part of error propagation. + +Detailed Explanation +-------------------- + +Consider a function that uses most of the facilities we have defined: + +``` +float example(int x) throws MyError +{ + if(someCondition) + { + throw MyError.TimedOut; + } + ... + defer { someCleanup(); } + ... + { + let y : int = 1 + try g(...); + + catch(e : MyError) + { ... } + } + ... + return someValue; +} +``` + +We will show how a function in this form can be transformed via incremental steps into something that can be understood and compiled without specific support for errors. + +### Change Signature + +First, we transform the signature of the function so that it returns something akin to an `Optional` and returns its result via an `out` parameter, and modify any `return` points to write the `out` parameter and return `null` (the not-present case of `Optional`): + +``` +MyError example_modified(int x, out float _result) +{ + ... + + _result = someValue; + return null; +} +``` + +### Desugar `try` Expressions + +Next we can convert any `try` expressions into a more explicit form, to match the transformation of signature. A statement like this: + +``` +let y : int = 1 + try g(...); +``` + +transforms into something like: + +``` +var _tmp : int; +let _err : Optional = g_modified(..., out _tmp); +if( _err != null ) +{ + throw _err.wrappedValue; +} +let y : int = 1 + _tmp; +``` + +### Desugar `throw` Expressions + +For every `throw` site in a function body, there will either be no in-scope `catch` clause that matches the type thrown, or there will be eactly one most-deeply-nested `catch` that statically matches. +Front-end semantic checking should be able to associate each `throw` with the appropriate `catch` if any. + +For `throw` sites with no matching `catch`, the operation simply translates to a `return` of the thrown error (because of the way we transformed the function signature). + +For `throw` sites with a matching `catch`, we treat the operation a a "`goto` with argument" that jumps to the `catch` clause and passes it the error. +Note that our IR structure already has a concept of "`goto` with arguments". + +### Desugar `defer` Statements + +Handling of `defer` statements is actually the hardest part of this proposal, and as such we should probably handle `defer` as a distinct feature that just happens to overlap with what is being proposed here. + +### Subtyping: Front-End + +We should (at some point) add a `Never` type to the Slang type system, which would be an uninhabitable type suitable for use as the return type of functions that never return: + +``` +func exit(code: int) -> Never; // C `exit()` never returns +``` + +`Never` is effectively a subtype of *every* type and, as such, an expression of type `Never` can be implicitly converted to any type. + +A `throw` expression has the type `Never`, allowing a user to write code like: + +``` +// Because `Never` can convert to `int`, this is valid: +int x = value > 0 ? value : throw MyError.OutOfBounds; +``` + +A function without a `throws` clause is semantically equivalent to a function with `throws Never`. +If we make that equivalence concrete at the type-system level, then a higher-order function can be generic over both throwing and non-throwing functions: + +``` +func map( + f: (D) throws E -> R, + l: List) + throws E -> List; +``` + +A function type with `throws X` is a subtype of a function with `throws Y` if `X` is a subtype of `Y`. +That includes the case where `X` is `Never`, so that a non-`throws` function type is a subtype of any `throws` function type with the same parameter/result signature. + +### Subtyping: Low-Level + +The subtyping relationship for `Never` *values* is irrelevant to codegen. Any place in the IR that has a `Never` value available to it represents unreachable code. + +The subtyping relationship for `Never` in function types is more challenging, both for result types and error types. At the most basic, we can inject trampoline/thunk functions at any points where we have a `Never`-yielding function and need a function that returns `X` to pass along. + +If we were doing low-level code generation for a platform where we can define our ABI, it would be possible to have `throws` and non-`throws` functions use distinct calling conventions, such that: + +* The orinary parameters and reuslts are passed in the same registers/locations in both conventions. + +* The error value (if any) in the `throws` convention is passed via registers/locations that are callee-save in the non-`throws` convention. + +Under that model, a call site to a potentially-`throws` function can initialize the registers/locations for the error result to `null`/zero before dispatching to the callee. +If the callee is actually a non-`throws` function it would not touch those registers, and no error would be detected. +In that case, a non-`throws` function/closure could be used directly as a `throws` one with no conversion. +Such calling-convention trickery isn't really possible to implement when emitting code in a high-level language like HLSL/GLSL or C/C++. + +Questions +-------------- + +### Should we support the superficially simpler case of "untyped" `throws`? + +Having an `IError` interface allows us to eventually decide that `throws` without an explicit type is equivalent to `throw IError`. +It doesn't seem necessary to implement that convenience for a first pass, especially when there are use cases for `throws` that might not want to get into the mess of existential types. + +### Should the transformations described here be implemented during AST->IR lowering, or at the IR level? + +That's a great question! My guess is that some desugaring will happen during lowering, but we will probably want to keep `throws` functions more explicitly represented in the IR until fairly late, so that we can desugar them differently for different targets (if desired). + +### Do we need `Optional` to be supported to make this work? + +It is unlikely that we'd need it to be a user-visible feature in a first pass, but we might want it at the IR level. +For this feature to work, we really need `sizeof(Optional)` to be the same as `sizeof(X)` for simple cases where `X` is an `enum` or (for suitable targets) a type that is pointer-based. + +A first pass at the feature might only support cases where error types are `enum`s and where the zero value is the "no error" case. + +### Should we have a `Result` type akin to what Rust/Swift have? Should a `throws E` function be equivalent to one that returns `Result`? + +That all sounds nice, but for now it seems like overkill. +Slang doesn't really have any facilities for programming with higher-order functions, pattern matching, etc. so adding types that mostly shine in those cases seems like a stretch. + +Alternatives Considered +----------------------- + +We could decide that Slang shouldn't be in the business of providing error-handling sugar *at all* and make this a problem for users. +That isn't really a reasonable plan for any modern language, but it is the status quo and null hypothesis if we don't start in on a better plan. + +We could try to focus on C++ interop/compatibility and decide that errors in Slang should use exceptions, and only make "proper" language-supported error handling available to platforms that support exceptions at a suitably low level. +Doing so would give us all the disadvantages of C++ exceptions, and also mean that most of our users wouldn't end up using our error-handling tools, because doing so would render code non-portable. diff --git a/docs/proposals/legacy/004-com-support.md b/docs/proposals/legacy/004-com-support.md new file mode 100644 index 000000000..dc0bd52d3 --- /dev/null +++ b/docs/proposals/legacy/004-com-support.md @@ -0,0 +1,240 @@ +COM Support +=========== + +When Slang is used as a host/CPU programming language, it is likely that users will want to use interact with COM interfaces, either by consuming them or implementing them. +The Slang language and compiler should provide some first-class features to make working with COM interfaces feel lightweight and natural. + +Status +------ + +Implemented. + +Background +---------- + +COM is not perfect, but it is one of the only real solutions for cross-platform portable C++ APIs that care about binary compatibility and versioning. +Developers who use Slang are likely to write code that uses COM, whether to interact with Slang itself (and/or GFX), or with platform APIs like D3D. + +While COM provides idioms for addressing many practical challenges, it is also inconvenient in that it introduces a lot of *boilerplate*: + +* COM types all need to implement the core `IUnknown` operations for casting/querying and reference counting. + +* Code using COM interfaces needs to perform `AddRef` and `Release` operations manually, or use smart pointer types to automate lifetime management. + +* Code that calls into COM interfaces typically needs to use boilerplate code and/or macros to deal with `HRESULT` error codes, handling or propagating them as needed. + +Our in-progress work on supporting CPU programming in Slang emphasizes supporting idiomatic code without a lot of boilerplate. +Our intended path includes things that are compatible with the COM philosophy, like reference-counted lifetime management and idiomatic use of result/error codes, but those features don't currently align with the more explicit style used by COM in C/C++. + +Related Work +------------ + +The .NET platform includes some support for allowing .NET `interface`s and COM interfaces to interoperate. +TODO: Need to study this and learn how it works. + +Proposed Approach +----------------- + +We propose to allow COM interfaces to be declared using the Slang `interface` construct, with an appropriate attribute or modifier: + +``` +[COM] interface IDevice +{ + ITexture createTexture(__read TextureDesc desc) throws HRESULT; + + void setTexture(int index, ITexture texture); +} +``` + +A declaration like the above will translate into output C++ along the lines of: + +``` +struct IDevice : public IUnknown +{ + virtual HRESULT SLANG_MCALL createTexture(TextureDesc const& desc, ITexture** _result) = 0; + + virtual void SLANG_MCALL setTexture(int index, ITexture* texture) = 0; +}; +``` + +Key things to note: + +* The `[COM] interface` becomes a C++ `struct` that inherits from `IUnknown` +* Methods defined in the `[COM] interface` become pure-virtual `SLANG_MCALL` methods in C++ +* Parametes/values of a `[COM] interface` type `IFoo` in Slang translate to `IFoo*` in C++ +* Methods that have a `throws HRESULT` clause are transformed to have an `HRESULT` return type and an output parameter for their result + +A Slang `class` can declare that it implements zero or more `[COM] interface`s. Code like this: + +``` +class MyTexture : ITexture +{ + // ... +} + +class MyDevice : IDevice +{ + ITexture createTexture(__read TextureDesc desc) throws HRESULT + { + return ...; + } + + void setTexture(int index, ITexture texture) + { + ...; + } +} +``` + +translates into output C++ like this: + +``` +class MyTexture : public slang::Object, public ITexture +{ + // ... +}; + +class MyDevice : public slang::Object, public IDevice +{ + virtual HRESULT QueryInterface(REFIID riid, void **ppvObject) { ... } + virtual ULONG AddRef() { ... } + virtual ULONG Release() { ... } + + HRESULT createTexture(TextureDesc const& desc, ITexture** _result) SLANG_OVERRIDE + { + _result = ...; + return S_OK; + } + + void setTexture(int index, ITexture* texture) SLANG_OVERRIDE + { + ... + } +} +``` + +All Slang `class`es translate to C++ classes that inherit from `slang::Object` (equivalent to the `RefObject` type within the current Slang implementation). +A `class` that inherits any `[COM]` interfaces includes an implementation of `IUnknown` plus the methods to override all the interface requirements. + +In ordinary code that makes use of `[COM] interface` types: + +``` +struct Stuff +{ + ITexture t; +} +ITexture getTexture(Stuff stuff) +{ + return stuff.t; +} +ITexture someOperations(IDevice device) +{ + let t = device.createTexture(...); + return t; +} +``` + +the C++ output uses C++ smart-pointer types for local variables, `struct` fields, and function results: + +``` +struct Stuff +{ + ComPtr t; +}; +ComPtr getTexture(Stuff stuff) +{ + return stuff.t; +} +HRESULT someOperations(IDevice* device, ComPtr& _result) +{ + ComPtr t; + HRESULT _err = device->createTexture(&t); + if(_err < 0) return _err; + + _result = t; + return 0; +} +``` + +As a small optimization, an `in` parameter of a `[COM] interface` type translates as a C++ parameter of just the matching pointer type (see `device` above). + +Note that the translation of idiomatic `HRESULT` return codes into `throws HRESULT` functions in Slang allows code working with COM interfaces to benefit from the convenience of the Slang error handling model. + + +Detailed Explanation +-------------------- + +This is a case where the simple explanation above covers most of the interesting stuff. + +There are a lot of semantic checks we'd need/want to implement to make sure `[COM]` interfaces are used correctly: + +* Any `interface` that inherits from one or more other `[COM]` interfaces must itself be `[COM]` + +* Any concrete type that implements a `[COM]` interface must be a `class` + +There are also detailed implementation questions to be answered around the in-memory layout of `class` types that implement `[COM]` interfaces. +In particular, we might want to be able to optimize for the case of a single-inheritance `class` hierarchy that mirrors a `[COM] interface` hierarchy, since this comes up often for COM-based APIs: + +``` +interface IBase { void baseFunc(); } +interface IDerived : IBase { void derivedFunc(); } + +class BaseImpl : IBase { ... } +class DerivedImpl : BaseImpl, IDerived { ... } +``` + +Using a naive translation to C++, the `DerivedImpl` type could end up with *three* different virtual function table (`vtbl`) pointers embedded in it: one for `slang::Object`, one for `IBase`, and one for `IDerived`. +Clearly the `vtbl`s for `IBase` and `IDerived` could be shared, but C++ `class`es can't easily express this. +Furthermore, if we are able to tune our strategy for layout, we can set things up so that `[COM] interface`s consume `vtbl` slots starting at index `0` and counting up, while any `virtual` methods in `class`es consume slots starting at index `-1` and counting *down*. +Using such a layout strategy we can actually allow a type like `DerivedImpl` above to use only a *single* `vtbl` pointer. + +Questions +-------------- + +### Should we emit COM code that works at the plain C level, or idiomatic C++? + +I honestly don't know. Emitting idomatic C++ (and using things like smart pointers) certainly makes the output code easier to understand. + +### Can we make this work with more advanced features of Slang interfaces? + +Some Slang features don't have perfect analogues. For example, given that `[COM] interface`s can only be conformed to by `class` types, the use of `[mutating]` isn't especially meaningful. + +There is no reason why a `[COM] interface` couldn't make use of `static` methods, but there would be no way to call those from C++ without an instance of the interface type. + +A `[COM] interface` could include `property` declarations, provided that we define the rules for how they translate into getter/setter operations in the generated output. + +One interesting case is that a `[COM] interface` could allow use of `This`, as well as `associatedtype`s that are themselves constrained by a `[COM] interface`. +For example, we could instead device our `IDevice` interface from before as: + +``` +[COM] interface IDevice +{ + associatedtype Texture : ITexture; + + Texture createTexture(__read TextureDesc desc) throws HRESULT; + + void setTexture(int index, Texture texture); +} +``` + +We could set things up so that the `associatedtype` has no impact on the C++ translation of `IDevice`: all parameter/result types that use the `Texture` associated type would translate to `ITexture*` in C++. +As such, a more refined `interface` like this would not disrupt the binary interface of a COM-based API, but could be allowed to express more of the constraints of the underlying API at compile time. +For example, the above use of `associatedtype` would prevent Slang code from mixing up textures across devices: + +``` +void someFunc( IDevice a, IDevice b ) +{ + let x = a.createTexture(...); + let y = b.createTexture(...); + + a.setTexture(0, y); // COMPILE TIME ERROR! +} +``` + +In this example, `y` has type `b.Texture`, while `a.setTexture` expects an argument of type `a.Texture` (a distinct type, even if it also conforms to `ITexture`). +The benefits of this approach are probably purely hypothetical until we make it a lot easier to work with dependent types like `a.Texture` in Slang code. + +Alternatives Considered +----------------------- + +The main alternative would be to have Slang's model for interop with C/C++ focus primarily on C alone, and only allow use of COM-based APIs through C-compatible interfaces. diff --git a/docs/proposals/legacy/005-components.md b/docs/proposals/legacy/005-components.md new file mode 100644 index 000000000..b257140a7 --- /dev/null +++ b/docs/proposals/legacy/005-components.md @@ -0,0 +1,507 @@ +Components +========== + +We propose to extend Slang with a construct for defining coarse-grained *components* that can be used to assemble shader programs. + +Status +------ + +Under discussion. + +Background +---------- + +First, a bit of terminology. In the context of a specific language like Slang, a term like "component" or "module" will often have a narrow meaning, but when we want to discuss "modularity" broadly we need to have a way to refer to the *things* we want to have be modular: the units of modularity. +In this document we will use the term *unit* to refer abstractly to anything that is a unit of modularity for some context/purpose/system, and try to reserve other terms for cases where we mean something more specific. + +While Slang has many features that address modularity for "small" units, it is still lacking in constructs that adequately address the needs of "large" units. +These are qualitative distinctions, but some examples may clarify the kind of distinction we mean. +An interface `ILight` for light sources, and a `struct` type `OmniLight` that conforms to it are small units. +An entire `LightSampling` module in a path tracer is a much larger unit. + +The main tools that Slang provides for "small" units are: `interface`s, `struct`s, and `ParameterBlock`s. +Interfaces allow a developer to codify the types and operations that clients of a unit may rely on, as well as the requirements that implementations must provide. +Structure types are the main way developers can implement an interface, and it is important for GPU efficiency that Slang `struct`s are *value types*. +By using `ParameterBlock`s, developers can connect the units of modularity used *within* shader code with the parameters passed from *outside* their shaders. + +When we talk about "large" units of modularity, the main thing Slang provides are, well, modules. +A Slang module is basically just a collection of global-scope declarations: types, functions, shader parameters, and entry points. +The `import` construct allows modules to express a dependency on one another, and if/when we add `public`/`internal` visibility qualifiers it will also be able to restrict clients of a module to its defined interface. + +What modules *don't* provide is any of the flexibility that `interface`s provide for "small" units like `struct` types. +There is no first-class way for a Slang programmer to define a common interface that multiple modules implement, and then to express another piece of code that can work with any of those implementations. +Aside from falling back to preprocessor hackery (which negates many of the benefits of Slang in terms of separate compilation), the only way for developers to try to recoup those benefits is to use the tools for "small" units. + +Let's consider a placeholder/pseudocode set of Slang modules that work together: + +``` +// Lighting.slang + +interface ILight { ... } + +StructuredBuffer gLights; + +void doLightingStuff() { ... } +``` + +``` +// Materials.slang +import Lighting; + +interface IMaterial { ... } +StructuredBuffer gMaterials; + +void doMaterialStuff() +{ + doLightingStuff(); +} +``` + +``` +// Integrator.slang +import Lighting; +import Materials; + +ParameterBlock gIntegratorParams; + +void doIntegeratorStuff() +{ + doLightingStuff(); + doMaterialsStuff(); +} +``` + +The details of the module implementations is not the important part here. +The key is that each module defines a collection of types, operations, and shader parameters, and there is a dependency relationship between the modules. +Note that the `Integrator` module depends on both `Lighting` and `Materials`, but it does *not* need to be actively concerned with the fact that `Materials` *also* depends on `Lighting`. + +If we look only at a leaf module like `Lighting` it seems simple enough to translate it into something based on our "small" modularity features: + +``` +// Lighting.slang + +interface ILight { ... } + +interface ILightingSystem +{ + void doLightingStuff(); +} + +struct DefaultLightingSystem : ILightingSystem +{ + StructuredBuffer lights; + + void doLightingStuff() { ... } +} +``` + +Here we were able to move most of the global-scope code in `Lighting.slang` into a `struct` type called `DefaultLightingSystem`. +We were also able to define an explicit `interface` for the system, which makes explicit that we don't consider `gLights` part of the public interface of the system (only `doLighting()`). +By defining the interface, we also create the possibility of plugging in other implementations of `ILightingSystem` - for example, we can imagine a `NullLightingSystem` that actually doesn't perform any lighting (perhaps useful for performane analysis). + +Translating `Materials` in the same way that we did for `Lighting` leads to some immediate questions: + +``` +// Materials.slang +import Lighting; + +interface IMaterial { ... } + +interface IMaterialSystem +{ + void doMaterialStuff(); +} + +struct DefaultMaterialSystem +{ + StructuredBuffer materials; + + void doMaterialStuff() + { + /* ???WHAT GOES HERE??? */.doLightingStuff(); + } +} +``` + +When our `DefaultMaterialSystem` wants to invoke code for lighting, it needs a way to refer to the lighting system. +Beyond that, we want it to be able to work with *any* implementation of `ILightingSystem`. + +A naive first attempt might be to give `DefaultMaterialSystem` field that refers to an `ILightingSystem`: + +``` +struct DefaultMaterialSystem +{ + ILightingSystem lighting; + ... + void doMaterialStuff() + { + lighting.doLightingStuff(); + } +} +``` + +An approach light that (or even one using a `ParameterBlock`) runs into the problem that it is going to force the Slang compiler to use its layout strategy for dynamic dispatch, which cannot handle the resource types in `DefaultLightingSystem` when compiling for most current GPU targets. +There are other problems with directly aggregating an `ILightingSystem` into our material system, but those will need to wait for a bit. + +If we want to allow the code in our `DefaultMaterialSystem` to statically specialize to the type of the lighting system, we end up to use generics, either by making the whole type generic: + +``` +struct DefaultMaterialSystem< L : ILightingSystem > +{ + ParameterBlock lighting; + ... +} +``` + +or by just making the `doMaterialStuff()` operation generic, with the lighting system being passed in as a parameter. + +``` +struct DefaultMaterialSystem +{ + ... + void doMaterialStuff< L : ILightingSystem>( L lighting ) + { + lighting.doLightingStuff(); + } +} +``` + +Each of those options moves the responsibility for managing the lighting system type up a level of abstraction: whatever code works with a material system needs to manage the details. + +When we now step up the next level to the `Integrator` module, the approach using `struct`s really starts to show cracks. +We have the option of making the `DefaultIntegeratorSystem` a generic on the type of *both* subsystems, and reference them via parameter blocks: + +``` +struct DefaultIntegratorSystem< L : ILightingSystem, M : IMaterialSystem > +{ + ParameterBlock lighting; + ParameterBlock material; + ... +} +``` + +or we have to make all the relevant operations on the integrator take both subsystems as pass the relevant subsystem instances in as parameters: + +``` +struct DefaultIntegratorSystem +{ + ... + void doIntegeratorStuff< L : ILightingSystem, M : IMaterialSystem >( + L lighting, + M material) + { + lighting.doLightingStuff(); + material.doMaterialsStuff(lighting); + } +} +``` + +In each case, more and more responsibiity for configuration of implementation details is being punted up to the next higher level of abstraction. +In the first case, somebody else is responsible for instantiating a type like: + +``` +DefaultIntegeratorSystem> +``` + +Also, the application code that works with that messy type needs to make sure to fill in *one* parameter block for the lighting system, but set it into *both* the material system and integrator. + +In the second case, note how the integrator already has to ensure that it passes along the `lighting` subsystem to the `doMaterialStuff` operation, and anybody who invokes `doIntegratorSutff` would have to do the same, but for *two* subsystems. + +The whole thing doesn't scale and becomes intractable with more than a few subsystems. +Trying to work anything like inheritance into the mix just falls flat completely. + +Related Work +------------ + +There is a lot of work in general-purpose programming languages around defining larger-scale modularity units. + +SML (Standard ML) has both modules and *signatures*, which are effectively interfaces for modules. +Modules can be parameterized on other modules based on signatures, and instantiated to use different concrete implementations. + +Beta and gbeta unify both classes and functions into a single construct called a *pattern*, and show that patterns (including pattern inheritance) can be used for things akin to traditional modules. +A variety of techniques for *family polymorphism* in world of Java and similar languages followed on from that tradition. +The Scala language continues in the same vein, with papers and presentations on Scala advocating for using `class`es to model large units of modularity akin to modules. + +In the world of "enterprise" software using Java, C#, JavaScript, etc. there is a large family of techniques and system for "dependency inversion" which is used to automate some or all of the process of "wiring up" the concrete implementations of various subsystems/components based on explicit representations of dependencies (often attached as metadata on the fields of a type). + +Modern general-purpose game engines like Unity and Unreal often use a "component" concept, where a game entity/object is composed of multiple loosely-coupled sub-objects (components). +Often these systems allow dependencies between component types to be stated explicitly, with runtime or tools support for ensuring that objects are not created with unsatisifed dependencies. + +Note that almost all of the approaches enumerated above rely deeply on the fact that a dependency of unit `X` on unit `Y` can be handily represented as a single pointer/reference in most general-purpose programming languages. For example, in C++: + +``` +class Y { ... }; +class X +{ + Y* y; + ... +}; +``` + +In the above, an instance of `X` can always find the `Y` it depends on easily and (relatively) efficiently. +There is no particularly high overhead to having `X` diretly store an indirect reference to `Y` (at least not for coarse-grained units), and it is trivial for multiple units like `X` to all share the same *instance* of `Y` (potentially even including mutable state, for applications that like that sort of thing). + +In general most CPU languages (and especially OOP ones) can express the concepts of "is-a" and "has-a" but they often don't distinguish between when "has-as" means "refers-to-and-depends-on-a" vs. when it means "aggregates-and-owns-a". +This is important when looking at a GPU language like Slang, where "aggergates-and-owns-a" is easy (we have `struct` types), but "refers-to-and-depends-on-a" is harder (not all of our targets can really support pointers). + +Proposed Approach +----------------- + +We propose to introduce a new construct called a *component type* to Slang, which can be used to describe units of modularity larger than what `struct`s are good for, but that is intentionally defined in a way that allows it to be used in cases where fully general `class` types could not be supported. + +To render our earlier examples in terms of component types: + +``` +// Lighting.slang + +interface ILight { ... } + +interface ILightingSystem +{ + void doLightingStuff(); +} + +__component_type DefaultLightingSystem : ILightingSystem +{ + StructuredBuffer lights; + + void doLightingStuff() { ... } +} +``` + +``` +// Materials.slang +import Lighting; + +interface IMaterial { ... } +interface IMaterialSystem +{ + void doMaterialStuff(); +} + +__component_type DefualtMaterialSystem : IMaterialSystem +{ + __require lighting : ILightingSystem; + + StructuredBuffer materials; + + void doMaterialStuff() + { + lighting.doLightingStuff(); + } +} +``` + +``` +// Integrator.slang +import Lighting; +import Materials; + +interface IIntegratorSystem +{ + void doIntegratorStuff(); +} + +__component_type DefaultIntegratorSystem : IIntegratorSystem +{ + __require ILightingSystem; + __require IMaterialSystem; + + ParameterBlock params; + + void doIntegeratorStuff() + { + doLightingStuff(); + doMaterialsStuff(); + } +} +``` + +The `__component_type` keyword is akin to `struct` or `class`, but introduces a component type. +Component types are similar to both structure and class types in that they can: + +* Conform to zero or more interfaces +* Define fields, methods, properties, and nested types +* Eventually: optionally inherit from one (or more) other component types + +The key thing that a `__component_type` can do that a `struct` cannot (but that a `class` might be allowed to) is include nested `__require` declarations. +In the simplest form, a require declaration is of the form: + +``` +__require someName : ISomeInterface; +``` + +Within the scope where the `__require` is visible, `someName` will refer to a value that conforms to `ISomeInterface`, but code need not know what the value is (nor what its type is). +The other form of `__require`: + +``` +__require ISomeInterface; +``` + +Can be seen as shorthand for something like: + +``` +__require _anon : ISomeInterface; +using anon.*; // NOTE: not actual Slang syntax +``` + +One more construct is needed to complete the feature, and it can introduced by illustrating a concerete type that pulls together our default implementations: + +``` +__component_type MyProgram +{ + __aggregate lighting : DefaultLightingSystem; + __aggregate DefaultMaterialSystem; + __aggregate DefaultIntegeratorSystem; +} +``` + +An `__aggregate` declaration is only allowed inside a `__component_type` (or a `class`, if we allow it). +Similar to `__require`, an `__aggregate` can either name the thing being aggregated, or leave it anonymous (and have its members imported into the current scope). +The semantics of `__aggregate SomeType` are similar to just declaring a field of `SomeType`, but the key distinction is that the aggregated sub-object is conceptually allocated and initialized as *part* of the outer object (one alternative name for the keyword would be `__part`). +It is not possible to assign to an `__aggregate` member like it is a field (although if the type is a reference type, *its* fields are visible and might still be mutable). + +At the point where an `__aggregate SomeType` member is declared, the front-end semantic checking must be able to find/infer the identity of a value to use to satisfy each `__require` member in `SomeType`. +For example, because `DefaultIntegratorSystem` declares `__require IMaterialSystem`, the compiler searches in the current context for a value that can provide that interface. +It finds a single suitable value: the value implicitly defined by `__aggregate DefaultMaterialSystem`, and thus "wires up" the input dependency of the `DefaultIntegratorSystem`. + +It is posible for a `__require` in an `__aggregate`d member to be satisfied via another `__require` of its outer type: + +``` +__component_type MyUnit +{ + __require ILightingSystem; + __aggregate DefaultMaterialSystem; +} +``` + +In the above example, the `ILightingSystem` requirement in `DefaultMaterialSystem` will be satisfied using the `ILightingSystem` `__require`d by `MyUnit`. + +In cases where automatic search and connection of dependencies does not work (or yields an ambiguity error), the user will need some mechanism to be able to explicitly specify how dependencies should be satisfied. + +While the above examples do not show it, component types should be allowed to contain shader entry points. + +Detailed Explanation +-------------------- + +Component types need to be restricted in where and how they can be used, to avoid creating situations that would give them all the flexiblity of arbitrary `class`es. +The only places where a component type may be used are: + +* `__require` and `__aggregate` declarations +* Function parameters +* Generic arguments (??? Need to double-check how this can go wrong) + +Any given `__component_type` either has no `__require`s and is thus concrete, or it has a nonzero number of `__require`s and is abstract. + +We can ignore the `__require`s in a component type (if any) and form an equivalent `struct` type. +In that `struct` type, `__aggregate`s turn into ordinary fields. +For example, the `MyProgram` (concrete) and `MyUnit` (abstract) types above become: + +``` +struct MyProgram +{ + DefaultLightingSystem lighting; + DefaultMaterialSystem _anon0; + DefaultIntegeratorSystem _anon1; +}; +struct MyUnit +{ + DefaultMaterialSystem _anon2; +}; +``` + +When a component type is used as a function parameter (including an implicit `this` parameter) it effectively maps to a function that takes additional (generic) parameters corresponding to each `__require`. +For example, given: + +``` +void doStuff( MyUnit u ) { ... u.doMaterialStuff(); ... } +``` + +we would generate something like: + +``` +void doStuff< T : ILightingSystem >( + T _anon3, // for the `__require : ILightingSystem` in `MyUnit` + MyUnit u) +{ + ... + DefaultMaterialSystem_doMaterialStuff(_anon3, u._anon2); + ... +} +``` + +Note that when the generated code invokes an operation through one of the `__aggregate`d members of a component type, where the aggregated type had `__require`ments, the compiler must pass along the additional parameters that represent those requirements in the current context. + +Effectively, the compiler generates all of the boilerplate parameter-passing that the programmer would have otherwise had to write by hand. + +It might or might not be obvious that the notion of "component type" being described here has a clear correspondance to the `IComponentType` interface provided by the Slang runtime/compilation API. +It should be possible for us to provide reflection services that allow a programmer to look up a component type by name and get an `IComponentType`. +The existing APIs for composing, specializing, and linking `IComponentType`s should Just Work for explicit `__component_type`s. +Large aggregates akin to `MyProgram` above can be defined entirely via the C++ `IComponentType` API at program runtime. + +Questions +--------- + +### How do we explain to users when to use component types vs. when to use modules? Or `struct`s? + +The basic advice should be something like: + +* If the thing feels subsystem-y, favor modules or component types. If it feels object-y or value-y, use a `struct`. This is loose, but intuition is good here. + +* If the thing wants to depend on other subsystems through `interface`s, to allow mix-and-match flexibility, it should probably be a component type and not a module. + +* If the thing wants to have its own shader parameters, then we encourage users to consider that a component type is likely to be what they want, so that they don't pollute the global scope. + +That last point is important, since a component type allows users to define a collection of global shader parameters and entry points that use them as a unit, without putting those parameters in the global scope, which is something that was not really possible before. + +### Can the `__component_type` construct just be subsumed by either `struct` or `class`? + +Maybe. The key challenge is that component types need to provide the "look and feel" of by-refernece re-use rather than by-value copying. A `__require T` should effectively act like a `T*` and not a bare `T` value, so I am reluctant to say that should map to `struct`. + +### But what about `[mutating]` operations and writing to fields of component types, then? + +Yeah... that's messy. If component types really are by-reference, then they should be implicitly mutable even without passing as `inout`, and should ideally also support aliasing. We need to make sure we get clarity on this. + +### Is `__aggregate` really required? Isn't it basically just a field? + +An `__aggregate X` acts a lot like a field if `X` is a *value* type, but in cases where `X` is a *reference* type, there is a large semantic distinction. + +### How does all this stuff relate to inheritance? + +There are some things that can be done with (multiple) inheritance that can also be expressed via `__require`s. For example, both can represent the "diamond" pattern: + +``` +class A { ... } +class B : A { ... } +class C : A { ... } +class D : B, C { ... } +``` + +``` +__component_type A { ... } +__component_type B { __require A; ... } +__component_type C { __require A; ... } +__component_type D { __require B; __require C; ... } +``` + +The Spark shading language research project used multiple mixin class inheritance to compose units of shader code akin to what are being proposed here as coponent types (hmm... I guess that should go into related work...). + +In general, using inheritance to model something that isn't an "is-a" relationship is poor modeling. +Inheritance as a modelling tool cannot capture some patterns that are possible with `__aggregate` (notably, with mixin inheritance you can't get multiple "copies" of a component). +Most importantly, when inheritance is abused for modeling like this, the resulting code can be confusing. Consider: + +``` +abstract class MyFeature : ISystemA, ISystemB +{ ... } +``` + +From this declaration, it is not possible to tell whether `MyFeature` implements just `ISystemA`, just `ISystemB`, both, or neither. +The distinction between an inheritance clause ("I implement this thing") vs. a `__require` ("I need *somebody else* to implement this thing") is important documentation. + +Alternatives Considered +----------------------- + +I'm not aware of any big design alternatives that don't amount to more or less the same thing with different syntax. +One alternatives is to try to do something like ML-style "signature" for our modules, and allow something like `import ILighting` to allow module-level dependencies on abstracted interfaces. + +Another alternative is to do what this document proposes, but make it work with the existing `struct` keyword (or `class`) instead of adding a new one. diff --git a/docs/proposals/legacy/006-artifact-container-format.md b/docs/proposals/legacy/006-artifact-container-format.md new file mode 100644 index 000000000..81910151b --- /dev/null +++ b/docs/proposals/legacy/006-artifact-container-format.md @@ -0,0 +1,1119 @@ +Shader Container Format +======================= + +This proposal is for a file hierarchy based structure that can be used to represent compile results and more generally a 'shader cache'. Ideally it would feature + +* Does not require an extensive code base to implement +* Flexible and customizable for specific use cases +* Possible to produce a simple fast implementation +* Use simple and open standards where appropriate +* Where possible/appropriate human readable/alterable +* A way to merge, or split out contents that is flexible and easy. Ideally without using Slang tooling. + +Should be able to store + +* Compiled kernels +* Reflection/layout information +* Diagnostic information +* "meta" information detailing user specific, and product specific information +* Source +* Debug information +* Binding meta data +* Customizable, and user specified additional information + +API support needs to allow + +* Interchangeable use of static shader cache/slang compilation/combination of the two + * Implies compilation needs to be initiated in some way that is compatible with shader cache keying +* Ability to store compilations as they are produced + +Needs to be able relate and group products such that with suitable keys, it is relatively fast and easy to find appropriate results. + +It's importance/relevance + +* Provides a way to represent complex compilation results +* Could be used to support an open standard around 'shader cache' +* Provide a standard 'shader cache' system that can be used for Slang tooling and customers +* Supports Slang tooling and language features + +## Use + +There are several kinds of usage scenario + +* A runtime shader cache +* A runtime shader cache with persistence +* A capture of compilations +* A baked persistent cache - must also work without shader source +* A baked persistent cache, that is obfuscated + +A runtime shader cache has the following characteristics: + +* Can works with mechanisms that do not require any user control (such as naming). Ie purely inputs/options can define a 'key'. +* It is okay to have keys/naming that are not human understandable/readable. +* The source is available - such that hashes based on source contents can be produced. +* It does not matter if hashes/keys are static between runs. +* It is not important that a future version would be compatible with a previous version or vice versa. +* Could be all in memory. +* May need mechanism/s to limit the working set +* Generated source can be made to work, because it is possible to hash generated source + +At the other end of the spectrum a baked persistent cache + +* Probably wants user control over naming +* Doesn't have access to source so can't use that as part of a hash +* Probably doesn't have access to dependencies +* Having some indirection between a request and a result is a useful feature +* Ideally can be manipulated and altered without significant tooling +* Generated source may need to be identified in some other way than the source itself + +It should be possible to serialize out a 'runtime shader cache' into the same format as used for persistent cache. It may be harder to use such a cache without Slang tooling, because the mapping from compilation options to keys will probably not be simple. + +Status +------ + +## Gfx + +There is a run time shader cache that is implemented in gfx. + +There is some work around a file system backed shader cache in gfx. + +## Artifact System + +The Artifact provides a mechanism to transport source/compile results through the Slang compiler. It already supports most of the different items that need to be stored. + +Artifact has support for "containers". An artifact container is an artifact that can contain other artifacts. Support for different 'file system' style container formats is also implemented. The currently supported underlying container formats supported are + +* Zip +* Riff + * Riff without compression + * Riff with deflate + * Riff with LZ4 compression + +Additionally the mechanisms already implemented support + +* The OS filesystem +* A virtual file system +* A 'chroot' of the file system (using RelativeFileSystem) + +In order to access a file system via artifact, is as simple as adding a modification to the default handler to load the container, and to implement `expandChildren`, which will allow traversal of the container. In general this works in a 'lazy' manner. Children are not expanded, unless requested, and files not decompressed unless required. The system also provides a caching mechanism such that a representation, such as uncompressed blob, can be associated with the artifact. + +Very little code is needed to support this behavior because the IExtFileArtifactRepresentation and the use of the ISlangFileSystemExt interface, mean it can work using the existing mechanisms. + +It is a desired feature of the container format that it can be represented as 'file system', and have the option of being human readable where appropriate. Doing so allows + +* Third party to develop tools/formats that suit their specific purposes +* Allows different containers to used +* Is generally simple to understand +* Allows editing and manipulating of contents using pre-existing extensive and cross platform tools +* Is a simple basis + +This documents is, at least in part, about how to structure the file system to represent a 'shader cache' like scenario. + +Incorporating the 'shader container' into the Artifact system will require a suitable Payload type. It may be acceptable to use `ArtifactPayload::CompileResults`. The IArtifactHandler will need to know how to interpret the contents. This will need to occur lazily at the `expandChildren` level. This will create IArtifacts for the children that some aspects are lazily evaluated, and others are interpreted at the expansion. For example setting up the ArtifactDesc will need to happen at expansion. + +Background +========== + +The following long section provides background discussion on a variety of topics. Jump to the [Proposed Approach](#proposed-approach) to describe what is actually being suggested in conclusion. + +To enumerate the major challenges + +* How to generate a key for the runtime scenario +* How to produce keys for the persistent scenario - implies user control, and human readability +* How to represent compilation in a composable 'nameable' way +* How to produce options from a named combination + +The mechanism for producing keys in the runtime scenario could be used to check if an entry in the cache is out of date. + +A compilation can be configured in many ways. Including + +* The source including source injection +* Pre-processor defines +* Compile options - optimization, debug information, include paths, libraries +* Specialization types and values +* Target and target specific features API/tools/operating system +* Specific version of slang and/or downstream compilers +* Pipeline aspects +* Slang components + +In general we probably don't want to use the combination of source and/or the above options as a 'key'. Such a key would be hard and slow to produce. It would not be something that could be created and used easily by an application. Moreover it is commonly useful to be able to name results such that the actual products can be changed and have things still work. + +Background: Hashing source +========================== + +Hashing source is something that is needed for runtime cache scenario, as it is necessary to generate a key purely from 'input' which source is part of. It can also be used in the persistent scenario, in order to validate if everything is in sync. That sync checking might perhaps only be performed when source and other resources are available. + +The fastest/simplest way to hash source, is to take the blob and hash that. Unfortunately there are several issues + +* Ignores dependencies - if this source includes another file the hash will also need to depend on that transitively +* Hash changes with line end character encoding +* Hash is sensitivve to white space changes in general + +A way to work around whitespace issues would be to use a tokenizer, or a 'simplified' tokenizer that only handles the necessary special cases. An example special case would be white space in a string is always important. Such a solution does not require an AST or rely on a specific tokenizer. A hash could be made of concatenation of all of the lexemes with white space inserted between. + +Another approach would be to hash each "token" as produced. Doing so doesn't require memory allocation for the concatenation. You could special case short strings or single chars, and hash longer strings. + +## Dependencies + +Its not enough to rely on hashing of input source, because `#include` or other resource references, such as modules or libraries may be involved. + +If we are are relying on dependencies specified at least in part by `#include`, it implies the preprocessor be executed. This could be used for other languages such as C/C++. Some care would need to be taken because *some* includes will probably not be located by our preprocessor, such as system include paths in C++. For the purpose of hashing, an implementation could ignore `#includes` that cannot be resolved. This may work for some scenarios - but doesn't work in general because symbols defined in unfound includes might cause other includes. Thus this could lead to other dependencies not being found, or being assumed when they weren't possible. + +In practice whilst not being perfect it may work well enough to be broadly usable. + +## AST + +A hash could be performed via the AST. This assumes + +1) You can produce an AST for the input source - this is not generally true as source could be CUDA, C++, etc +2) The AST would have to be produced post preprocessing - because prior to preprocessing it may not be valid source +3) If 3rd parties are supposed to be able to produce a hash it requires their implementing a Slang lexer/parser in general +4) Depending on how the AST is used, may not be stable between versions + +Another disadvantage around using the AST is that it requires the extra work and space for parsing. + +Using the AST does allow using pre-existing Slang code. It is probably more resilient to structure changes. It would also provide slang specific information more simply - such as imports. + +## Slang lexer + +If we wanted to use Slang lexer it would imply the hash process would + +1) Load the file +2) Lex the file +3) Preprocess the file (to get dependencies). Throw these tokens away (we want the hash of just the source) +4) Hash the original lex of the files tokens +5) Dependencies would require hashing and combining + +For hashing Slang language and probably HLSL we can use the Slang preprocessor tokenizer, and hash the tokens (actually probably just the token text). + +Using the Slang lexer/preprocessor may work reasonably for other languages such as C++/C/HLSL/GLSL. It does imply a reliance on a fairly large amount of slang source. + +## Simplified Lexer + +We may want to use some simple lexer. A problem with using a lexer at all is that it adds a great amount of complexity to a stand alone implementation. The simplified lexer would + +* Simplify white space - much we can strip +* Honor string representations (we can't strip whitespace) +* Honor identifiers +* We may want some special cases around operators and the like +* Honor `#include` (but ignore preprocessor behavior in general) +* Ignore comments + +We need to handle `#include` such that we have dependencies. This can lead to dependencies that aren't required in actual compilation. + +We need to honor some language specific features - such as say `import` in Slang. + +Such an implementation would be significantly simpler, and more broadly applicable than Slang lexer/parser etc. Writing an implementation would determine how complex - but would seem to be at a minimum 100s of lines of code. + +We can provide source for an implementation. We could also provide a shared library that made the functionality available via COM interface. This may help many usage scenarios, but we would want to limit the complexity as much as possible. + +## Generated Source + +Generated source can be part of a hash if the source is available. As touched on there are scenarios where generated source may not be available. + +We could side step the issues around source generation if we push that problem back onto users. If they are using code generation, the system could require providing a string that uniquely identifies the generation that is being used. This perhaps being a requirement for a persistent cache. For a temporary runtime cache, we can allow hash generation from source. + +Background: Hashing Stability +============================= + +Ideally a hashing mechanism can be resilient to unimportant changes. The previous section described some approaches for changes in source. The other area of significant complexity is around options. If options are defined as JSON (or some other 'bag of values') hashing can be performed relatively easily with a few rules. If the representation is such that if a value is not set, the default is used, it is resilient to changes of options that are explicitly set. + +When the hashing is on some native representation this isn't quite so simple, as a typical has function will include all fields. A field value, default or not will alter the hash. Therefore adding or removing a field will necessarily change the hash. + +One way around this would be to use a hashing regime that only altered the hash if the values are not default. + +```C++ + +Hash calcHash(const Options& options) +{ + const Options defaultOptions; + + Hash = ...; + if (option.someOption != defaultOption.someValue) + { + hash = combineHash(hash, option.someOption.getHash()); + } + // ... +} +``` + +This could perhaps be simplified with some macro magic. + +``` + +// Another + +struct HashCalculator +{ + template + void hashIfDifferent(const field& f, const T& defaultValue) + { + if (value != defaultValue) + { + hash = combineHash(hash, value.getHash()); + } + } + + const T defaultValue; + const T* value; + Hash hash; +}; + +Hash calcHash(const Options& options) +{ + HashCalculator calc; + const Options defaultOptions; + + calc.hashIfDifferent(options.someOption, defaultOptions.someOption); + // ... + Hash = ...; + +} +``` + +This is a little more clumsy, but if we wanted to use a final native representation, it is workable. + +Note that the ordering of hashing is also important for stability. + +Background: Key Naming +====================== + +The container could be seen as a glorified key value store, with the key identifying a kernel and associated data. + +Much of the difficulty here is how to define the key. If it's a combination of the 'inputs' it would be huge and complicated. If it's a hash, then it can be short, but not human readable, and without considerable care not stable to small or irrelevant changes. + +For a runtime cache type scenario, the instability and lack of human readability of the key probably doesn't matter too much. It probably is a consideration how slow and complicated it is to produce the key. + +For any cache that is persistent how naming occurs probably is important. Because + +* Our 'options' aren't going to make much sense with other compilers (if we want the standard to be more broadly applicable) +* The options we have will not remain static +* Having an indirection is useful from an application development and shipping perspective +* That the *name* perhaps doesn't always have to indicate every aspect of a compilation from the point of view of the application + +One idea touched on in this document is to move 'naming' into a user space problem. That compilations are defined by the combination of 'named' options. In order to produce a shader cache name we have a concatination of names. The order can be user specified. Order could also break down into "directory" hierarchy as necessary. + +Some options will need to be part of some order. This is perhaps all a little abstract so as an example + +```JSON +{ + // Configuration + + "configuration" : { + + "debug" : { + "group" : "configuration", + "optimization" : "0", + "debug-info" : true, + "defines" : [ "-DDEBUG=1" ] + }, + "release" : { + "optimization" : "2", + "debug-info" : true, + "defines" : [ "-DRELEASE=1" ] + }, + "full-release" : { + "optimization" : "3", + "debug-info" : false, + "defines" : [ "-DRELEASE=1", "-DFULL_RELEASE=1" ] + } + }, + + // Target + "target" : { + "vk" : { + } + "d3d12" : { + } + + "cpu" : { + } + }, + + // Stage + "stage" : { + "compute" : { + }, + }, + + combinations : [ + { + key : [ "vk", "compute", ["release", "full-release"] ], + options : + { + "optimization" : 1 + } + } + ] +} +``` + +The combination in this manner doesn't quite work, because some combinations may imply different options. The "combinations" section tries to address this by providing a way to 'override' behavior. This could of course be achieved with a call back mechanism. We may also want to have options that don't appear in the key, allowing 'overriding' behavior without needing a 'combinations' section. The implication is that when used in the application when looking up only the 'named' configuration is needed. + +This whole mechanism provides a way of specifying a compilation by a series of names, that can produce a unique human readable key. It is under user control, but the mechanism on how the combination takes place is at least as a default defined within an implementation. + +It may be necessary to define options by tool chain. Doing so would mean the names can group together what might be quite different options on different compilers. Having options defined in JSON means that the mechanisms described here can be used for other tooling. If the desire is to have some more broadly applicable 'shader cache' representation this is desirable. + +If it is necessary obfuscate the contents, it would be possible to put the human readable key though a hash, and then the hash can be used for lookup. + +## Container Location + +We could consider the contents of the container as 'flat' with files for each of the keys. There could be several files related to a key if we use file association mechanism (as opposed to a single manifest). + +Whilst this works it probably isn't particularly great from an organizational point of view. It might be more convenient if we can use the directory hierarchy if at least optionally. For example putting all the kernels for a target together... + +``` +/target/name-entryPoint +``` + +Or + +``` +/target/name/entryPoint-generated-hash +``` + +Where 'generated-hash' was the hash for generated source code. + +Perhaps this information would be configured in a JSON file for the repository. + +What happens if we want to obfuscate? We could make the whole path obfuscated. We could in the configuration describe which parts of the name will be obfuscated, such that it's okay to see there are different 'target' names. + +## Default names + +When originally discussed, the idea was that all options can be named, and thus any combination is just a combination of names. That combination produces the key. + +Whilst this works, it might make sense to allow 'meta' names for common types. The things that will typically change for a fixed set of options would be + +* The input translation unit source +* The target (in the Slang sense) +* The entryPoint/stage (can we say the entry point name implies the stage?) + +We could have pseudo names for these commonly changed values. If there are multiple input source files for a translation unit, we could key on the first. + +We could also have some 'names' that are built in. For example default configuration names such as 'debug' and 'release'. They can be changed in part of configuration but have some default meaning. That options can perhaps override the defaults. + +Using the pseudo name idea might mean it is possible to produce reasonable default names. Moreover we can still use the hashing mechanism to either report a validation issue, or trigger recompilation when everything needed to do as much is available. + +## Target + +A target can be quite a complicated thing to represent. Artifact has + +* 'Kind' - executable, library, object code, shared library +* 'Payload' - SPIR-V, DIXL, DXBC, Host code, Universal, x86_64 etc... + * Version +* 'Style' - Kernel, host, unknown + +This doesn't take into account a specific 'platform', where that could vary a kernel depending on the specific features of the platform. There are different versions of SPIR-V and there are different extensions. + +This doesn't cover the breadth though because for CPU targets there is additionally + +* Operating system - including operating system version +* Tool chain - Compiler + +Making this part of the filename could lead to very long filenames. The more detailed information could be made available in JSON associated files. + +This section doesn't provide a specific plan on how to encapsulate the subtlety around a 'target'. Again how this is named is probably something that is controllable in user space, but there are some reasonable defaults when it is not defined. + +Background: Describing Options +============================== + +We need some way to describe options for compilation. The most 'obvious' way would be something like the IDownstreamCompiler interface and associated types + +```C++ +struct Options +{ + Includes ...; + Optimizations ...; + Miscellaneous ...; + Source* source[]; + EntryPoint ... ; + ... + CompilerSpecific options; +} + +ICompiler +{ + Result compile(const Options* options, IArtifact** outArtifact); +} +``` + +For this to work we need a mapping from 'options' to the cached result (if there is one). There are problem around this, because + +* Items may be specified multiple times +* Ideally the hash would or at least could remain stable with updated to options +* Also ideally the user might want control over what constitutes a new version/key +* Calculating a hash is fairly complicated, and would need to take into account ordering + +Another option might be to split common options from options that are likely to be modified per compilation. For example + +``` +struct Options +{ + const char* name; ///< Human readable name + Includes ...; + Optimizations ...; + Miscellaneous ...; + Source* source[]; + ... + CompilerSpecific options; +} + +struct CompileOptions +{ + Stage stage ...; + SpecializationArgs ...; + EntryPoint ...; +}; + +ICompiler +{ + Result createOptions(Options* options, IOptions* outOptions); + + Result compile(IOptions* options, const CompileOptions* compileOptions, IArtifact** outArtifact); +} +``` + +Having the split greatly simplifies the key production, because we can use the unique human name, and the very much simpler values of CompileOptions to produce a key. + +This specifying of options in this way is tied fairly tightly to the Slang API. We can generalize the named options by allowing more than one named option set. + +## Bag of Named Options + +Perhaps identification is something that is largely in user space for the persistent scenario. You could imagine a bag of 'options', that are typically named. Then the output name is the concatenation of the names. If an option set isn't named it doesn't get included. Perhaps the order of the naming defines the precedence. + +This 'bag of options' would need some way to know the order the names would be combined. This could be achieved with another parameter or option that describes name ordering. Defining the ordering could be achieved if different types of options are grouped, by specifying the group. The ordering would only be significant for named items that will be concatenated. The ordering of the options could define the order of precedence of application. + +Problems: + +How to combine all of these options to compile? +How to define what options are set? Working at the level of a struct doesn't work if you want to override a single option. +The grouping - how does it actually work? It might require specifying what group a set of options is in. + +An advantage to this approach is that policy of how naming works as a user space problem. It is also powerful in that it allows control on compilation that has some independence from the name. + +We could have some options that are named, but do not appear as part of the name/path within the container. The purpose of this is to allow customization of a compilation, without that customization necessarily appearing withing the application code. The container could store group of named options that is used, such that it is possible to recreate the compilation or perhaps to detect there is a difference. + +### JSON options + +One way of dealing with the 'bag of options' issue would be to just make the runtime json options representation, describe options. Merging JSON at a most basic level is straight forward. For certain options it may make sense to have them describe adding, merging or replacing. We could add this control via adding a key prefix. + +```JSON +{ + "includePaths" : ["somePath", "another/path"], + "someValue" : 10, + "someEnum" : enumValue, + "someFlags" : 12 +} +``` + +As an example + +```JSON +{ + "+includePaths" : ["yet/another"], + "intValue" : 20, + "-someValue" : null, + "+someFlags" : 1 +} +``` + +When merged produces + +```JSON +{ + "includePaths" : ["somePath", "another/path", "yet/another"], + "someEnum" : enumValue, + "someFlags" : 13, + "intValue" : 20 +} +``` + +It's perhaps also worth pointing out that using JSON as the representation provides a level of compatibility. Things that are not understood can be ignored. It is human readable and understandable. We only need to convert the final JSON into the options that are then finally processed. + +One nice property of a JSON representation is that it is potentially the same for processing and hashing. + +### Producing a hash from JSON options + +One approach would be to just hash the JSON if that is the representation. We might want a pass to filter out to just known fields and perhaps some other sanity processing. + +* Filtering +* Ordering - the order of fields is generally not the order we want to combine. One option would be to order by key in alphabetical order. +* Handling values that can have multiple representations (if we allow an enum as int or text, we need to hash with one or ther other) +* Duplicate handling + +Alternatively the JSON could be converted into a native representation and that hashed. The problem with this is that without a lot of care, the hash will not be stable with respect to small changes in the native representation. + +Another advantage of using JSON for hash production, is that it is something that could be performed fairly easily in user space. + +Two issues remain significant with this approach + +* Filtering - how? +* Handling multiple representations for values + +Filtering is not trivial - it's not a question of just specifying what fields are valid, because doing so requires context. In essence it is necessary to describe types, and then describe where in a hierarchy a type is used. + +I guess this could be achieved with... JSON. For example + +``` +{ + "types": + { + "SomeType": + { + "kind" : "struct", + "derivesFrom": "..." + fields: { + [ "name", "type", "default"] + } + }, + "MainOptions": + { + "..." + } + }, + "structure" : + { + "MainOptions" + } +} +``` + +When traversing we use the 'structure' to work out where a type is used. + +This is workable, but adds additional significant complexity. + +The issue around different representations could also use the information in the description to convert into some canonical form. + +The structure could potentially generated via reflection information. + +## Native bag of options + +Options could be represented via an internal struct on which a hash can be performed. + +Input can be described as "deltas" to the current options. The final options is the combination of all the deltas - which would produce the final options structure for use. The hash of all of the options is the hash of the final structure. + +How are the deltas described? + +The in memory representation is not trivial in that if we want to add a struct to a list we would need a way to describe this. + +Whilst in the runtime the 'field' could be uniquely identified an offset, within a file format representation it would need to be by something that works across targets, and resistant to change in contents. + +## Slangs Component System + +Slang has a component system that can be used for combining options to produce a compilation. An argument can be made that it should be part of the hashing representation, as it is part of compilation. + +If combination is at the level of components, then as long as components are serializable, we can represent a compilation by a collection of components. Has several derived interfaces... + +* IEntryPoint +* ITypeConformance +* IModule + +Can be constructed into composites, through `createCompositeComponentType`, which describes aspects of the combination takes place. + +If the components were serializable (as say as JSON), we could describe a compilation as combination of components. If components are named, a concatenation of names could name a compilation. + +It doesn't appear as if there is a way to more finely control the application of component types. For example if there was a desire to change the optimization option, it would appear to remain part of the ICompileRequest (it's not part of a component). This implies this mechanism as it stands whilst allowing composition, doesn't provide the more nuanced composition. Additional component types could perhaps be added which would add such control. + +Perhaps having components is not necessary as part of the representation, as 'component' system is a mechanism for achieving a 'bag of options' and so we can get the same effect by using that mechanism without components. + +Background: Describing Options +============================== + +The 'naming' options idea implies that options and ways of combining options can be stored within the configuration for a container. Perhaps there is additionally a runtime API that allows creation of deltas. + +## 'Bag of named options' + +Perhaps identification is something that is largely in user space for the persistent scenario. You could imagine a bag of 'options', that are typically named. Then the output name is the concatenation of the names. If an option set isn't named it doesn't get included. Perhaps the order of the naming defines the precedence. + +This 'bag of options' would need some way to know the order the names would be combined. This could be achieved with another parameter or option that describes name ordering. Defining the ordering could be achieved if different types of options are grouped, by specifying the group. The ordering would only be significant for named items that will be concatenated. The ordering of the options could define the order of precedence of application. + +Problems: + +How to combine all of these options to compile? +How to define what options are set? Working at the level of a struct doesn't work if you want to override a single option. +The grouping - how does it actually work? It might require specifying what group a set of options is in. + +An advantage to this approach is that policy of how naming works as a user space problem. It is also powerful in that it allows control on compilation that has some independence from the name. + +### JSON options + +One way of dealing with the 'bag of options' issue would be to just make the runtime JSON options representation, describe options. Merging JSON at a most basic level is straight forward. For certain options it may make sense to have them describe adding, merging or replacing. We could add this control via adding a key prefix. + +```JSON +{ + "includePaths" : ["somePath", "another/path"], + "someValue" : 10, + "someEnum" : enumValue, + "someFlags" : 12 +} +``` + +As an example + +```JSON +{ + "+includePaths" : ["yet/another"], + "intValue" : 20, + "-someValue" : null, + "+someFlags" : 1 +} +``` + +When merged produces + +```JSON +{ + "includePaths" : ["somePath", "another/path", "yet/another"], + "someEnum" : enumValue, + "someFlags" : 13, + "intValue" : 20 +} +``` + +It's perhaps also worth pointing out that using JSON as the representation provides a level of compatibility. Things that are not understood can be ignored. It is human readable and understandable. We only need to convert the final JSON into the options that are then finally processed. + +One nice property of a JSON representation is that it is potentially the same for processing and hashing. + +### Producing a hash from JSON options + +One approach would be to just hash the JSON if that is the representation. We might want a pass to filter out to just known fields and perhaps some other sanity processing. + +* Filtering +* Ordering - the order of fields is generally not the order we want to combine. One option would be to order by key in alphabetical order. +* Handling values that can have multiple representations (if we allow an enum as int or text, we need to hash with one or ther other) +* Duplicate handling + +Alternatively the JSON could be converted into a native representation and that hashed. The problem with this is that without a lot of care, the hash will not be stable with respect to small changes in the native representation. + +Another advantage of using JSON for hash production, is that it is something that could be performed fairly easily in user space. + +Two issues remain significant with this approach + +* Filtering - how? +* Handling multiple representations for values + +Filtering is not trivial - it's not a question of just specifying what fields are valid, because doing so requires context. In essence it is necessary to describe types, and then describe where in a hierarchy a type is used. + +I guess this could be achieved with... JSON. For example + +``` +{ + "types": { + "SomeType": + { + "kind" : "struct", + "derivesFrom": "..." + fields: { + [ "name", "type", "default"] + } + }, + "MainOptions": + { + "..." + } + }, + "structure" : + { + "MainOptions" + } +} +``` + +When traversing we use the 'structure' to work out where a type is used. + +This is workable, but adds additional significant complexity. + +The issue around different representations could also use the information in the description to convert into some canonical form. + +The structure could potentially generated via reflection information. + +## Native bag of options + +Options could be represented via an internal struct on which a hash can be performed. + +Input can be described as "deltas" to the current options. The final options is the combination of all the deltas - which would produce the final options structure for use. The hash of all of the options is the hash of the final structure. + +How are the deltas described? + +The in memory representation is not trivial in that if we want to add a struct to a list we would need a way to describe this. + +Whilst in the runtime the 'field' could be uniquely identified an offset, within a file format representation it would need to be by something that works across targets, and resistant to change in contents. That implies it should be a name. + +If we ensure that all types involved in options as JSON serializable via reflection, this does provide a way for code to traffic between and manipulate the native types. + +How do we add a structure to a list? + +```JSON +{ + "+listField", { "structField" : 10, "anotherField" : 20 } +} +``` + +The problem perhaps is how to implement this in native code? It looks like it is workable with the functionality already available in RttiUtil. + +## Slangs Component System + +Slang has a component system that can be used for combining options to produce a compilation. An argument can be made that it should be part of the hashing representation, as it is part of compilation. + +If combination is at the level of components, then as long as components are serializable, we can represent a compilation by a collection of components. Has several derived interfaces... + +* IEntryPoint +* ITypeConformance +* IModule + +Can be constructed into composites, through `createCompositeComponentType`, which describes aspects of the combination takes place. + +If the components were serializable (as say as JSON), we could describe a compilation as combination of components. If components are named, a concatenation of names could name a compilation. + +It doesn't appear as if there is a way to more finely control the application of component types. For example if there was a desire to change the optimization option, it would appear to remain part of the ICompileRequest (it's not part of a component). This implies this mechanism as it stands whilst allowing composition, doesn't provide the more nuanced composition. Additional component types could perhaps be added which would add such control. + +Perhaps having components is not necessary as part of the representation, as 'component' system is a mechanism for achieving a 'bag of options' and so we can get the same effect by using that mechanism without components. + +Discussion: Container +===================== + +## Manifest or association + +A typical container will contain kernels - in effect blobs. The blobs themselves, or the blob names are not going to be sufficient to express the amount of information that is necessary to meet the goals laid out at the start of this document. Some extra information may be user supplied. Some extra information might be user based to know how to classify different kernels. Therefore it is necessary to have some system to handle this metadata. + +As previously discussed the underlying container format is a file system. Some limited information could be infered from the filename. For example a .spv extension file is probably SPIR-V blob. For more rich meta data describing a kernel something more is needed. Two possible approaches could be to have a 'manifest' that described the contents of the container. Another approach would to have a file associated with the kernel that describes it's contents. + +Single Manifest Pros + +* Single file describes contents +* Probably faster to load and use +* Reduces the amount of extra files +* Everything describing how the contents is to be interpreted is all in one place + +Single Manifest Cons + +* Not possible to easily add and remove contents - requires editing of the manifest, or tooling + * Extra tooling specialized tooling was deemed undesirable in original problem description +* Manifest could easily get out of sync with the contents + +Associated Files Pros + +* Simple +* Can use normal file system tooling to manipulate +* The contents of the container is implied by the contents of the file system + * Easier to keep in sync + +Associated Files Cons + +* Requires traversal of the container 'file system' to find the contents +* Might mean a more 'loose' association between results + +Another possible way of doing the association is via a directory structure. The directory might contain the 'manifest' for that directory. + +Given that we want the format to represent a file system, and that we would want it to be easy and intuitive how to manipulate the representation, using a single manifest is probably ruled out. It remains to be seen which is preferable in practice, but it seems likely that using 'associated files' is probably the way to go. + +## How to represent data + +As previously discussed, unless there is a very compelling reason not to we want to use representations that are open standards and easy to use. We also need such representations to be resilient to changes. It is important that file formats can be human readable or easily changeable into something that is human readable. For these reasons, JSON seems to be a good option for our main 'meta data' representation. Additionally Slang already has a JSON system. + +If it was necessary to have meta data stored in a more compressed format we could consider also supporting [BSON](https://en.wikipedia.org/wiki/BSON). Conversion between BSON and JSON can be made quickly and simply. BSON is a well known and used standard. + +Discussion: Container Layout +============================ + +We probably want + +* Global configuration information - describe how names map to contents +* Configuration that is compiler specific + * The format could support configuration for different compilers +* Use 'associated' file style for additional information to a result + +``` +config +config/global.json +config/slang.json +config/dxc.json +source/ +source/some-source.slang +source/some-header.h +``` + +The `source` path holds all the unique source used during a compilation. This is the 'deduped' representation. Any include hierarchy is lost. Names are generated such that they remain the same as the original where possible, but are made unique if not. The 'dependency' file for a compilation specifies how the source as included maps to the source held in the source directory. The source held in the repository like this provides a way to repeat compilations from source, but isn't the same as the source hierarchy for compilation and is typically a subset. + +`config/global.json` holds configuration that applies to the whole of the container. In particular how names map to the container contents - say the use of directories, or naming concat order. +`config/slang.json` holds how names map to option configuration that is specific to Slang + +We may want to have some config that applies to all different compilers. + +We may want to use the 'name' mechanism for some options, but commonly changing items such as the translation unit source name, entry point name can be passed directly (and used as part of the name). + +Let's say we have a config that consists of `target`, `configuration`. And we use the source name and entry point directly. We could have a configuration that expressed this as a location + +```JSON +{ + "keyPath" : "$(target)/$(filename)/$(entry-point)-$(configuration)" +} +``` + +Lets say we compile `thing.slang`, with entry point 'computeMain' and options + +``` +target: vk +configuration: release +``` + +We end up with + +``` +vk/thing/computeMain-release.spv +vk/thing/computeMain-release.spv-info.json +vk/thing/computeMain-release.spv-diagnostics.json +vk/thing/computeMain-release.spv-layout.json +vk/thing/computeMain-release.spv-dependency.json +``` + +`-info.json` holds the detailed information about what is in spv to identify the artifact, but also for the 'system' in general. Including + +`-dependency.json` is a mapping of source 'names' as part of compilation to the file + +* Artifact type +* The combination of options (which might be *more* than in the path) +* Hashes/other extra information + +Other items associated with the main 'artifact' - typically stored as 'associated' in an artifact, are optional and could be deleted. + +Having an extension, on the associated types is perhaps not necessary. Doing so makes it more clear what items are from a usability point of view. If this data can be represented in multiple ways - say JSON and BSON it also makes clear which it is. + +Discussion: Interface +===================== + +There are perhaps two ends of the spectrum of how an interface might work. One one end would be the interface is a 'slang like' compiler interface, with the the most extreme version being it *is* the Slang compiler interface. Having this interface like this means + +* There is direct access to all options +* If your application already uses the Slang interface, it can just be switched out for the cache +* Has full knowledge of the compilation - making identification unambiguous, and trivially allowing fallback to actually doing a compilation +* Trivially supports more unusual aspects of the API such as the component system +* There is no (or very little) new API, the shader container API *is* the Slang API + +It also means + +* The API has a very large surface area +* It works at the detail of the API +* Does not provide an application level indirection to some more meaningful naming/identification +* It is tied to the Slang compiler - so can't be seen as an interface to 'shader container's more generally +* Naming will almost certainly need to include a hash +* The hash will be hard to produce independently (it will be hard to calculate just anyway) + +More significantly + +* Higher requirement for source + * Could store hashes of source seen + * If there is source injection does this even make sense? + * Could store modules as Slang IR +* For generated source it requires the source +* All source does in general need to be hashed - as paths do not indicate uniqueness +* How is this obfuscated? The amount of information needed is *all of the settings*. + +At the other end of the spectrum the interface could be akin to passing in a set of user configurable parameter "names", that identify the input 'options'. The most extreme form might look something like.... + +``` +class IShaderContainer +{ + Result getArtifact(const char*const* configNames, Count configNamesCount, const Options& options, IArtifact** outArtifact); + Result getOrCreateArtifact(const char*const* configNames, Count configNamesCount, const Options& options, IArtifact** outArtifact); +}; +``` + +Q: Perhaps we don't return an Artifact, because the IArtifact interface is not simple enough. Maybe it returns a blob and an ArtifactDesc? +Q: We could simplify the IArtifact interface by moving to IArtifactContainer. Perhaps we should do this just for this reason? +Q: Is there a way to access other information - diagnostics for example? With IArtifact that can be returned as associated data. We don't want to create by default probably. +Q: If we wanted to associate 'user data' with a result how do we do that? It could just be JSON stored in the `-info`? +Q: We could have a JSON like interface for arbitrary data? + +The combination of the 'configNames' produce the key/paths within the container. + +It would probably be desirable to be able to create 'configNames' through an API. This would have to be Slang specific, and not part of this interface. The config system could be passed into the construction of the container. Doing so might contain all the information to map names to how to compile something. + +This interface may be a little too abstract, and perhaps should have parameters for common types of controls. + +As previously touched on it may be useful to pass in configuration that is *not* part of the key name to override compilation behavior. + +This style means + +* Naming is trivial +* Hashing is often not necessary +* Issues such as 'generated source' are pushed to user space +* Is user configurable +* Main interface is very simple and small +* Can be used with other compilers - because the interface is not tied to Slang in any way +* Implies the container format itself can be used trivially +* Human/application centered +* Hashing of source/options is still possible, for a variety of purposes, but is not a *requirement* as it doesn't identify a compilation. + * Meaning a simpler/less stable hashing might be fine + +With this style is it implied that the identification of a unique combination is a user space problem. For example that the source is static in general, and if not generated source identification is a user space problem. It's perhaps important to note that mechanisms previously discussed - such as hashing the source can still be useful and used. The hashing of source could be used to identify in a development environment that a recompilation is required. Or an edit of source could be made, and a single command could update all contents that is applicable automatically. These are more advanced features, and are not necessary for a user space implementation, which typically do not require the capability. + +More problematically + +* Doesn't provide a runtime cache that 'just works' for example just using the slang API +* Needs to provide a way given a combination of config names to produce the appropriate settings + * If it is just a delivery mechanism this isn't a requirement +* Probably needs both an API and 'config' mechanisms to describe options +* The indirection may lose some control + +All things considered, based on the goals of the effort it seems to make more sense to have an interface that is in the named config style. Because + +* It allows trivial 3rd party implementation +* It works with other compilers - (important if it's to work as some kind of standard) +* Provides an easy to understand mapping from input to the contents of the cache +* Can use more advanced features (like source hashing) if desired + +How config options are described or combined may be somewhat complicated, but is not necessary to use the system, and allows different compilers to implement however is appropriate. + +Discussion : Deduping source +============================ + +When compiling shaders, typically much of the source is shared. Unfortunately it is not generally possible to just save 'used source', because some source can be generated on demand. One way this is already performed by users is to use a specialized include handler, that will inject the necessary code. + +It is not generally possible therefore to identify source by path, or unique identity (as used by the slang file system interface). + +It is also the case that compilations can be performed where the source is passed by contents, and the name is not set, or not unique. + +The `slang-repro` system already handles these cases, and outputs a map from the input path to the potentially 'uniquified' name within the repro. + +You could imagine a container holding a folder of source that is shared between all the kernels. In general it would additionally require a map of each kernel that would map names to uniqified files. + +In the `slang-repro` mechanism the source is actually stored in a 'flat' manner, with the actual looked up paths stored within a map for the compilation. It would be preferable if the source could be stored in a hierarchy similar to the file system it originates. This would be possible for source that are on a file system, but would in general lead to deeper and more complex hierarchies contained in container. + +Including source, provides a way to distribute a 'compilation' much like the `slang-repro` file. It may also be useful such that a shader could be recompiled on a target. This could be for many reasons - allowing support for future platforms, allowing recompilation to improve performance or allowing compilation to happen on client machines for rare scenarios on demand. + +We may want to have tooling such that directories of source can be specified and are added to the deduplicated source library. + +We may also want to have configuration information that describes how the contents maps to search paths. It might be useful to have only the differences for lookup stored for a compilation, and some or perhaps multiple configuration files that describe the common cases. + +Discussion : Artifact With Runtime Interface +============================================ + +It should be noted *by design* `IArtifactContainer`s children is *not* a mechanism that automatically updates some underlying representation, such as files on the file system. Once a IArtifactContainer has been expanded, it allows for manipulation of the children (for example adding and removing). The typical way to produce a zip from an artifact hierarchy would be to call a function that writes it out as such. This is not something that happens incrementally. + +For an in memory caching scenario this choice works well. We can update the artifact hierarchy as needed and all is good. + +In terms of just saving off the whole container - this is also fine as we can have a function given a hierarchy that saves off the contents into a ISlangMutableFileSystem, such that it's on the file system or compressed. + +If we want the representation to be *synced* to some backing store this presents some problems. It seems this most logically happens as part of the compilation interface implementation. The Artifact system doesn't need to know anything about such issues directly. + +Once a compilation is complete, an implementation could save the result in Artifact hierarchy and write out a representation to disk from that part of the hierarchy. For some file systems doing this on demand is probably not a great idea. For example the Zip file system does not free memory directly when a file is deleted. Perhaps as part of the interface there needs to be a way to 'flush' cached data to backing store. Lastly there could be a mechanism to write out the changes (or the new archive). + +Discussion : Other +================== + +It would be a useful feature to have tooling where it is possible to + +## Generating the container + +* Generate updated kernels automatically offline + * For example when a source file changed + * For example when a config file changed + * Just force rebuilding the whole container +* Specify the combinations that are wanted in some offline manner + * Perhaps compiling in parallel + * Perhaps noticing aspects such that work can be shared + +## Obfuscation + +* Most simply could be using a hash of a 'key'. +* Or perhaps if the desire is to obfuscate at the application level, a hash of the *names* as input could be used + +## Stripping containers + +At a minimum there needs to be mechanisms to be able to strip out information that is not needed for use on a target. + +There probably also additionally needs to be a way to specify items such that names, such as type names, source names, entry point names, compile options and so forth are not trivially contained in the format, as their existence could leak sensitive information about the specifics of a compilation. + +## Indexing + +No optimized indexed scheme is described as part of this proposal. + +Indexing is probably something that happens at the 'runtime interface' level. The index can be built up using the contents of the file system. + +No attempt at an index is made as part of the container, unless later we find scenarios where this is important. Not having an index means that the file system structure itself describes it's contents, and allows manipulation of the containers contents, without manipulation of an index or some other tooling. + +## Slang IR + +It may be useful for a representation to hold `slang-ir` of a compilation. This would allow some future proofing of the representation, because it would allow support for newer versions of Slang and downstream compilers without distributing source. + +Related Work +============ + +* Shader cache system as part of gfx (https://github.com/lucy96chen/slang/tree/shader-cache) +* Lumberyard [shader cache](https://docs.aws.amazon.com/lumberyard/latest/userguide/mat-shaders-custom-dev-cache-intro.html) +* Unreal [FShaderCache](https://docs.unrealengine.com/5.0/en-US/fshadercache-in-unreal-engine/) +* Unreal [Derived Data Cache - DDC](https://docs.unrealengine.com/4.26/en-US/ProductionPipelines/DerivedDataCache/) +* Microsoft [D3DSCache](https://github.com/microsoft/DirectX-Specs/blob/master/d3d/ShaderCache.md) + +Lumberyard uses the zip format for its '.pak' format. + +Microsoft D3DSCache provides a binary keyed key-value store. + +## Gfx + +Gfx has a runtime shader cache based on `PipelineKey`, `ComponentKey` and `ShaderCache`. ShaderCache is a key value store. + +A key for a pipeline is a combination of + +``` + PipelineStateBase* pipeline; + Slang::ShortList specializationArgs; +``` + +`ShaderComponentID` can be created on the ShaderCache, from + +``` + Slang::UnownedStringSlice typeName; + Slang::ShortList specializationArgs; + Slang::HashCode hash; +``` + +For reflected types, a type name is generated if specialized. + +The shader cache can be thought of as being parameterized by the pipeline and associated specialization args. It appears to currently only support specialization types. + +Gfx does not appear to support any serialization/file representation. + + + +Proposed Approach +================= + +Based on the goals described in the introduction, the proposed approach is + +* Use a collection of named options to describe a compilation + * Requires a mechanism to provide combining + * Have some additional symbols (such as + field name prefix as described elsewhere) to describe how options should be applied +* Meaning of names can be described within configuration and through an API + * API might be compiler specific +* Some names contribute to the key, whilst others do not + * Non inclusion in key allows customization for a specific result without a key change +* Some options within a configuration can use standard names, others will require being compiler specific +* Probably easiest to use a native representation for combining + * Using the collection of names approach makes hash stability and hashes in general less important +* Use JSON/BSON as the format for configuration files + * Possible to have some options defined that *aren't* part of the key name + * The actual combination can be stored along with products, such that the combination can be recreated, or an inconsistency detected +* Use JSON to native conversion to produce native types that can then be combined +* Can have some standard ways to generate names for standard scenarios + * Such as using input source name as part of key +* Use associated files (not a global manifest), to allow easy manipulation/tooling +* Source will in general be deduped, with a compilation describing where it's source originated + * This is similar to how repro files work +* Some names will be automatically available by default +* Ideally a 'non configured' (or default configured) cache can work for common usage scenarios. + +For the runtime cache scenario this all still works. If an application wants a runtime cache that is memory based that works transparently (ie just through the use of Slang API), this is of course possible and it's output can be made compatible with the format. It will be fragile to Slang API changes, and probably not usable outside of the Slang ecosystem. + +Alternatives Considered +----------------------- + +Discussed elsewhere. + +## Issues On Github + +* Support low-overhead runtime "shader cache" lookups [#595](https://github.com/shader-slang/slang/issues/595) +* Compilation id/hash [#2050](https://github.com/shader-slang/slang/issues/2050) +* Support a simple zip-based container format [#860](https://github.com/shader-slang/slang/issues/860) + diff --git a/docs/user-guide/06-interfaces-generics.md b/docs/user-guide/06-interfaces-generics.md index 61bc43f89..df51ac950 100644 --- a/docs/user-guide/06-interfaces-generics.md +++ b/docs/user-guide/06-interfaces-generics.md @@ -104,6 +104,22 @@ Generic value parameters can also be defined using the traditional C-style synta void g1() { ... } ``` +Slang allows multiple `where` clauses, and multiple interface types in a single `where` clause: +```csharp +struct MyType + where T: IFoo, IBar + where U : IBaz +{ +} +// equivalent to: +struct MyType + where T: IFoo + where T : IBar + where U : IBaz +{ +} +``` + Supported Constructs in Interface Definitions ----------------------------------------------------- @@ -793,6 +809,61 @@ void main() See [if-let syntax](convenience-features.html#if_let-syntax) for more details. +Generic Interfaces +------------------ + +Slang allows interfaces themselves to be generic. A common use of generic interfaces is to define the `IEnumerable` type: +```csharp +interface IEnumerator +{ + This moveNext(); + bool isEnd(); + T getValue(); +} + +interface IEnumerable +{ + assoicatedtype Enumerator : IEnumerator; + Enumerator getEnumerator(); +} +``` + +You can constrain a generic type parameter to conform to a generic interface: +```csharp +void traverse(TCollection c) + where TCollection : IEnumerable +{ + ... +} +``` + + +Generic Extensions +---------------------- +You can use generic extensions to extend a generic type. For example, +```csharp +interface IFoo { void foo(); } +interface IBar { void bar(); } + +struct MyType +{ + void foo() { ... } +} + +// Extend `MyType` so it conforms to `IBar`. +extension MyType : IBar +{ + void bar() { ... } +} +// Equivalent to: +__generic +extension MyType : IBar +{ + void bar() { ... } +} +``` + + Extensions to Interfaces ----------------------------- diff --git a/docs/user-guide/09-targets.md b/docs/user-guide/09-targets.md index e1219a39d..d6aebab0c 100644 --- a/docs/user-guide/09-targets.md +++ b/docs/user-guide/09-targets.md @@ -302,14 +302,14 @@ Metal > #### Note #### > Slang support for Metal is a work in progress. -Metal is a shading language exclusive on Apple slicons. The functionality from Metal is similar to DX12 or Vulkan with more or less features. +Metal is a shading language exclusive on Apple platforms. The functionality from Metal is similar to DX12 or Vulkan with more or less features. ### Pipelines -Metal includes rasterization, compute, and ray tracing pipelines with the same set of stages as described for D3D12 above. +Metal includes vertex, fragment, task, mesh and tessellation stages for rasterization, as well as compute, and ray tracing stages. > #### Note #### -> Ray-tracing and Mesh support for Metal is a work in progress. +> Ray-tracing support for Metal is a work in progress. ### Parameter Passing diff --git a/docs/user-guide/a2-01-spirv-target-specific.md b/docs/user-guide/a2-01-spirv-target-specific.md index a1ecbfefd..e0d6fd69b 100644 --- a/docs/user-guide/a2-01-spirv-target-specific.md +++ b/docs/user-guide/a2-01-spirv-target-specific.md @@ -134,7 +134,7 @@ SPIR-V 1.5 with [SPV_EXT_shader_atomic_float16_add](https://github.com/KhronosGr ConstantBuffer, (RW/RasterizerOrdered)StructuredBuffer, (RW/RasterizerOrdered)ByteAddressBuffer ----------------------------------------------------------------------------------------------- -Each member in a `ConstantBuffer` will be emitted as `uniform` parameter. +Each member in a `ConstantBuffer` will be emitted as `uniform` parameter in a uniform block. StructuredBuffer and ByteAddressBuffer are translated to a shader storage buffer with `readonly` layout. RWStructuredBuffer and RWByteAddressBuffer are translated to a shader storage buffer with `read-write` layout. RasterizerOrderedStructuredBuffer and RasterizerOrderedByteAddressBuffer will use an extension, `SPV_EXT_fragment_shader_interlock`. @@ -156,6 +156,34 @@ It is similar to `ConstantBuffer` in HLSL, and `ParameterBlock` can include not When both ordinary data fields and resource typed fields exist in a parameter block, all ordinary data fields will be grouped together into a uniform buffer and appear as a binding 0 of the resulting descriptor set. +Push Constants +--------------------- + +By default, a `uniform` parameter defined in the parameter list of an entrypoint function is translated to a push constant in SPIRV, if the type of the parameter is ordinary data type (no resources/textures). +All `uniform` parameter defined in global scope are grouped together and placed in a default constant bbuffer. You can make a global uniform parameter laid out as a push constant by using the `[vk::push_constant]` attribute +on the uniform parameter. + +Specialization Constants +------------------------ + +You can specify a global constant to translate into a SPIRV specialization constant with the `[SpecializationConstant]` attribute. +For example: +```csharp +[SpecializationConstant] +const int myConst = 1; // Maps to a SPIRV specialization constant +``` + +By default, Slang will automatically assign `constant_id` number for specialization constants. If you wish to explicitly specify them, use `[vk::constant_id]` attribute: +```csharp +[vk::constant_id(1)] +const int myConst = 1; +``` + +Alternatively, the GLSL `layout` syntax is also supported by Slang: +```glsl +layout(constant_id = 1) const int MyConst = 1; +``` + SPIR-V specific Compiler options -------------------------------- diff --git a/docs/user-guide/toc.html b/docs/user-guide/toc.html index df1b4ba9d..77c4f16d8 100644 --- a/docs/user-guide/toc.html +++ b/docs/user-guide/toc.html @@ -83,6 +83,8 @@
  • Interface-typed Values
  • Extending a Type with Additional Interface Conformances
  • `is` and `as` Operator
    • +
  • Generic Interfaces
    • +
  • Generic Extensions
  • Extensions to Interfaces
  • Variadic Generics
  • Builtin Interfaces
    • @@ -207,6 +209,8 @@
  • Supported atomic types for each target
  • ConstantBuffer, (RW/RasterizerOrdered)StructuredBuffer, (RW/RasterizerOrdered)ByteAddressBuffer
  • ParameterBlock for SPIR-V target
    • +
  • Push Constants
    • +
  • Specialization Constants
  • SPIR-V specific Compiler options
  • SPIR-V specific Attributes
  • Multiple entry points support
    • -- cgit v1.2.3