Add better control over image formats for GLSL/SPIR-V targets (#939)

* Add better control over image formats for GLSL/SPIR-V targets

Currently Slang emits GLSL code assuming all R/W images need to have explicit formats, and thus we try to infer a format from the element type of the image.
E.g., given a `RWTexture2D<half4>` we might infer that a qualifier of `layout(rgba16f)` should be used.

This strategy has two notable shortcomings:

* Sometimes the user will want a format that doesn't match an existing HLSL type. E.g., if they want the equivalent of `layout(r11f_g11f_b10f)`, then what should they put in their `RWTexture2D<...>` to make the inference do what they need?

* Sometimes the user knows that they don't need to specify a format *at all*, because using the `GL_EXT_shader_image_load_formatted` extension, they can still perform non-atomic load/store on images with no format specified in the SPIR-V.

This change adds two features directed at these challenges.

First, we add an explicit `[format(...)]` attribute that can be used to specify an explicit image format, including ones that don't match any HLSL type.
An example of using this new attribute is:

```hlsl
[format("r11f_g11f_b10f")]
RWTexture2D<float3> myImage;
```

For simplicity in initial bring-up, the new formats all use the same naming as formats in GLSL (this should make it easy for a programmer who knows what they expect to get in the GLSL output). We can change the naming convention for formats at a later time, so long as we keep these existing names in as a compatibility feature.

Note that this is *not* given a `vk::` prefix since the attribute should signal the programmer's intent to provide an image with that format on *all* targets (although only some targets might act on that information).

Also note that the attribute takes a string (`[format("rgba8")`) instead of a bare identifier (`[format(rgba8)]`) because this is consistent with the existing convention for attributes in HLSL.

When `[format(...)]` is left off, the default compiler behavior will still be to infer a format, but this behavior can be overidden for a single image using an explicit format of `"unknown"`:

```hlsl
[format("unknown")]
RWTexture2D<float4> mysteryMachine;
```

The second new feature is that if a user knows they are coding for a GPU that supports the `"unknown"` format in all non-atomic cases, then they can opt into making that the default for images without an explicit `[format(...)]`, using the new `-default-image-format-unknown` command-line option for `slangc`.

The new test case included with this change confirms that we correctly see the explicit formats in the output GLSL and *no* formats for images without explicit `[format(...)]` when using the new command-line option. The test stresses images declared at global scope, in parameter blocks, and in entry-point parameter lists, to try and make sure that all the relevant IR passes in the compiler preserve the format information.

* fixup: missing file

1 files changed, 174 insertions, 104 deletions

diff --git a/source/slang/emit.cpp b/source/slang/emit.cpp
index 7615b0b28..1e9970b1a 100644
--- a/source/slang/emit.cpp
+++ b/source/slang/emit.cpp
@@ -5678,6 +5678,178 @@ struct EmitVisitor
         return value;
     }
 
+    void emitGLSLImageFormatModifier(
+        IRInst*         var,
+        IRTextureType*  resourceType)
+    {
+        // If the user specified a format manually, using `[format(...)]`,
+        // then we will respect that format and emit a matching `layout` modifier.
+        //
+        if(auto formatDecoration = var->findDecoration<IRFormatDecoration>())
+        {
+            auto format = formatDecoration->getFormat();
+            if(format == ImageFormat::unknown)
+            {
+                // If the user explicitly opts out of having a format, then
+                // the output shader will require the extension to support
+                // load/store from format-less images.
+                //
+                // TODO: We should have a validation somewhere in the compiler
+                // that atomic operations are only allowed on images with
+                // explicit formats (and then only on specific formats).
+                // This is really an argument that format should be part of
+                // the image *type* (with a "base type" for images with
+                // unknown format).
+                //
+                requireGLSLExtension("GL_EXT_shader_image_load_formatted");
+            }
+            else
+            {
+                // If there is an explicit format specified, then we
+                // should emit a `layout` modifier using the GLSL name
+                // for the format.
+                //
+                Emit("layout(");
+                Emit(getGLSLNameForImageFormat(format));
+                Emit(")\n");
+            }
+
+            // No matter what, if an explicit `[format(...)]` was given,
+            // then we don't need to emit anything else.
+            //
+            return;
+        }
+
+
+        // When no explicit format is specified, we need to either
+        // emit the image as having an unknown format, or else infer
+        // a format from the type.
+        //
+        // For now our default behavior is to infer (so that unmodified
+        // HLSL input is more likely to generate valid SPIR-V that
+        // runs anywhere), but we provide a flag to opt into
+        // treating images without explicit formats as having
+        // unknown format.
+        //
+        if(this->context->shared->compileRequest->useUnknownImageFormatAsDefault)
+        {
+            requireGLSLExtension("GL_EXT_shader_image_load_formatted");
+            return;
+        }
+
+        // At this point we have a resource type like `RWTexture2D<X>`
+        // and we want to infer a reasonable format from the element
+        // type `X` that was specified.
+        //
+        // E.g., if `X` is `float` then we can infer a format like `r32f`,
+        // and so forth. The catch of course is that it is possible to
+        // specify a shader parameter with a type like `RWTexture2D<float4>` but
+        // provide an image at runtime with a format like `rgba8`, so
+        // this inference is never guaranteed to give perfect results.
+        //
+        // If users don't like our inferred result, they need to use a
+        // `[format(...)]` attribute to manually specify what they want.
+        //
+        // TODO: We should consider whether we can expand the space of
+        // allowed types for `X` in `RWTexture2D<X>` to include special
+        // pseudo-types that act just like, e.g., `float4`, but come
+        // with attached/implied format information.
+        //
+        auto elementType = resourceType->getElementType();
+        Int vectorWidth = 1;
+        if(auto elementVecType = as<IRVectorType>(elementType))
+        {
+            if(auto intLitVal = as<IRIntLit>(elementVecType->getElementCount()))
+            {
+                vectorWidth = (Int) intLitVal->getValue();
+            }
+            else
+            {
+                vectorWidth = 0;
+            }
+            elementType = elementVecType->getElementType();
+        }
+        if(auto elementBasicType = as<IRBasicType>(elementType))
+        {
+            Emit("layout(");
+            switch(vectorWidth)
+            {
+            default: Emit("rgba");  break;
+
+            case 3:
+            {
+                // TODO: GLSL doesn't support 3-component formats so for now we are going to
+                // default to rgba
+                //
+                // The SPIR-V spec (https://www.khronos.org/registry/spir-v/specs/unified1/SPIRV.pdf)
+                // section 3.11 on Image Formats it does not list rgbf32.
+                //
+                // It seems SPIR-V can support having an image with an unknown-at-compile-time
+                // format, so long as the underlying API supports it. Ideally this would mean that we can
+                // just drop all these qualifiers when emitting GLSL for Vulkan targets.
+                //
+                // This raises the question of what to do more long term. For Vulkan hopefully we can just
+                // drop the layout. For OpenGL targets it would seem reasonable to have well-defined rules
+                // for inferring the format (and just document that 3-component formats map to 4-component formats,
+                // but that shouldn't matter because the API wouldn't let the user allocate those 3-component formats anyway),
+                // and add an attribute for specifying the format manually if you really want to override our
+                // inference (e.g., to specify r11fg11fb10f).
+
+                Emit("rgba");
+                //Emit("rgb");                                
+                break;
+            }
+
+            case 2:  Emit("rg");    break;
+            case 1:  Emit("r");     break;
+            }
+            switch(elementBasicType->getBaseType())
+            {
+            default:
+            case BaseType::Float:   Emit("32f");  break;
+            case BaseType::Half:    Emit("16f");  break;
+            case BaseType::UInt:    Emit("32ui"); break;
+            case BaseType::Int:     Emit("32i"); break;
+
+            // TODO: Here are formats that are available in GLSL,
+            // but that are not handled by the above cases.
+            //
+            // r11f_g11f_b10f
+            //
+            // rgba16
+            // rgb10_a2
+            // rgba8
+            // rg16
+            // rg8
+            // r16
+            // r8
+            //
+            // rgba16_snorm
+            // rgba8_snorm
+            // rg16_snorm
+            // rg8_snorm
+            // r16_snorm
+            // r8_snorm
+            //
+            // rgba16i
+            // rgba8i
+            // rg16i
+            // rg8i
+            // r16i
+            // r8i
+            //
+            // rgba16ui
+            // rgb10_a2ui
+            // rgba8ui
+            // rg16ui
+            // rg8ui
+            // r16ui
+            // r8ui
+            }
+            Emit(")\n");
+        }
+    }
+
     void emitIRVarModifiers(
         EmitContext*    ctx,
         VarLayout*      layout,
@@ -5703,6 +5875,7 @@ struct EmitVisitor
             emit(")\n");
             emit("callableDataNV\n");
         }
+
         if(varDecl->findDecoration<IRVulkanHitAttributesDecoration>())
         {
             emit("hitAttributeNV\n");
@@ -5741,110 +5914,7 @@ struct EmitVisitor
                 case SLANG_RESOURCE_ACCESS_READ_WRITE:
                 case SLANG_RESOURCE_ACCESS_RASTER_ORDERED:
                     {
-                        // We have a resource type like `RWTexture2D<X>` and when we emit
-                        // this as an image declaration in GLSL, we need to specify the
-                        // correct in-memory format for the image (e.g., `layout(rgba32f)`).
-                        //
-                        // TODO: There are modifiers in GLSL that can specify this information,
-                        // and we should support the same ones that dxc does (e.g.,
-                        // some kind of `[[vk::format(...)]]` or what-have-you.
-                        //
-                        // For now we will simply infer a reasonable format from the
-                        // element type that was specified.
-                        //
-                        auto elementType = resourceType->getElementType();
-                        Int vectorWidth = 1;
-                        if(auto elementVecType = as<IRVectorType>(elementType))
-                        {
-                            if(auto intLitVal = as<IRIntLit>(elementVecType->getElementCount()))
-                            {
-                                vectorWidth = (Int) intLitVal->getValue();
-                            }
-                            else
-                            {
-                                vectorWidth = 0;
-                            }
-                            elementType = elementVecType->getElementType();
-                        }
-                        if(auto elementBasicType = as<IRBasicType>(elementType))
-                        {
-                            Emit("layout(");
-                            switch(vectorWidth)
-                            {
-                            default: Emit("rgba");  break;
-
-                            case 3:
-                            {
-                                // TODO: GLSL doesn't support 3-component formats so for now we are going to
-                                // default to rgba
-                                //
-                                // The SPIR-V spec (https://www.khronos.org/registry/spir-v/specs/unified1/SPIRV.pdf)
-                                // section 3.11 on Image Formats it does not list rgbf32.
-                                //
-                                // It seems SPIR-V can support having an image with an unknown-at-compile-time
-                                // format, so long as the underlying API supports it. Ideally this would mean that we can
-                                // just drop all these qualifiers when emitting GLSL for Vulkan targets.
-                                //
-                                // This raises the question of what to do more long term. For Vulkan hopefully we can just
-                                // drop the layout. For OpenGL targets it would seem reasonable to have well-defined rules
-                                // for inferring the format (and just document that 3-component formats map to 4-component formats,
-                                // but that shouldn't matter because the API wouldn't let the user allocate those 3-component formats anyway),
-                                // and add an attribute for specifying the format manually if you really want to override our
-                                // inference (e.g., to specify r11fg11fb10f).
-
-                                Emit("rgba");
-                                //Emit("rgb");                                
-                                break;
-                            }
-
-                            case 2:  Emit("rg");    break;
-                            case 1:  Emit("r");     break;
-                            }
-                            switch(elementBasicType->getBaseType())
-                            {
-                            default:
-                            case BaseType::Float:   Emit("32f");  break;
-                            case BaseType::Half:    Emit("16f");  break;
-                            case BaseType::UInt:    Emit("32ui"); break;
-                            case BaseType::Int:     Emit("32i"); break;
-
-                            // TODO: Here are formats that are available in GLSL,
-                            // but that are not handled by the above cases.
-                            //
-                            // r11f_g11f_b10f
-                            //
-                            // rgba16
-                            // rgb10_a2
-                            // rgba8
-                            // rg16
-                            // rg8
-                            // r16
-                            // r8
-                            //
-                            // rgba16_snorm
-                            // rgba8_snorm
-                            // rg16_snorm
-                            // rg8_snorm
-                            // r16_snorm
-                            // r8_snorm
-                            //
-                            // rgba16i
-                            // rgba8i
-                            // rg16i
-                            // rg8i
-                            // r16i
-                            // r8i
-                            //
-                            // rgba16ui
-                            // rgb10_a2ui
-                            // rgba8ui
-                            // rg16ui
-                            // rg8ui
-                            // r16ui
-                            // r8ui
-                            }
-                            Emit(")\n");
-                        }
+                        emitGLSLImageFormatModifier(varDecl, resourceType);
                     }
                     break;