Avoid crash when bad argument given to [instance(...)] attribute (#464)

Fixes #463

Some of the attributes were failing to check for a `null` result from `checkConstantIntVal`, and so they crashed when a bad expression was used in an attribute. The particular way this had been triggered was that a user put an HLSL geometry shader in the same file with other code, using an entry point like:

```hlsl
[instance(COUNT)]
void myGeometryShader(...) {...}
```

They then defined `COUNT` as a preprocessor macro when compiling using the GS, but left it undefined otherwise. The result was that the argument to the `instance` attribute would fail to type check, and thus wouldn't count as a constant integer value, so that `checkConstantIntVal` returns `null` and results in the crash.

The workaround for the user is to always define `COUNT`, even when not compiling the GS.

The fix in the compiler is to guard against `null` in these cases and bail out of attribute checking. I also implemented logic so that `CheckIntegerConstantExpression` (which is invoked by `checkConstantIntVal`) will not produce an additional error message if the underlying expression failed to type check. In this casem the user will get an `undefined identifier: COUNT` error message, and we don't need to waste their time by also telling them that this isn't a compile-time constant expression.

1 files changed, 11 insertions, 0 deletions

diff --git a/source/slang/check.cpp b/source/slang/check.cpp
index e06578c86..b1df71428 100644
--- a/source/slang/check.cpp
+++ b/source/slang/check.cpp
@@ -1530,6 +1530,8 @@ namespace Slang
                     SLANG_ASSERT(attr->args.Count() == 1);
                     auto val = checkConstantIntVal(attr->args[0]);
 
+                    if(!val) return false;
+
                     maxVertexCountAttr->value = (int32_t)val->value;
                 }
                 else if(auto instanceAttr = attr.As<InstanceAttribute>())
@@ -1537,6 +1539,8 @@ namespace Slang
                     SLANG_ASSERT(attr->args.Count() == 1);
                     auto val = checkConstantIntVal(attr->args[0]);
 
+                    if(!val) return false;
+
                     instanceAttr->value = (int32_t)val->value;
                 }
                 else if(auto entryPointAttr = attr.As<EntryPointAttribute>())
@@ -3555,8 +3559,15 @@ namespace Slang
         // Enforce that an expression resolves to an integer constant, and get its value
         RefPtr<IntVal> CheckIntegerConstantExpression(Expr* inExpr)
         {
+            // No need to issue further errors if the expression didn't even type-check.
+            if(IsErrorExpr(inExpr)) return nullptr;
+
             // First coerce the expression to the expected type
             auto expr = Coerce(getSession()->getIntType(),inExpr);
+
+            // No need to issue further errors if the type coercion failed.
+            if(IsErrorExpr(expr)) return nullptr;
+
             auto result = TryCheckIntegerConstantExpression(expr.Ptr());
             if (!result)
             {