Avoid crash when bad argument given to [instance(...)] attribute (#464)

Fixes #463

Some of the attributes were failing to check for a `null` result from `checkConstantIntVal`, and so they crashed when a bad expression was used in an attribute. The particular way this had been triggered was that a user put an HLSL geometry shader in the same file with other code, using an entry point like:

```hlsl
[instance(COUNT)]
void myGeometryShader(...) {...}
```

They then defined `COUNT` as a preprocessor macro when compiling using the GS, but left it undefined otherwise. The result was that the argument to the `instance` attribute would fail to type check, and thus wouldn't count as a constant integer value, so that `checkConstantIntVal` returns `null` and results in the crash.

The workaround for the user is to always define `COUNT`, even when not compiling the GS.

The fix in the compiler is to guard against `null` in these cases and bail out of attribute checking. I also implemented logic so that `CheckIntegerConstantExpression` (which is invoked by `checkConstantIntVal`) will not produce an additional error message if the underlying expression failed to type check. In this casem the user will get an `undefined identifier: COUNT` error message, and we don't need to waste their time by also telling them that this isn't a compile-time constant expression.

3 files changed, 28 insertions, 0 deletions

diff --git a/source/slang/check.cpp b/source/slang/check.cpp
index e06578c86..b1df71428 100644
--- a/source/slang/check.cpp
+++ b/source/slang/check.cpp
@@ -1530,6 +1530,8 @@ namespace Slang
                     SLANG_ASSERT(attr->args.Count() == 1);
                     auto val = checkConstantIntVal(attr->args[0]);
 
+                    if(!val) return false;
+
                     maxVertexCountAttr->value = (int32_t)val->value;
                 }
                 else if(auto instanceAttr = attr.As<InstanceAttribute>())
@@ -1537,6 +1539,8 @@ namespace Slang
                     SLANG_ASSERT(attr->args.Count() == 1);
                     auto val = checkConstantIntVal(attr->args[0]);
 
+                    if(!val) return false;
+
                     instanceAttr->value = (int32_t)val->value;
                 }
                 else if(auto entryPointAttr = attr.As<EntryPointAttribute>())
@@ -3555,8 +3559,15 @@ namespace Slang
         // Enforce that an expression resolves to an integer constant, and get its value
         RefPtr<IntVal> CheckIntegerConstantExpression(Expr* inExpr)
         {
+            // No need to issue further errors if the expression didn't even type-check.
+            if(IsErrorExpr(inExpr)) return nullptr;
+
             // First coerce the expression to the expected type
             auto expr = Coerce(getSession()->getIntType(),inExpr);
+
+            // No need to issue further errors if the type coercion failed.
+            if(IsErrorExpr(expr)) return nullptr;
+
             auto result = TryCheckIntegerConstantExpression(expr.Ptr());
             if (!result)
             {
diff --git a/tests/bugs/gh-463.slang b/tests/bugs/gh-463.slang
new file mode 100644
index 000000000..95890eba5
--- /dev/null
+++ b/tests/bugs/gh-463.slang
@@ -0,0 +1,11 @@
+//TEST:SIMPLE:
+
+// undefined identifier used in `[instance(...)]` attribute
+
+struct V { float4 u : U; };
+
+[instance(COUNT)]
+void myGS(triangle V input[3], TriangleStream<V> output)
+{
+	
+}
diff --git a/tests/bugs/gh-463.slang.expected b/tests/bugs/gh-463.slang.expected
new file mode 100644
index 000000000..d40a845a3
--- /dev/null
+++ b/tests/bugs/gh-463.slang.expected
@@ -0,0 +1,6 @@
+result code = -1
+standard error = {
+tests/bugs/gh-463.slang(7): error 30015: undefined identifier 'COUNT'.
+}
+standard output = {
+}