Prefer IntegerType over LogicalType integer matrix mul() overloads (#8426)

Integer mul(matrix, matrix) and mul(vector, matrix) are not
disambiguated between __BuiltinIntegerType and __BuiltinLogicalType,
emitting an ambiguous call compilation error.

Use the OverloadRank attribute to prefer the IntegerType overload over
the LogicalType overload.

Fixes #8424

1 files changed, 18 insertions, 0 deletions

diff --git a/tests/hlsl-intrinsic/matrix-int-mul.slang b/tests/hlsl-intrinsic/matrix-int-mul.slang
new file mode 100644
index 000000000..d78a486a9
--- /dev/null
+++ b/tests/hlsl-intrinsic/matrix-int-mul.slang
@@ -0,0 +1,18 @@
+//TEST:SIMPLE(filecheck=CHECK):-target hlsl -entry main -stage compute
+
+//CHECK-NOT: error 39999: ambiguous call to 'mul'
+
+[shader("compute")]
+void main() {
+    int2x2 matrixA = {
+        1, 2,
+        3, 4
+    };
+    int2 vecA = {
+        1, 2
+    };
+
+    int2x2 matrixB = mul(matrixA, matrixA); // M * M
+    int2   vecB    = mul(vecA,    matrixA); // V * M
+    int2   vecC    = mul(matrixA, vecA);    // M * V
+}