Overhaul the preprocessor (#1849)

* Overhaul the preprocessor

The old Slang preprocessor was based on a simple mental model that tried to unify two parts of macro expansion:

* Scanning for macro invocations in a sequence of tokens

* Producing the expanded tokens for a macro expansion by substituting arguments into its body

The basic was that substitution of macro arguments into a macro definition is superficially similar to top-level macro expansion, just with an environment where the macro arguments act like `#define`s for the corresponding parameter names. That approach was "clever" and could conceivably have been extended to include a lot of advanced preprocessor features (e.g., a preprocessor-level `lambda` would be easy to support!), but it was basically impossible to make it correctly handle all the corner cases of the full C/C++ preprocessor.

The fundamental problem with the old approach was that it conflated the two parts of expansion listed above into one implementation, while the various special cases of the C/C++ preprocessor rely on treating the two cases very differently. The new approach here (which is somewhere between a refactor and a full rewrite of the preprocessor) changes things up in a few key ways:

* The abstraction still cares a lot about streams of tokens, but it now treats the top level streams (`InputFile`s) as fairly different from the lower-level streams (`InputStream`s)

* Macro expansion is handled as a dedicated type of stream that wraps another stream. This allows macro expansion to be applied to anything, and supports cases where multiple rounds of macro expansion are required by the spec.

* Macro *invocations* and the substitution of their arguments are now handled by a completely new system.

    * Macro arguments are no longer treated as if they were `#define`s

    * The macro body/definition is analyzed at definition time to detect various kinds of issues, and to derive a list of "ops" that make it easier to "play back" the definition at substitution time

* Token pasting and stringizing are now only handled in macro definitions (rather than being allowed anywhere), and their use cases are restricted to only those that make sense (e.g., you can't stringize anythign except a macro parameter, because anything else wouldn't make sense)

The key new types here are the `ExpansionInputStream` which handles scanning for macro invocations, and the `MacroInvocation` type, which handles playing back the macro body with substitutions.

The `ExpansionInputStream` is the easier of the two to understand. By refactoring it to use a single token of lookahead, the one major detail it had to deal with before (abandoning expansion of a function-like macro if the macro name was not followed by `(`) is significantly easier to manage.

The more subtle part is the `MacroInvocation` type, and most of the complexity there is around handling of token pasting, and the fact that either or both of the operands to a token paste might be empty.

Many of the test cases that exposed the problems in the preprocessor have been moved from `current-bugs` to `preprocessor` since they now work correctly.

* debugging: enable extractor command line dump

* fixup

* fixup

26 files changed, 225 insertions, 2 deletions

diff --git a/tests/preprocessor/error.slang.expected b/tests/preprocessor/error.slang.expected
index 927819780..f191f7aaa 100644
--- a/tests/preprocessor/error.slang.expected
+++ b/tests/preprocessor/error.slang.expected
@@ -1,6 +1,6 @@
 result code = -1
 standard error = {
-tests/preprocessor/error.slang(11): error 15900: #error:  This isn't valid!
+tests/preprocessor/error.slang(11): error 15900: #error: This isn't valid!
 #error This isn't valid!
  ^~~~~
 }
diff --git a/tests/preprocessor/paste-non-expansion.slang b/tests/preprocessor/paste-non-expansion.slang
new file mode 100644
index 000000000..d3487aeef
--- /dev/null
+++ b/tests/preprocessor/paste-non-expansion.slang
@@ -0,0 +1,13 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+// This os a regression test for a bug in Slang preprocessor macro expansion.
+
+#define CONCAT2(x, y) x ## y
+#define CONCAT(x, y) CONCAT2(x, y)
+
+#define SOMETHING someThing
+
+// Should be someThingElse
+CONCAT(SOMETHING, Else)
+// Should be SOMETHINGAnother, but old Slang expands to produce someThingAnother
+CONCAT2(SOMETHING, Another)
\ No newline at end of file
diff --git a/tests/preprocessor/paste-non-expansion.slang.expected b/tests/preprocessor/paste-non-expansion.slang.expected
new file mode 100644
index 000000000..b62416de7
--- /dev/null
+++ b/tests/preprocessor/paste-non-expansion.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+someThingElse SOMETHINGAnother 
+}
diff --git a/tests/preprocessor/preproc-concat-1.slang b/tests/preprocessor/preproc-concat-1.slang
new file mode 100644
index 000000000..44d6c1d61
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-1.slang
@@ -0,0 +1,12 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define CONCAT(a, b) a ## b
+
+// Correct output AB;
+// Old Slang output
+// ab;
+
+#define A a
+#define B b
+
+CONCAT(A, B);
diff --git a/tests/preprocessor/preproc-concat-1.slang.expected b/tests/preprocessor/preproc-concat-1.slang.expected
new file mode 100644
index 000000000..84e8c280f
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-1.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+AB ; 
+}
diff --git a/tests/preprocessor/preproc-concat-2.slang b/tests/preprocessor/preproc-concat-2.slang
new file mode 100644
index 000000000..b965eeaa3
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-2.slang
@@ -0,0 +1,15 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define CONCAT(a, b) a ## b
+
+#define A a
+#define B b
+
+#define A2 A
+#define B2 B
+
+// Correct output: a A2B2 b;
+// Old Slang output 
+// a ab b ;
+
+CONCAT(A2 A2, B2 B2);
diff --git a/tests/preprocessor/preproc-concat-2.slang.expected b/tests/preprocessor/preproc-concat-2.slang.expected
new file mode 100644
index 000000000..1badd0da0
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-2.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+a A2B2 b ; 
+}
diff --git a/tests/preprocessor/preproc-concat-3.slang b/tests/preprocessor/preproc-concat-3.slang
new file mode 100644
index 000000000..7f1953a2d
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-3.slang
@@ -0,0 +1,16 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define CONCAT(a, b) a ## b
+
+#define A a
+#define B b
+
+#define A2 A
+#define B2 B
+
+// Gives error (as trys to concat unexpanded input)
+// <source>:11:1: error: pasting formed ')CONCAT', an invalid preprocessing token
+//
+// Old Slang output: aabb ;
+
+CONCAT(CONCAT(A2, A2), CONCAT(B2, B2));
diff --git a/tests/preprocessor/preproc-concat-3.slang.expected b/tests/preprocessor/preproc-concat-3.slang.expected
new file mode 100644
index 000000000..b39a659ed
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-3.slang.expected
@@ -0,0 +1,9 @@
+result code = 0
+standard error = {
+tests/preprocessor/preproc-concat-3.slang(5): warning 15503: toking pasting with '##' resulted in the invalid token ')CONCAT'
+#define CONCAT(a, b) a ## b
+                       ^~
+}
+standard output = {
+CONCAT ( a , a ) CONCAT ( b , b ) ; 
+}
diff --git a/tests/preprocessor/preproc-concat-4.slang b/tests/preprocessor/preproc-concat-4.slang
new file mode 100644
index 000000000..4d1e3425f
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-4.slang
@@ -0,0 +1,21 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define CONCAT(a, b) a ## b
+
+#define A a
+#define B b
+
+#define A2 A
+#define B2 B
+
+#define STRINGIFY(x) #x
+
+// Should be 
+// CONCAT(a, b) A2B2 CONCAT(a, b)
+// CONCAT is disabled, A and B are expanded on next pass
+// A2 B2 are first and last tokens pre expansion args
+// 
+// Old Slang outputs 
+// ab ab ab 
+
+CONCAT(CONCAT(A, B) A2, B2 CONCAT(A, B))
diff --git a/tests/preprocessor/preproc-concat-4.slang.expected b/tests/preprocessor/preproc-concat-4.slang.expected
new file mode 100644
index 000000000..6d3b1ca59
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-4.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+CONCAT ( a , b ) A2B2 CONCAT ( a , b ) 
+}
diff --git a/tests/preprocessor/preproc-concat-5.slang.expected b/tests/preprocessor/preproc-concat-5.slang.expected
new file mode 100644
index 000000000..a401f9c6c
--- /dev/null
+++ b/tests/preprocessor/preproc-concat-5.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+THING 
+}
diff --git a/tests/preprocessor/preproc-detail-1.slang b/tests/preprocessor/preproc-detail-1.slang
new file mode 100644
index 000000000..ed465c38c
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-1.slang
@@ -0,0 +1,10 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+// If a macro can take a single parameter, it is valid to pass in 'nothing'.
+// Old Slang outputs an error about the wrong amount of parameters
+// Correct output: a b
+
+#define A(x) a x b
+
+A()
+
diff --git a/tests/preprocessor/preproc-detail-1.slang.expected b/tests/preprocessor/preproc-detail-1.slang.expected
new file mode 100644
index 000000000..7ebaef377
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-1.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+a b 
+}
diff --git a/tests/preprocessor/preproc-detail-2.slang b/tests/preprocessor/preproc-detail-2.slang
new file mode 100644
index 000000000..7cb0c4f7e
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-2.slang
@@ -0,0 +1,8 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+// Macro parameters must have unique names
+
+#define A(x, x) a x b x c
+
+A(,)
+
diff --git a/tests/preprocessor/preproc-detail-2.slang.expected b/tests/preprocessor/preproc-detail-2.slang.expected
new file mode 100644
index 000000000..16468fc68
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-2.slang.expected
@@ -0,0 +1,9 @@
+result code = 0
+standard error = {
+tests/preprocessor/preproc-detail-2.slang(7): error 15408: redefinition of macro parameter 'A'
+#define A(x, x) a x b x c
+             ^
+}
+standard output = {
+a b c 
+}
diff --git a/tests/preprocessor/preproc-detail-3.slang b/tests/preprocessor/preproc-detail-3.slang
new file mode 100644
index 000000000..7d07af33b
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-3.slang
@@ -0,0 +1,7 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+// Undefining a macro that is not defined within C/C++ is defined as *not* an error or a warning.
+// On checking with DXC/FXC they also have this behavior (ie they don't output anything)
+// It's arguable if Slang should match this behavior - at least it is a warning.
+
+#undef C
\ No newline at end of file
diff --git a/tests/preprocessor/preproc-detail-3.slang.expected b/tests/preprocessor/preproc-detail-3.slang.expected
new file mode 100644
index 000000000..44f38e85f
--- /dev/null
+++ b/tests/preprocessor/preproc-detail-3.slang.expected
@@ -0,0 +1,9 @@
+result code = 0
+standard error = {
+tests/preprocessor/preproc-detail-3.slang(9): warning 15401: macro 'C' is not defined
+#undef C
+       ^
+}
+standard output = {
+
+}
diff --git a/tests/preprocessor/preproc-expand-1.slang.expected b/tests/preprocessor/preproc-expand-1.slang.expected
new file mode 100644
index 000000000..70aa3f352
--- /dev/null
+++ b/tests/preprocessor/preproc-expand-1.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+Hi 
+}
diff --git a/tests/preprocessor/preproc-pound-pound-1.slang b/tests/preprocessor/preproc-pound-pound-1.slang
new file mode 100644
index 000000000..3aa157bae
--- /dev/null
+++ b/tests/preprocessor/preproc-pound-pound-1.slang
@@ -0,0 +1,8 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+// GCC: <source>:1:9: error: '##' cannot appear at either end of a macro expansion. 
+// Clang: <source>:1:21: error: '##' cannot appear at start of macro expansion
+// Old Slang outputs Hello ## There;
+#define POUND_POUND ##
+
+Hello POUND_POUND There;
diff --git a/tests/preprocessor/preproc-pound-pound-1.slang.expected b/tests/preprocessor/preproc-pound-pound-1.slang.expected
new file mode 100644
index 000000000..92ba92f20
--- /dev/null
+++ b/tests/preprocessor/preproc-pound-pound-1.slang.expected
@@ -0,0 +1,9 @@
+result code = 0
+standard error = {
+tests/preprocessor/preproc-pound-pound-1.slang(8): error 15405: '##' is not allowed at the start of a macro body
+#define POUND_POUND ##
+                    ^~
+}
+standard output = {
+Hello ## There ; 
+}
diff --git a/tests/preprocessor/preproc-pound-pound-2.slang b/tests/preprocessor/preproc-pound-pound-2.slang
new file mode 100644
index 000000000..d8038c2d4
--- /dev/null
+++ b/tests/preprocessor/preproc-pound-pound-2.slang
@@ -0,0 +1,11 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define A a
+#define B b
+#define OBJ A ## B
+
+// Should output AB 
+// Old Slang outputs ab
+
+OBJ
+
diff --git a/tests/preprocessor/preproc-pound-pound-2.slang.expected b/tests/preprocessor/preproc-pound-pound-2.slang.expected
new file mode 100644
index 000000000..cacad8abd
--- /dev/null
+++ b/tests/preprocessor/preproc-pound-pound-2.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+AB 
+}
diff --git a/tests/preprocessor/preproc-stringify-1.slang b/tests/preprocessor/preproc-stringify-1.slang
new file mode 100644
index 000000000..32bfb00cc
--- /dev/null
+++ b/tests/preprocessor/preproc-stringify-1.slang
@@ -0,0 +1,12 @@
+//DIAGNOSTIC_TEST:SIMPLE:-E
+
+#define A a
+#define B b
+
+// Correct output 
+// "A B"
+// Old Slang output 
+// # a b ;
+
+#define STRINGIFY(x) #x
+STRINGIFY(A B);
\ No newline at end of file
diff --git a/tests/preprocessor/preproc-stringify-1.slang.expected b/tests/preprocessor/preproc-stringify-1.slang.expected
new file mode 100644
index 000000000..dac301880
--- /dev/null
+++ b/tests/preprocessor/preproc-stringify-1.slang.expected
@@ -0,0 +1,6 @@
+result code = 0
+standard error = {
+}
+standard output = {
+"A B" ; 
+}
diff --git a/tests/preprocessor/warning.slang.expected b/tests/preprocessor/warning.slang.expected
index 66b1e5f17..23efd1479 100644
--- a/tests/preprocessor/warning.slang.expected
+++ b/tests/preprocessor/warning.slang.expected
@@ -1,6 +1,6 @@
 result code = 0
 standard error = {
-tests/preprocessor/warning.slang(9): warning 15901: #warning:  You wouldn't like me when I'm angry...
+tests/preprocessor/warning.slang(9): warning 15901: #warning: You wouldn't like me when I'm angry...
 #warning You wouldn't like me when I'm angry...
  ^~~~~~~
 }