Overhaul the preprocessor (#1849)

* Overhaul the preprocessor

The old Slang preprocessor was based on a simple mental model that tried to unify two parts of macro expansion:

* Scanning for macro invocations in a sequence of tokens

* Producing the expanded tokens for a macro expansion by substituting arguments into its body

The basic was that substitution of macro arguments into a macro definition is superficially similar to top-level macro expansion, just with an environment where the macro arguments act like `#define`s for the corresponding parameter names. That approach was "clever" and could conceivably have been extended to include a lot of advanced preprocessor features (e.g., a preprocessor-level `lambda` would be easy to support!), but it was basically impossible to make it correctly handle all the corner cases of the full C/C++ preprocessor.

The fundamental problem with the old approach was that it conflated the two parts of expansion listed above into one implementation, while the various special cases of the C/C++ preprocessor rely on treating the two cases very differently. The new approach here (which is somewhere between a refactor and a full rewrite of the preprocessor) changes things up in a few key ways:

* The abstraction still cares a lot about streams of tokens, but it now treats the top level streams (`InputFile`s) as fairly different from the lower-level streams (`InputStream`s)

* Macro expansion is handled as a dedicated type of stream that wraps another stream. This allows macro expansion to be applied to anything, and supports cases where multiple rounds of macro expansion are required by the spec.

* Macro *invocations* and the substitution of their arguments are now handled by a completely new system.

    * Macro arguments are no longer treated as if they were `#define`s

    * The macro body/definition is analyzed at definition time to detect various kinds of issues, and to derive a list of "ops" that make it easier to "play back" the definition at substitution time

* Token pasting and stringizing are now only handled in macro definitions (rather than being allowed anywhere), and their use cases are restricted to only those that make sense (e.g., you can't stringize anythign except a macro parameter, because anything else wouldn't make sense)

The key new types here are the `ExpansionInputStream` which handles scanning for macro invocations, and the `MacroInvocation` type, which handles playing back the macro body with substitutions.

The `ExpansionInputStream` is the easier of the two to understand. By refactoring it to use a single token of lookahead, the one major detail it had to deal with before (abandoning expansion of a function-like macro if the macro name was not followed by `(`) is significantly easier to manage.

The more subtle part is the `MacroInvocation` type, and most of the complexity there is around handling of token pasting, and the fact that either or both of the operands to a token paste might be empty.

Many of the test cases that exposed the problems in the preprocessor have been moved from `current-bugs` to `preprocessor` since they now work correctly.

* debugging: enable extractor command line dump

* fixup

* fixup

25 files changed, 0 insertions, 261 deletions

diff --git a/tests/current-bugs/paste-non-expansion.slang b/tests/current-bugs/paste-non-expansion.slang
deleted file mode 100644
index 8270db676..000000000
--- a/tests/current-bugs/paste-non-expansion.slang
+++ /dev/null
@@ -1,16 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// This demonstrates the existance of a bug in Slang preprocessor macro expansion. Could be due to incorrect paste handling
-// or perhaps the rules around parameter expansion.
-
-#define CONCAT2(x, y) x ## y
-#define CONCAT(x, y) CONCAT2(x, y)
-
-#define SOMETHING someThing
-
-// Should be someThingElse
-CONCAT(SOMETHING, Else)
-// Should be SOMETHINGAnother, but Slang expands to produce someThingAnother
-CONCAT2(SOMETHING, Another)
\ No newline at end of file
diff --git a/tests/current-bugs/paste-non-expansion.slang.expected b/tests/current-bugs/paste-non-expansion.slang.expected
deleted file mode 100644
index 2fbb7bad3..000000000
--- a/tests/current-bugs/paste-non-expansion.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-someThingElse someThingAnother 
-}
diff --git a/tests/current-bugs/preproc-concat-1.slang b/tests/current-bugs/preproc-concat-1.slang
deleted file mode 100644
index d9f205457..000000000
--- a/tests/current-bugs/preproc-concat-1.slang
+++ /dev/null
@@ -1,14 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define CONCAT(a, b) a ## b
-
-// Correct output AB;
-// Slang output
-// ab;
-
-#define A a
-#define B b
-
-CONCAT(A, B);
diff --git a/tests/current-bugs/preproc-concat-1.slang.expected b/tests/current-bugs/preproc-concat-1.slang.expected
deleted file mode 100644
index 673d7bc4c..000000000
--- a/tests/current-bugs/preproc-concat-1.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-ab ; 
-}
diff --git a/tests/current-bugs/preproc-concat-2.slang b/tests/current-bugs/preproc-concat-2.slang
deleted file mode 100644
index f3c4d28ff..000000000
--- a/tests/current-bugs/preproc-concat-2.slang
+++ /dev/null
@@ -1,17 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define CONCAT(a, b) a ## b
-
-#define A a
-#define B b
-
-#define A2 A
-#define B2 B
-
-// Correct output: a A2B2 b;
-// Slang output 
-// a ab b ;
-
-CONCAT(A2 A2, B2 B2);
diff --git a/tests/current-bugs/preproc-concat-2.slang.expected b/tests/current-bugs/preproc-concat-2.slang.expected
deleted file mode 100644
index 4b30fc7c2..000000000
--- a/tests/current-bugs/preproc-concat-2.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-a ab b ; 
-}
diff --git a/tests/current-bugs/preproc-concat-3.slang b/tests/current-bugs/preproc-concat-3.slang
deleted file mode 100644
index c4077f75d..000000000
--- a/tests/current-bugs/preproc-concat-3.slang
+++ /dev/null
@@ -1,18 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define CONCAT(a, b) a ## b
-
-#define A a
-#define B b
-
-#define A2 A
-#define B2 B
-
-// Gives error (as trys to concat unexpanded input)
-// <source>:11:1: error: pasting formed ')CONCAT', an invalid preprocessing token
-//
-// Slang output: aabb ;
-
-CONCAT(CONCAT(A2, A2), CONCAT(B2, B2));
diff --git a/tests/current-bugs/preproc-concat-3.slang.expected b/tests/current-bugs/preproc-concat-3.slang.expected
deleted file mode 100644
index 8be308266..000000000
--- a/tests/current-bugs/preproc-concat-3.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-aabb ; 
-}
diff --git a/tests/current-bugs/preproc-concat-4.slang b/tests/current-bugs/preproc-concat-4.slang
deleted file mode 100644
index 31ec16268..000000000
--- a/tests/current-bugs/preproc-concat-4.slang
+++ /dev/null
@@ -1,23 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define CONCAT(a, b) a ## b
-
-#define A a
-#define B b
-
-#define A2 A
-#define B2 B
-
-#define STRINGIFY(x) #x
-
-// Should be 
-// CONCAT(a, b) A2B2 CONCAT(a, b)
-// CONCAT is disabled, A and B are expanded on next pass
-// A2 B2 are first and last tokens pre expansion args
-// 
-// Slang outputs 
-// ab ab ab 
-
-CONCAT(CONCAT(A, B) A2, B2 CONCAT(A, B))
diff --git a/tests/current-bugs/preproc-concat-4.slang.expected b/tests/current-bugs/preproc-concat-4.slang.expected
deleted file mode 100644
index 82d7225b2..000000000
--- a/tests/current-bugs/preproc-concat-4.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-ab ab ab 
-}
diff --git a/tests/current-bugs/preproc-concat-5.slang b/tests/current-bugs/preproc-concat-5.slang
deleted file mode 100644
index 2be554f89..000000000
--- a/tests/current-bugs/preproc-concat-5.slang
+++ /dev/null
@@ -1,13 +0,0 @@
-//DISABLE_DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// It should produce 'THING', as the original invocation should have disabled THING, 
-// but it actually ends up in an infinite loop.
-
-#define CONCAT(x, y) x ## y
-
-#define THING2 THING
-#define THING CONCAT(THING, 2)
-
-THING
diff --git a/tests/current-bugs/preproc-detail-1.slang b/tests/current-bugs/preproc-detail-1.slang
deleted file mode 100644
index 4cff0d205..000000000
--- a/tests/current-bugs/preproc-detail-1.slang
+++ /dev/null
@@ -1,12 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// If a macro can take a single parameter, it is valid to pass in 'nothing'.
-// Slang outputs an error about the wrong amount of parameters
-// Correct output: a b
-
-#define A(x) a x b
-
-A()
-
diff --git a/tests/current-bugs/preproc-detail-1.slang.expected b/tests/current-bugs/preproc-detail-1.slang.expected
deleted file mode 100644
index ba93983af..000000000
--- a/tests/current-bugs/preproc-detail-1.slang.expected
+++ /dev/null
@@ -1,7 +0,0 @@
-result code = 0
-standard error = {
-tests/current-bugs/preproc-detail-1.slang(10): error 15501: wrong number of arguments to macro (expected 1, got 0)
-}
-standard output = {
-
-}
diff --git a/tests/current-bugs/preproc-detail-2.slang b/tests/current-bugs/preproc-detail-2.slang
deleted file mode 100644
index eb687db31..000000000
--- a/tests/current-bugs/preproc-detail-2.slang
+++ /dev/null
@@ -1,10 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// Macro parameters must have unique names
-
-#define A(x, x) a x b x c
-
-A(,)
-
diff --git a/tests/current-bugs/preproc-detail-2.slang.expected b/tests/current-bugs/preproc-detail-2.slang.expected
deleted file mode 100644
index b48d7909f..000000000
--- a/tests/current-bugs/preproc-detail-2.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-a b c 
-}
diff --git a/tests/current-bugs/preproc-detail-3.slang b/tests/current-bugs/preproc-detail-3.slang
deleted file mode 100644
index b0675204b..000000000
--- a/tests/current-bugs/preproc-detail-3.slang
+++ /dev/null
@@ -1,9 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// Undefining a macro that is not defined within C/C++ is defined as *not* an error or a warning.
-// On checking with DXC/FXC they also have this behavior (ie they don't output anything)
-// It's arguable if Slang should match this behavior - at least it is a warning.
-
-#undef C
\ No newline at end of file
diff --git a/tests/current-bugs/preproc-detail-3.slang.expected b/tests/current-bugs/preproc-detail-3.slang.expected
deleted file mode 100644
index a44c60220..000000000
--- a/tests/current-bugs/preproc-detail-3.slang.expected
+++ /dev/null
@@ -1,9 +0,0 @@
-result code = 0
-standard error = {
-tests/current-bugs/preproc-detail-3.slang(9): warning 15401: macro 'C' is not defined
-#undef C
-       ^
-}
-standard output = {
-
-}
diff --git a/tests/current-bugs/preproc-expand-1.slang b/tests/current-bugs/preproc-expand-1.slang
deleted file mode 100644
index f95b432e2..000000000
--- a/tests/current-bugs/preproc-expand-1.slang
+++ /dev/null
@@ -1,16 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// Should produce: Hi
-// Slang produces: C ( Hi ) 
-
-#define OPEN (
-#define CLOSE )
-
-#define C(x) x
-
-#define B(x) x
-
-B(C OPEN Hi CLOSE)
-
diff --git a/tests/current-bugs/preproc-expand-1.slang.expected b/tests/current-bugs/preproc-expand-1.slang.expected
deleted file mode 100644
index d46fa1159..000000000
--- a/tests/current-bugs/preproc-expand-1.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-C ( Hi ) 
-}
diff --git a/tests/current-bugs/preproc-pound-pound-1.slang b/tests/current-bugs/preproc-pound-pound-1.slang
deleted file mode 100644
index 7f369b861..000000000
--- a/tests/current-bugs/preproc-pound-pound-1.slang
+++ /dev/null
@@ -1,10 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-// GCC: <source>:1:9: error: '##' cannot appear at either end of a macro expansion. 
-// Clang: <source>:1:21: error: '##' cannot appear at start of macro expansion
-// Slang outputs Hello ## There;
-#define POUND_POUND ##
-
-Hello POUND_POUND There;
diff --git a/tests/current-bugs/preproc-pound-pound-1.slang.expected b/tests/current-bugs/preproc-pound-pound-1.slang.expected
deleted file mode 100644
index 80a64402a..000000000
--- a/tests/current-bugs/preproc-pound-pound-1.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-Hello ## There ; 
-}
diff --git a/tests/current-bugs/preproc-pound-pound-2.slang b/tests/current-bugs/preproc-pound-pound-2.slang
deleted file mode 100644
index 44181261b..000000000
--- a/tests/current-bugs/preproc-pound-pound-2.slang
+++ /dev/null
@@ -1,13 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define A a
-#define B b
-#define OBJ A ## B
-
-// Should output AB 
-// Slang outputs ab
-
-OBJ
-
diff --git a/tests/current-bugs/preproc-pound-pound-2.slang.expected b/tests/current-bugs/preproc-pound-pound-2.slang.expected
deleted file mode 100644
index a818a4683..000000000
--- a/tests/current-bugs/preproc-pound-pound-2.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-ab 
-}
diff --git a/tests/current-bugs/preproc-stringify-1.slang b/tests/current-bugs/preproc-stringify-1.slang
deleted file mode 100644
index 03e8366b5..000000000
--- a/tests/current-bugs/preproc-stringify-1.slang
+++ /dev/null
@@ -1,14 +0,0 @@
-//DIAGNOSTIC_TEST:SIMPLE:-E
-
-// NOTE! This test should *fail*, if preprocessor is working correctly!
-
-#define A a
-#define B b
-
-// Correct output 
-// "A B"
-// Slang output 
-// # a b ;
-
-#define STRINGIFY(x) #x
-STRINGIFY(A B);
\ No newline at end of file
diff --git a/tests/current-bugs/preproc-stringify-1.slang.expected b/tests/current-bugs/preproc-stringify-1.slang.expected
deleted file mode 100644
index 53efce117..000000000
--- a/tests/current-bugs/preproc-stringify-1.slang.expected
+++ /dev/null
@@ -1,6 +0,0 @@
-result code = 0
-standard error = {
-}
-standard output = {
-# a b ; 
-}