Add Yet Another Source Code Generator (#6844)

* Add Yet Another Source Code Generator

This change introduces an offline source code generation tool,
provisionally called `fiddle`. More information about the design of
the tool can be found in `tools/slang-fiddle/README.md`.

Yes... this is yet another code generator in a project that already
has too many. Yes, this could easily be a very obvious instnace of
[XKCD 927](https://xkcd.com/927/).

This change is part of a larger effort to change how the AST
types are being serialized, and the way code generation for them
is implemented.

Right now, the source code for the new tool is being checked in and
the relevant build step is enabled, just to make sure everything is
working as intended, but please note that this change does *not*
introduce any code in the repository that actually makes use of
the new generator. All of the AST-related reflection information that
feeds the current serialization system is still being generated using
`slang-cpp-extractor`.

The design of the new tool is primarily motivated by the new approach
to serialization that I'm implementing, and once that new approach
lands we should be able to deprecate the `slang-cpp-extractor`.

In addition, the new tool should in principle be able to handle
many of the kinds of code generation tasks that are currently being
implemented with other tools like `slang-generate` (used for the core
and glsl libraries). This tool should also be well suited to the task
of generating more of the code related to the IR instructions.

* format code

* Build fixes caught by CI

* Fix another warning coming from CI

* Another CI-caught fix

* Change bare 	hrows over to more proper abort execptions

* format code

---------

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

2 files changed, 15 insertions, 0 deletions

diff --git a/source/compiler-core/slang-lexer.cpp b/source/compiler-core/slang-lexer.cpp
index 84a4df93b..048c266ca 100644
--- a/source/compiler-core/slang-lexer.cpp
+++ b/source/compiler-core/slang-lexer.cpp
@@ -1810,6 +1810,18 @@ TokenList Lexer::lexAllMarkupTokens()
     }
 }
 
+TokenList Lexer::lexAllTokens()
+{
+    TokenList tokenList;
+    for (;;)
+    {
+        Token token = lexToken();
+        tokenList.add(token);
+        if (token.type == TokenType::EndOfFile)
+            return tokenList;
+    }
+}
+
 /* static */ UnownedStringSlice Lexer::sourceLocationLexer(const UnownedStringSlice& in)
 {
     Lexer lexer;
diff --git a/source/compiler-core/slang-lexer.h b/source/compiler-core/slang-lexer.h
index a36719ab7..0152ff6f5 100644
--- a/source/compiler-core/slang-lexer.h
+++ b/source/compiler-core/slang-lexer.h
@@ -139,6 +139,9 @@ struct Lexer
     /// Lex all tokens (up to the end of the stream) that are relevant to things like markup
     TokenList lexAllMarkupTokens();
 
+    /// Lex all tokens (up to the end of the stream) whether relevant or not.
+    TokenList lexAllTokens();
+
     /// Get the diagnostic sink, taking into account flags. Will return null if suppressing
     /// diagnostics.
     DiagnosticSink* getDiagnosticSink()