Partial fix for macro expasnion of token pastes (#1727)

The underlying problem here requires that we have an object-like macro with an expansion that starts with a non-identifier token:

```
```

Then we need a function-like macro that uses a token paste in a way that can expand to that object-like macro:

```
```

Finally, for the specific case a user ran into, we need to invoke that function-like macro in the context of a preprocessor conditional expression:

```
    // ...
    #error "unimplemented"
```

The way a problem manifest is that the preprocessor logic that handles conditional expressions tries to "peek" one token ahead and see what is coming, and while the peeking logic handles macro expansion it does *not* handle token pasting right now. That means that the peek operation sees `MY_FEATURE` and assumes that it is seeing an identifier in a preprocessor conditional that doesn't have a macro expansion. The logic then goes on to read the token, but what it gets back is *not* an identifier, and is instead the numeric literal token `1`, because the reading logic handles token pasting.

The quick fix I applied here is to make the logic that deals with preprocessor conditionals go ahead and automatically consume a token from the input, and then decide what to do based on that token, so that it always makes use of the reading logic that handles token pasting.

The lingering problem is that we still have cases in the preprocessor that use the peeking logic which doesn't handle pasting, and we might find that those cases have reason to want the same kind of expansion behavior we needed here. A more systematic fix would be to have the peeking logic automatically handle token pasting as well as macro expansion, but doing so would be a more complicated change because detecting the `##` when peeking ahead requires two tokens of lookahead, and our current implementation only assumes we can support one.

Co-authored-by: Yong He <yonghe@outlook.com>

1 files changed, 11 insertions, 8 deletions

diff --git a/source/slang/slang-preprocessor.cpp b/source/slang/slang-preprocessor.cpp
index 41b2c5bdf..f233d46f0 100644
--- a/source/slang/slang-preprocessor.cpp
+++ b/source/slang/slang-preprocessor.cpp
@@ -1348,23 +1348,27 @@ static PreprocessorExpressionValue ParseAndEvaluateExpression(PreprocessorDirect
 // Parse a unary (prefix) expression inside of a preprocessor directive.
 static PreprocessorExpressionValue ParseAndEvaluateUnaryExpression(PreprocessorDirectiveContext* context)
 {
-    switch (PeekTokenType(context))
+    if( PeekTokenType(context) == TokenType::EndOfDirective )
+    {
+        GetSink(context)->diagnose(PeekLoc(context), Diagnostics::syntaxErrorInPreprocessorExpression);
+        return 0;
+    }
+
+    auto token = AdvanceToken(context);
+    switch (token.type)
     {
     // handle prefix unary ops
     case TokenType::OpSub:
-        AdvanceToken(context);
         return -ParseAndEvaluateUnaryExpression(context);
     case TokenType::OpNot:
-        AdvanceToken(context);
         return !ParseAndEvaluateUnaryExpression(context);
     case TokenType::OpBitNot:
-        AdvanceToken(context);
         return ~ParseAndEvaluateUnaryExpression(context);
 
     // handle parenthized sub-expression
     case TokenType::LParent:
         {
-            Token leftParen = AdvanceToken(context);
+            Token leftParen = token;
             PreprocessorExpressionValue value = ParseAndEvaluateExpression(context);
             if (!Expect(context, TokenType::RParent, Diagnostics::expectedTokenInPreprocessorExpression))
             {
@@ -1374,11 +1378,10 @@ static PreprocessorExpressionValue ParseAndEvaluateUnaryExpression(PreprocessorD
         }
 
     case TokenType::IntegerLiteral:
-        return StringToInt(AdvanceToken(context).getContent());
+        return StringToInt(token.getContent());
 
     case TokenType::Identifier:
         {
-            Token token = AdvanceToken(context);
             if (token.getContent() == "defined")
             {
                 // handle `defined(someName)`
@@ -1419,7 +1422,7 @@ static PreprocessorExpressionValue ParseAndEvaluateUnaryExpression(PreprocessorD
         }
 
     default:
-        GetSink(context)->diagnose(PeekLoc(context), Diagnostics::syntaxErrorInPreprocessorExpression);
+        GetSink(context)->diagnose(token.loc, Diagnostics::syntaxErrorInPreprocessorExpression);
         return 0;
     }
 }