| Age | Commit message (Collapse) | Author |
|---|
|
The previous changes had left out logic for "scrubbing" a token value that includes an escaped newline, because I expected it would only occur within whitespace. Unfortunately, some user code looked like this:
```
a + b
```
That is, there was a token at the very start of the line, after the escaped newline.
As a result, after consuming the leading whitespace (which didn't end up consuming the escaped newline - but we could consider making it do so in future), the lexer started to lex a token that *starts* with an escaped newline, but turns out to be an identifer (which gets an invalid name).
This change adds some ad-hoc code to "scrub" the value of *every* token, which wasteful but at least solves the problem.
|
|
This gets rid of one unecessary namespace.
|
|
This is really messy and I'm not entirely happy with the result, but at least we handle a couple more corner cases.
|
|
This is mostly to allow for the idiomatic style of defining a multi-line macro in C:
#define FOO(a,b) \
x(a) \
y(b) \
/* end */
The handling is reasonably general: in the lexer whenever we need to consume or "peek" the next code point, we check if we are at the start of a backslash-newline sequence, and if so we skip past that to find what we were looking for.
However, the way I'm handling things right now there is no step taken to "clean" a token and remove the backslash or newline from its value, so downstream code that actually inspects token values will probably break if users start putting escaped newlines in the middle of names.
We can fix that issue when (if) it comes up.
|
|
|
