Bug fix for newline escaping.

The previous changes had left out logic for "scrubbing" a token value that includes an escaped newline, because I expected it would only occur within whitespace. Unfortunately, some user code looked like this:

```
a + b
```

That is, there was a token at the very start of the line, after the escaped newline.
As a result, after consuming the leading whitespace (which didn't end up consuming the escaped newline - but we could consider making it do so in future), the lexer started to lex a token that *starts* with an escaped newline, but turns out to be an identifer (which gets an invalid name).

This change adds some ad-hoc code to "scrub" the value of *every* token, which wasteful but at least solves the problem.

0 files changed, 0 insertions, 0 deletions