| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The existing parser code was doing string-based matching on the lookahead token to figure out how to parse a declaration, e.g.:
```
if(lookAhead == "struct") { /* do struct thing */ }
else if(lookAhead == "interface") { /* do interface thing * }
...
```
That approach has some annoying down-sides:
- It is slower than it needs to be
- It is annoying to deal with cases where the available declaration keywords might differ by language
- Most importantly, it is not possible for us to introduce "extended" keywords that the user can make use of, but which can be ignored by the user and treated as an ordinary identifier.
That last part is important. Suppose the user wanted to have a local variable named `import`, but we also had a Slang extension that added an `import` keyword. Then a line of code like `import += 1` would lead to a failure because we'd try to parse an import declaration, even when it is obvious that the user meant their local variable. This would mean that Slang can't parse existing user code that might clash with syntax extensions. This issue is the reason why we currently have keywords like `__import`.
A traditional solution in a compiler is to map keywords to distinct token codes as part of lexing, which eliminates the first conern (performance) because now we can dispatch with `switch`. It can also aleviate the second concern if we add/remove names from the string->code mapping based on language (the rest of the parsing logic doesn't have to know about keywords being added/removed).
The solution we go for here is more aggressive.
Instead of mapping keyword names to special token codes during lexing, we instead introduce logical "syntax declarations" into the AST, which are looked up using the ordinary scoping rules of the language.
Depending on what code is imported into the scope where parsing is going on, different keywords may then be visible.
This solves our last concern, since a user-defined variable that just happens to use the same name as a keyword is now allowed to shadow the imported declaration for syntax (this is akin to, e.g., Scheme where there really aren't any "keywords").
This also opens the door to the possibility of eventually allowing user to define their own syntax (again, like Scheme).
For now I'm only using this for the declaration keywords.
With this change it should be pretty easy to also add statement keywords in the same fashion.
|
|
Fixes #24
So far the code has used a representation for source locations that is heavy-weight, but typical of research or hobby compilers: a `struct` type containing a line number and a (heap-allocated) string.
This is actually very convenient for debugging, but it means that any data structure that might contain a source location needs careful memory management (because of those strings) and has a tendency to bloat.
The new represnetation is that a source location is just a pointer-sized integer.
In the simplest mental model, you can think of this as just counting every byte of source text that is passed in, and using those to name locations.
Finding the path and line number that corresponds to a location involves a lookup step, but we can arrange to store all the files in an array sorted by their start locations, and do a binary search.
Finding line numbers inside a file is similarly fast (one you pay a one-time cost to build an array of starting offsets for lines).
More advanced compilers like clang actually go further and create a unique range of source locations to represent a file each time it gets included, so that they can track the include stack and reproduce it in diagnostic messages.
I'm not doing anything that clever here.
|
|
The main user-visible change here is that instead of `spAddTranslationUnitEntryPoint` we have `spAddEntryPoint`, to reflect that the list of entry points is "global" to a compile request.
As a result, `spGetEntryPointSource` now only needs the entry point index, and not the translation unit index.
There are a bunch more behind-the-scenes changes, though, reflecting a streamlining of the concepts related to compilation into a smaller number of classes.
Now there is:
- `Session` (unchanged) to manage the lifetimes of shared stuff like the stdlib
- `CompileRequest` (merges in `CompileOptions`) to handle all the lifetime related to a single invocation of the compiler
- `TranslationUnitRequest` (merges `TranslationUnitOptions`, `CompileUnit`) to represent a single translation unit ("module") that the user is trying to compile. This is a single file for HLSL/GLSL, but can be multiple files for Slang.
- `EntryPointRequest` (merges `EntryPointOption` and a bit of `EntryPointResult`) to track a single entry point that the user is asking to compile (that entry point always comes from a single translation unit)
A lot of functions used to take some combination of these and end up with really long signatures.
I've given most of the objects "parent" pointers so that they can get back to all the context they need, so most functions don't need as many parameters.
It may eventually be important to tease these apart again, in particular:
- The code-generation side of things (the `*Result` types) might need to be pulled out in case we want to codegen multiple times from the same AST
- Similarly, the layout stuff may also need to be pulled out, in case we want to lay things out multiple times with different rules.
|
|
That is, even if hte user specified the `-no-checking` option (or the equivalent via API), we still want/need to apply full semantic checks to Slang code, so that cross-compilation will be possible.
|
|
This gets rid of one unecessary namespace.
|
|
|
