Feature/lex memory reduction (#762)

* Only do scrubbing if needed. When allocating content try to limit size (with scrubbing each token takes up 1k), now it's 16 bytes min size.

* Don't allocate for every call to write on the CallbackWriter - use the m_appendBuffer.

* Don't allocate memory for CallbackWriter use m_appendBuffer.

* Use UnownedStringSlice for suffix output for parsing float/int literals.
Fix typo in invalidFloatingPointLiteralSuffix

* Using memory arena to hold tokens that are not in SourceManager.

* Improve comment on lexing.

* Make UnownedStringSlice allocation simpler on SourceManager.

* Fix error on gcc around UnownedStringSlice - because VC converted string + UnownedStringSlice automatically into a String.

* Fix generateName needing concat string for gcc.

* When constructing a Token in parseAttributeName - because it's a Identifier, we have to set the Name.

* Remove translation through String on getIntrinsicOp

* Make func-cbuffer-param disablable with -exclude compatibility-issue

* Move memory leak in render-test.

* From review - can just use "?:" instead of performing a concat.

1 files changed, 6 insertions, 3 deletions

diff --git a/source/slang/lexer.h b/source/slang/lexer.h
index 85aca77dd..593e07c22 100644
--- a/source/slang/lexer.h
+++ b/source/slang/lexer.h
@@ -75,7 +75,8 @@ namespace Slang
         void initialize(
             SourceView*     sourceView,
             DiagnosticSink* sink,
-            NamePool*       namePool);
+            NamePool*       namePool,
+            MemoryArena*    memoryArena);
 
         ~Lexer();
 
@@ -97,6 +98,8 @@ namespace Slang
 
         TokenFlags      tokenFlags;
         LexerFlags      lexerFlags;
+
+        MemoryArena*    memoryArena;
     };
 
     // Helper routines for extracting values from tokens
@@ -106,8 +109,8 @@ namespace Slang
     typedef int64_t IntegerLiteralValue;
     typedef double FloatingPointLiteralValue;
 
-    IntegerLiteralValue getIntegerLiteralValue(Token const& token, String* outSuffix = 0);
-    FloatingPointLiteralValue getFloatingPointLiteralValue(Token const& token, String* outSuffix = 0);
+    IntegerLiteralValue getIntegerLiteralValue(Token const& token, UnownedStringSlice* outSuffix = 0);
+    FloatingPointLiteralValue getFloatingPointLiteralValue(Token const& token, UnownedStringSlice* outSuffix = 0);
 }
 
 #endif