Feature/lex memory reduction (#762)

* Only do scrubbing if needed. When allocating content try to limit size (with scrubbing each token takes up 1k), now it's 16 bytes min size.

* Don't allocate for every call to write on the CallbackWriter - use the m_appendBuffer.

* Don't allocate memory for CallbackWriter use m_appendBuffer.

* Use UnownedStringSlice for suffix output for parsing float/int literals.
Fix typo in invalidFloatingPointLiteralSuffix

* Using memory arena to hold tokens that are not in SourceManager.

* Improve comment on lexing.

* Make UnownedStringSlice allocation simpler on SourceManager.

* Fix error on gcc around UnownedStringSlice - because VC converted string + UnownedStringSlice automatically into a String.

* Fix generateName needing concat string for gcc.

* When constructing a Token in parseAttributeName - because it's a Identifier, we have to set the Name.

* Remove translation through String on getIntrinsicOp

* Make func-cbuffer-param disablable with -exclude compatibility-issue

* Move memory leak in render-test.

* From review - can just use "?:" instead of performing a concat.

1 files changed, 10 insertions, 0 deletions

diff --git a/source/slang/source-loc.cpp b/source/slang/source-loc.cpp
index 62926aa8c..6d6d63ae1 100644
--- a/source/slang/source-loc.cpp
+++ b/source/slang/source-loc.cpp
@@ -292,6 +292,16 @@ void SourceManager::initialize(
     m_nextLoc = m_startLoc;
 }
 
+UnownedStringSlice SourceManager::allocateStringSlice(const UnownedStringSlice& slice)
+{
+    const UInt numChars = slice.size();
+
+    char* dst = (char*)m_memoryArena.allocate(numChars);
+    ::memcpy(dst, slice.begin(), numChars);
+
+    return UnownedStringSlice(dst, numChars);
+}
+
 SourceRange SourceManager::allocateSourceRange(UInt size)
 {
     // TODO: consider using atomics here