Feature/lex memory reduction (#762)

* Only do scrubbing if needed. When allocating content try to limit size (with scrubbing each token takes up 1k), now it's 16 bytes min size.

* Don't allocate for every call to write on the CallbackWriter - use the m_appendBuffer.

* Don't allocate memory for CallbackWriter use m_appendBuffer.

* Use UnownedStringSlice for suffix output for parsing float/int literals.
Fix typo in invalidFloatingPointLiteralSuffix

* Using memory arena to hold tokens that are not in SourceManager.

* Improve comment on lexing.

* Make UnownedStringSlice allocation simpler on SourceManager.

* Fix error on gcc around UnownedStringSlice - because VC converted string + UnownedStringSlice automatically into a String.

* Fix generateName needing concat string for gcc.

* When constructing a Token in parseAttributeName - because it's a Identifier, we have to set the Name.

* Remove translation through String on getIntrinsicOp

* Make func-cbuffer-param disablable with -exclude compatibility-issue

* Move memory leak in render-test.

* From review - can just use "?:" instead of performing a concat.

1 files changed, 15 insertions, 1 deletions

diff --git a/source/slang/source-loc.h b/source/slang/source-loc.h
index aba9b7bd9..6949d659a 100644
--- a/source/slang/source-loc.h
+++ b/source/slang/source-loc.h
@@ -1,4 +1,4 @@
-// source-loc.h
+// source-loc.h
 #ifndef SLANG_SOURCE_LOC_H_INCLUDED
 #define SLANG_SOURCE_LOC_H_INCLUDED
 
@@ -303,6 +303,16 @@ struct SourceManager
         /// Get the parent manager to this manager. Returns nullptr if there isn't any.
     SourceManager* getParent() const { return m_parent; }
 
+        /// A memory arena to hold allocations that are in scope for the same time as SourceManager
+    MemoryArena* getMemoryArena() { return &m_memoryArena;  }
+
+        /// Allocate a string slice
+    UnownedStringSlice allocateStringSlice(const UnownedStringSlice& slice);
+    
+    SourceManager() :
+        m_memoryArena(2048)
+    {}
+
     protected:
 
     // The first location available to this source manager
@@ -320,6 +330,10 @@ struct SourceManager
     List<RefPtr<SourceView> > m_sourceViews;                
     StringSlicePool m_slicePool;
 
+    // Memory arena that can be used for holding data to held in scope as long as the Source is
+    // Can be used for storing the decoded contents of Token.Content for exampel
+    MemoryArena m_memoryArena;
+
     // Maps canonical paths to source files
     Dictionary<String, RefPtr<SourceFile> > m_sourceFiles;  
 };