Feature/lex memory reduction (#762)

* Only do scrubbing if needed. When allocating content try to limit size (with scrubbing each token takes up 1k), now it's 16 bytes min size.

* Don't allocate for every call to write on the CallbackWriter - use the m_appendBuffer.

* Don't allocate memory for CallbackWriter use m_appendBuffer.

* Use UnownedStringSlice for suffix output for parsing float/int literals.
Fix typo in invalidFloatingPointLiteralSuffix

* Using memory arena to hold tokens that are not in SourceManager.

* Improve comment on lexing.

* Make UnownedStringSlice allocation simpler on SourceManager.

* Fix error on gcc around UnownedStringSlice - because VC converted string + UnownedStringSlice automatically into a String.

* Fix generateName needing concat string for gcc.

* When constructing a Token in parseAttributeName - because it's a Identifier, we have to set the Name.

* Remove translation through String on getIntrinsicOp

* Make func-cbuffer-param disablable with -exclude compatibility-issue

* Move memory leak in render-test.

* From review - can just use "?:" instead of performing a concat.

1 files changed, 7 insertions, 10 deletions

diff --git a/tools/render-test/main.cpp b/tools/render-test/main.cpp
index 631085c2b..dab8e9c6e 100644
--- a/tools/render-test/main.cpp
+++ b/tools/render-test/main.cpp
@@ -348,13 +348,10 @@ Result RenderTestApp::initializeShaders(ShaderCompiler* shaderCompiler)
 	fseek(sourceFile, 0, SEEK_END);
 	size_t sourceSize = ftell(sourceFile);
 	fseek(sourceFile, 0, SEEK_SET);
-	char* sourceText = (char*)malloc(sourceSize + 1);
-	if (!sourceText)
-	{
-		fprintf(stderr, "error: out of memory");
-		return SLANG_FAIL;
-	}
-	fread(sourceText, sourceSize, 1, sourceFile);
+
+    List<char> sourceText;
+    sourceText.SetSize(sourceSize + 1);
+	fread(sourceText.Buffer(), sourceSize, 1, sourceFile);
 	fclose(sourceFile);
 	sourceText[sourceSize] = 0;
 
@@ -368,12 +365,12 @@ Result RenderTestApp::initializeShaders(ShaderCompiler* shaderCompiler)
         m_shaderInputLayout.numRenderTargets = 0;
         break;
     }
-	m_shaderInputLayout.Parse(sourceText);
+	m_shaderInputLayout.Parse(sourceText.Buffer());
 
 	ShaderCompileRequest::SourceInfo sourceInfo;
 	sourceInfo.path = sourcePath;
-	sourceInfo.dataBegin = sourceText;
-	sourceInfo.dataEnd = sourceText + sourceSize;
+	sourceInfo.dataBegin = sourceText.Buffer();
+	sourceInfo.dataEnd = sourceText.Buffer() + sourceSize;
 
 	ShaderCompileRequest compileRequest;
 	compileRequest.source = sourceInfo;