PersistentJSONValue (#2028)

* #include an absolute path didn't work - because paths were taken to always be relative.

* Use 'Process' to communicate with an command line tool.

* Remove slang-win-stream

* Tidy up windows ProcessUtil.

* First version of BufferedReadStream.

* Windows working IPC for steams.

* Test proxy count option.

* Split Process/ProcessUtil. Process is platform dependant. ProcessUtil are functions that are platform independent.

* First implementation of Unix Process interface.

* Unix process compiles on cygwin.

* Fix typo in unix process.

* Separate unix pipe stream error of invalid access, from pipe availability.

* Fix in standard line extraction.

* Make fd non blocking.

* Fix issues with Windows Process streams.

* Added UnixPipe.

* Some fixes around UnixPipeStream.

* Make a unix stream closed explicit.

* Hack to debug linux process/stream.

* Revert to old linux pipe handling.

* Pass executable path for unit tests.
Split out CommandLine into own source.

* Small improvements in process/command line.

* Check process behavior with crash.

* Make stderr and stdout unbuffered for crash testing.

* Only turn disable buffering in crash test.

* Disable crash test on CI.

* Fix crash on clang/linux.

* Enable crash test.
Remove _appendBuffer as can use StreamUtil functionality.

* Added inital processing for http headers.

* Small improvements to HttpHeader.

* First pass HTTPPacketConnection working on windows.

* Enable other Process communication tests.

* Update comments.

* WIP JSON RPC.

* Add terminate to Process.
Made JSONRPC a Util.

* Small tidy up around HTTPPacketConnection.

* Improve process termination options.

* WIP for test-server.

* Add diagnostics error handling to test-server.

* Improved JSON support.
Parsing/creating JSON-RPC messages.

* WIP JSONRPC parsing.

* First pass RttiInfo support.

* WIP converting between JSON/native types.

* Project files.

* Split out RttiUtil.
Made RttiInfo constuction thread safe.

* WIP RTTI<->JSON.

* Add diagnostics to JSON<->native conversions.

* Make RttiInfo for structs globals. Avoids problem around derived types (like pointers), being able to cause an abort.

* Add pointer support to RTTI.
Fixed some compilation issues on linux.

* Add fixed array support.

* Added Rtti unit test.

* Add rtti unit test.

* Split out quoted/unquoted key handling.
Fix bugs in JSON value/container.
Added JSON native test.

* Make default array allocator use malloc/free.
Remove the new[] handler (doesn't work on visuals studio).

* Fix for linux warning.

* Remove some test code.

* Fix issues on x86 win.

* Fix warning on aarch64.

* Fix some bugs in JSON parsing/handling.
Make Rtti work copy/dtor/ctor struct types.

* Testing JSON<->native with fixed array.
Make makeArrayView explicit if it's just a single value.
Added array type.

* Fix getting arrayView.

* Improve JSON diagnostic name.

* First pass refactor using Rtti for JSON RPC.

* First pass of test server using RTTI/JSON-RPC.

* Added JSONRPCConnection.

* Fix some naming issues.

* First pass of test-server working.

* Added unit test support for JSON-RPC test server.

* Fix compilation issues on linux around template handling.

* Typo fix.

* Fix a bug around SourceLoc lookup with JSONContainer.

* Set the console type to console for ISlangWriters.

* Small improvements to test-server.

* Small improvements in test-server.

* Small fix.

* Remove test-proxy. Make test-process a process that can be used to unit test 'Process'.
Adding mechanism to control spawning that will create a new process for every test.

* Ability to remove source manager for JSONValue.

* WIP SimpleJSONValue.

* Add PersistentJSONValue

* Testing around PersistentJSONValue.
Bug fixes.

* Small code improvements.

4 files changed, 386 insertions, 20 deletions

diff --git a/source/compiler-core/slang-json-value.cpp b/source/compiler-core/slang-json-value.cpp
index ba4ccad09..9edb81b67 100644
--- a/source/compiler-core/slang-json-value.cpp
+++ b/source/compiler-core/slang-json-value.cpp
@@ -22,6 +22,8 @@ namespace Slang {
     JSONValue::Kind::Float,             // FloatValue,
     JSONValue::Kind::String,            // StringValue,
 
+    JSONValue::Kind::String,            // StringRepresentation
+
     JSONValue::Kind::Array,             // Array,
     JSONValue::Kind::Object,            // Object,
 };
@@ -143,6 +145,221 @@ double JSONValue::asFloat() const
 
 /* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 
+                             PersistentJSONValue
+
+!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! */
+
+PersistentJSONValue::PersistentJSONValue(const ThisType& rhs)
+{
+    *(JSONValue*)this = rhs;
+
+    if (type == Type::StringRepresentation && stringRep)
+    {
+        stringRep->addReference();
+    }
+}
+
+void PersistentJSONValue::operator=(const ThisType& rhs)
+{
+    if (this != &rhs)
+    {
+        if (rhs.type == Type::StringRepresentation && rhs.stringRep)
+        {
+            rhs.stringRep->addReference();
+        }
+        if (type == Type::StringRepresentation && stringRep)
+        {
+            stringRep->releaseReference();
+        }
+        *(JSONValue*)this = rhs;
+    }
+}
+
+String PersistentJSONValue::getString() const
+{
+    if (type == Type::StringRepresentation)
+    {
+        return String(stringRep);
+    }
+    SLANG_ASSERT(!"Not a string type");
+    return String();
+}
+
+UnownedStringSlice PersistentJSONValue::getSlice() const
+{
+    if (type == Type::StringRepresentation)
+    {
+        return StringRepresentation::asSlice(stringRep);
+    }
+    SLANG_ASSERT(!"Not a string type");
+    return UnownedStringSlice();
+}
+
+void PersistentJSONValue::set(const UnownedStringSlice& slice, SourceLoc inLoc)
+{
+    StringRepresentation* oldRep = (type == JSONValue::Type::StringRepresentation) ? stringRep : nullptr;
+
+    type = Type::StringRepresentation;
+    loc = inLoc;
+
+    StringRepresentation* newRep = nullptr;
+
+    const auto sliceLength = slice.getLength();
+    
+    // If we have an oldRep that is unique and large enough reuse it
+    if (sliceLength)
+    {
+        if (oldRep &&
+            oldRep->isUniquelyReferenced() &&
+            sliceLength <= oldRep->capacity)
+        {
+            oldRep->setContents(slice);
+            newRep = oldRep;
+            // We are reusing so make null so not freed
+            oldRep = nullptr;
+        }
+        else
+        {
+            newRep = StringRepresentation::createWithReference(slice);    
+        }
+
+        SLANG_ASSERT(newRep->debugGetReferenceCount() >= 1);
+    }
+
+    stringRep = newRep;
+
+    if (oldRep)
+    {
+        oldRep->releaseReference();
+    }
+}
+
+void PersistentJSONValue::_init(const UnownedStringSlice& slice, SourceLoc inLoc)
+{
+    loc = inLoc;
+    type = Type::StringRepresentation;
+    stringRep = StringRepresentation::createWithReference(slice);
+}
+
+bool PersistentJSONValue::operator==(const ThisType& rhs) const
+{
+    if (this == &rhs)
+    {
+        return true;
+    }
+
+    if (type != rhs.type ||
+        loc != rhs.loc)
+    {
+        return false;
+    }
+
+    switch (type)
+    {
+        case Type::Invalid:
+        case Type::True:
+        case Type::False:
+        case Type::Null:
+        {
+            // The type is all that needs to be checked
+            return true;
+        }
+        case Type::IntegerValue:    return intValue == rhs.intValue;
+        case Type::FloatValue:      return floatValue == rhs.floatValue;
+        case Type::StringRepresentation:
+        {
+            if (stringRep == rhs.stringRep)
+            {
+                return true;
+            }
+            auto thisSlice = StringRepresentation::asSlice(stringRep);
+            auto rhsSlice = StringRepresentation::asSlice(rhs.stringRep);
+            return thisSlice == rhsSlice;
+        }
+        default: break;
+    }
+
+    SLANG_ASSERT(!"Not valid Persistent type");
+    return false;
+}
+
+void PersistentJSONValue::_init(const JSONValue& in, JSONContainer* container)
+{
+    // We are assuming this is invalid, so it can't be the same as in
+    SLANG_ASSERT(&in != this);
+
+    switch (in.type)
+    {
+        case Type::StringValue:
+        case Type::StringLexeme:
+        {
+            if (!container)
+            {
+                SLANG_ASSERT(!"Requires container");
+                return;
+            }
+            _init(container->getTransientString(in), in.loc);
+            break;
+        }
+        case Type::StringRepresentation:
+        {
+            *(JSONValue*)this = in;
+            if (stringRep)
+            {
+                stringRep->addReference();
+            }
+            break;
+        }
+        case Type::IntegerLexeme:
+        {
+            type = JSONValue::Type::IntegerValue;
+            intValue = container->asInteger(in);
+            loc = in.loc;
+            break;
+        }
+        case Type::FloatLexeme:
+        {
+            type = JSONValue::Type::FloatValue;
+            floatValue = container->asFloat(in);
+            loc = in.loc;
+            break;
+        }
+        case Type::Array:
+        case Type::Object:
+        {
+            SLANG_ASSERT(!"Not a simple JSON type");
+            break;
+        }
+        default:
+        {
+            *(JSONValue*)this = in;
+            break;
+        }
+    }
+}
+
+void PersistentJSONValue::set(const JSONValue& in, JSONContainer* container)
+{
+    if (&in != this)
+    {
+        if (type == Type::StringRepresentation)
+        {
+            StringRepresentation* oldStringRep = stringRep;
+            _init(in, container);
+            if (oldStringRep)
+            {
+                oldStringRep->releaseReference();
+            }
+        }
+        else
+        {
+            _init(in, container);
+        }
+    }
+}
+
+/* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+
                              JSONContainer
 
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! */
@@ -279,8 +496,8 @@ ConstArrayView<JSONValue> JSONContainer::getArray(const JSONValue& in) const
 
 ConstArrayView<JSONKeyValue> JSONContainer::getObject(const JSONValue& in) const
 {
-    SLANG_ASSERT(in.type == JSONValue::Type::Array);
-    if (in.type != JSONValue::Type::Array || in.rangeIndex == 0)
+    SLANG_ASSERT(in.type == JSONValue::Type::Object);
+    if (in.type != JSONValue::Type::Object || in.rangeIndex == 0)
     {
         return ConstArrayView<JSONKeyValue>((const JSONKeyValue*)nullptr, 0);
     }
@@ -339,17 +556,25 @@ UnownedStringSlice JSONContainer::getLexeme(const JSONValue& in)
 
 UnownedStringSlice JSONContainer::getString(const JSONValue& in)
 {
-    if (in.type == JSONValue::Type::StringValue)
-    {
-        return getStringFromKey(in.stringKey);
-    }
-    else if (in.type == JSONValue::Type::StringLexeme)
+    switch (in.type)
     {
-        auto slice = getTransientString(in);
-        auto handle = m_slicePool.add(slice);
-        return m_slicePool.getSlice(handle);
+        case JSONValue::Type::StringValue:
+        {
+            return getStringFromKey(in.stringKey);
+        }
+        case JSONValue::Type::StringLexeme:
+        {
+            auto slice = getTransientString(in);
+            auto handle = m_slicePool.add(slice);
+            return m_slicePool.getSlice(handle);
+        }
+        case JSONValue::Type::StringRepresentation:
+        {
+            return StringRepresentation::asSlice(in.stringRep);
+        }
+        default: break;
     }
-
+   
     SLANG_ASSERT(!"Not a string type");
     return UnownedStringSlice();
 }
@@ -358,6 +583,10 @@ UnownedStringSlice JSONContainer::getTransientString(const JSONValue& in)
 {
     switch (in.type)
     {
+        case JSONValue::Type::StringRepresentation:
+        {
+            return StringRepresentation::asSlice(in.stringRep);
+        }
         case JSONValue::Type::StringValue:
         {
             return getStringFromKey(in.stringKey);
@@ -917,7 +1146,11 @@ bool JSONContainer::areEqual(const JSONValue& a, const JSONValue& b)
             case JSONValue::Type::IntegerValue: return a.intValue == b.intValue;
             case JSONValue::Type::FloatValue:   return a.floatValue == b.floatValue;
             case JSONValue::Type::StringValue:  return a.stringKey == b.stringKey;
-
+            case JSONValue::Type::StringRepresentation:
+            {
+                return a.stringRep == b.stringRep ||
+                    StringRepresentation::asSlice(a.stringRep) == StringRepresentation::asSlice(b.stringRep);
+            }
             case JSONValue::Type::Array:
             {
                 if (a.rangeIndex == b.rangeIndex)
@@ -982,6 +1215,10 @@ void JSONContainer::traverseRecursively(const JSONValue& value, JSONListener* li
             const auto slice = getStringFromKey(value.stringKey);
             return listener->addStringValue(slice, value.loc);
         }
+        case Type::StringRepresentation:
+        {
+            return listener->addStringValue(getTransientString(value), value.loc);
+        }
         case Type::Array:
         {
             listener->startArray(value.loc);
diff --git a/source/compiler-core/slang-json-value.h b/source/compiler-core/slang-json-value.h
index 2e8707a87..25e74cb36 100644
--- a/source/compiler-core/slang-json-value.h
+++ b/source/compiler-core/slang-json-value.h
@@ -50,6 +50,8 @@ struct JSONValue
         FloatValue,
         StringValue,
 
+        StringRepresentation,
+
         Array,
         Object,
 
@@ -105,14 +107,14 @@ struct JSONValue
 
     union 
     {
-        Index rangeIndex;           ///< Used for Array/Object
-        Index length;               ///< Length in bytes if it is a 'Lexeme'
-        double floatValue;          ///< Float value
-        int64_t intValue;           ///< Integer value
-        JSONKey stringKey;          ///< The pool key if it's a string
+        Index rangeIndex;                       ///< Used for Array/Object
+        Index length;                           ///< Length in bytes if it is a 'Lexeme'
+        double floatValue;                      ///< Float value
+        int64_t intValue;                       ///< Integer value
+        JSONKey stringKey;                      ///< The pool key if it's a string
+        StringRepresentation* stringRep;        ///< Only ever used on a 'PersistentJSONValue'
     };
 
-
     static const Kind g_typeToKind[Index(Type::CountOf)];
 
     static const OtherRttiInfo g_rttiInfo;
@@ -145,6 +147,65 @@ struct JSONKeyValue
     static JSONKeyValue g_invalid;
 };
 
+class JSONContainer;
+
+/* Is similar to JSONValue, but is designed to
+
+* Only be able to hold 'Simple' types (ie not array/object)
+* Does not reference/require JSONContainer.
+
+Not requiring JSONContainer means it's useful to hold state when JSONContainer goes out of scope.
+Care may need to be taken if sourceManager goes out of scope, sourceLocs may become invalid. This
+is true of a regular JSONValue.
+
+Care must also be taken because it is derived from JSONValue. It *can* be sliced and work correctly,
+but *requires* that the PersistentJSONValue with same value to stay in scope in general. In practice
+this is only an issue with StringRepresention type.
+*/
+class PersistentJSONValue : public JSONValue
+{
+public:
+    typedef JSONValue Super;
+    typedef PersistentJSONValue ThisType;
+
+        /// If it's a string type this will always work
+    String getString() const;
+    UnownedStringSlice getSlice() const;
+
+        /// Set to the value
+    void set(const JSONValue& in, JSONContainer* container);
+        /// Set directly to a string
+    void set(const UnownedStringSlice& slice, SourceLoc loc);
+
+        /// True if identical
+    bool operator==(const ThisType& rhs) const;
+    bool operator!=(const ThisType& rhs) const { return !(*this == rhs); }
+
+        /// Assignable
+    void operator=(const ThisType& rhs);
+
+    PersistentJSONValue(const JSONValue& in, JSONContainer* container) { _init(in, container); }
+    PersistentJSONValue(const JSONValue& in, JSONContainer* container, SourceLoc inLoc) { _init(in, container); loc = inLoc; }
+
+        /// Copy Ctor
+    PersistentJSONValue(const ThisType& rhs);
+        /// Default Ctor (will be set to invalid)
+    PersistentJSONValue() {}
+
+    
+    ~PersistentJSONValue()
+    {
+        if (type == Type::StringRepresentation && stringRep)
+        {
+            stringRep->releaseReference();
+        }
+    }
+protected:
+        /// Assumes this has no valid data
+    void _init(const JSONValue& in, JSONContainer* container);
+    void _init(const UnownedStringSlice& slice, SourceLoc loc);
+};
+
 class JSONContainer : public RefObject
 {
 public:
@@ -170,7 +231,7 @@ public:
 
         /// Returns the index of key in obj, or -1 if not found
     Index findObjectIndex(const JSONValue& obj, JSONKey key) const;
-        /// Get the value in the object at key. REturns invalid if not found.
+        /// Get the value in the object at key. Returns invalid if not found.
     JSONValue findObjectValue(const JSONValue& obj, JSONKey key) const;
 
         /// Returns the index 
@@ -201,7 +262,7 @@ public:
         /// Get as a string. The contents will stay in scope as long as the container
     UnownedStringSlice getString(const JSONValue& in);
 
-    /// Gets the lexeme
+        /// Gets the lexeme
     UnownedStringSlice getLexeme(const JSONValue& in);
 
         /// Get a key for a name
diff --git a/source/core/slang-string.cpp b/source/core/slang-string.cpp
index e21333809..1a6221cc8 100644
--- a/source/core/slang-string.cpp
+++ b/source/core/slang-string.cpp
@@ -11,6 +11,66 @@ namespace Slang
     // for anything that uses core
     static const auto s_charUtilLink = CharUtil::_ensureLink();
 
+
+    // StringRepresentation
+
+    void StringRepresentation::setContents(const UnownedStringSlice& slice)
+    {
+        const auto sliceLength = slice.getLength();
+        SLANG_ASSERT(sliceLength <= capacity);
+
+        char* chars = getData();
+
+        // Use move (rather than memcpy), because the slice *could* be contained in the StringRepresentation
+        ::memmove(chars, slice.begin(), sliceLength * sizeof(char));
+        // Zero terminate. 
+        chars[sliceLength] = 0;
+        // Set the length
+        length = sliceLength;
+    }
+
+
+    /* static */StringRepresentation* StringRepresentation::create(const UnownedStringSlice& slice)
+    {
+        const auto sliceLength = slice.getLength();
+
+        if (sliceLength)
+        {
+            StringRepresentation* rep = StringRepresentation::createWithLength(sliceLength);
+         
+            char* chars = rep->getData();
+            ::memcpy(chars, slice.begin(), sizeof(char) * sliceLength);
+            chars[sliceLength] = 0;
+
+            return rep;
+        }
+        else
+        {
+            return nullptr;
+        }
+    }
+
+    /* static */StringRepresentation* StringRepresentation::createWithReference(const UnownedStringSlice& slice)
+    {
+        const auto sliceLength = slice.getLength();
+
+        if (sliceLength)
+        {
+            StringRepresentation* rep = StringRepresentation::createWithLength(sliceLength);
+            rep->addReference();
+
+            char* chars = rep->getData();
+            ::memcpy(chars, slice.begin(), sizeof(char) * sliceLength);
+            chars[sliceLength] = 0;
+
+            return rep;
+        }
+        else
+        {
+            return nullptr;
+        }
+    }
+
     // OSString
 
     OSString::OSString()
diff --git a/source/core/slang-string.h b/source/core/slang-string.h
index 85f7b894f..cb13ec415 100644
--- a/source/core/slang-string.h
+++ b/source/core/slang-string.h
@@ -210,6 +210,9 @@ namespace Slang
             return (const char*)(this + 1);
         }
 
+            /// Set the contents to be the slice. Must be enough capacity to hold the slice. 
+        void setContents(const UnownedStringSlice& slice);
+
         static const char* getData(const StringRepresentation* stringRep)
         {
             return stringRep ? stringRep->getData() : "";
@@ -246,6 +249,11 @@ namespace Slang
             return createWithCapacityAndLength(length, length);
         }
 
+            /// Create a representation from the slice. If slice is empty will return nullptr.
+        static StringRepresentation* create(const UnownedStringSlice& slice);
+            /// Same as create, but representation will have refcount of 1 (if not nullptr)
+        static StringRepresentation* createWithReference(const UnownedStringSlice& slice);
+
         StringRepresentation* cloneWithCapacity(Index newCapacity)
         {
             StringRepresentation* newObj = createWithCapacityAndLength(newCapacity, length);