JSON-RPC test server (#2026)

* #include an absolute path didn't work - because paths were taken to always be relative.

* Use 'Process' to communicate with an command line tool.

* Remove slang-win-stream

* Tidy up windows ProcessUtil.

* First version of BufferedReadStream.

* Windows working IPC for steams.

* Test proxy count option.

* Split Process/ProcessUtil. Process is platform dependant. ProcessUtil are functions that are platform independent.

* First implementation of Unix Process interface.

* Unix process compiles on cygwin.

* Fix typo in unix process.

* Separate unix pipe stream error of invalid access, from pipe availability.

* Fix in standard line extraction.

* Make fd non blocking.

* Fix issues with Windows Process streams.

* Added UnixPipe.

* Some fixes around UnixPipeStream.

* Make a unix stream closed explicit.

* Hack to debug linux process/stream.

* Revert to old linux pipe handling.

* Pass executable path for unit tests.
Split out CommandLine into own source.

* Small improvements in process/command line.

* Check process behavior with crash.

* Make stderr and stdout unbuffered for crash testing.

* Only turn disable buffering in crash test.

* Disable crash test on CI.

* Fix crash on clang/linux.

* Enable crash test.
Remove _appendBuffer as can use StreamUtil functionality.

* Added inital processing for http headers.

* Small improvements to HttpHeader.

* First pass HTTPPacketConnection working on windows.

* Enable other Process communication tests.

* Update comments.

* WIP JSON RPC.

* Add terminate to Process.
Made JSONRPC a Util.

* Small tidy up around HTTPPacketConnection.

* Improve process termination options.

* WIP for test-server.

* Add diagnostics error handling to test-server.

* Improved JSON support.
Parsing/creating JSON-RPC messages.

* WIP JSONRPC parsing.

* First pass RttiInfo support.

* WIP converting between JSON/native types.

* Project files.

* Split out RttiUtil.
Made RttiInfo constuction thread safe.

* WIP RTTI<->JSON.

* Add diagnostics to JSON<->native conversions.

* Make RttiInfo for structs globals. Avoids problem around derived types (like pointers), being able to cause an abort.

* Add pointer support to RTTI.
Fixed some compilation issues on linux.

* Add fixed array support.

* Added Rtti unit test.

* Add rtti unit test.

* Split out quoted/unquoted key handling.
Fix bugs in JSON value/container.
Added JSON native test.

* Make default array allocator use malloc/free.
Remove the new[] handler (doesn't work on visuals studio).

* Fix for linux warning.

* Remove some test code.

* Fix issues on x86 win.

* Fix warning on aarch64.

* Fix some bugs in JSON parsing/handling.
Make Rtti work copy/dtor/ctor struct types.

* Testing JSON<->native with fixed array.
Make makeArrayView explicit if it's just a single value.
Added array type.

* Fix getting arrayView.

* Improve JSON diagnostic name.

* First pass refactor using Rtti for JSON RPC.

* First pass of test server using RTTI/JSON-RPC.

* Added JSONRPCConnection.

* Fix some naming issues.

* First pass of test-server working.

* Added unit test support for JSON-RPC test server.

* Fix compilation issues on linux around template handling.

* Typo fix.

* Fix a bug around SourceLoc lookup with JSONContainer.

* Set the console type to console for ISlangWriters.

* Small improvements to test-server.

* Small improvements in test-server.

* Small fix.

1 files changed, 63 insertions, 341 deletions

diff --git a/source/compiler-core/slang-json-rpc.cpp b/source/compiler-core/slang-json-rpc.cpp
index 1ca55e9eb..660bbd4b7 100644
--- a/source/compiler-core/slang-json-rpc.cpp
+++ b/source/compiler-core/slang-json-rpc.cpp
@@ -2,23 +2,20 @@
 
 #include "../../slang-com-helper.h"
 
+#include "slang-json-native.h"
+
 namespace Slang {
 
 // https://www.jsonrpc.org/specification
 
-//    m_sourceManager.initialize(nullptr, nullptr);
-//    m_diagnosticSink.init(&m_sourceManager, &JSONLexer::calcLexemeLocation);
 
-static const auto g_jsonRpc = UnownedStringSlice::fromLiteral("jsonrpc");
-static const auto g_jsonRpcVersion = UnownedStringSlice::fromLiteral("2.0");
-static const auto g_method = UnownedStringSlice::fromLiteral("method");
-static const auto g_id = UnownedStringSlice::fromLiteral("id");
-static const auto g_params = UnownedStringSlice::fromLiteral("params");
-static const auto g_code = UnownedStringSlice::fromLiteral("code");
-static const auto g_error = UnownedStringSlice::fromLiteral("error");
-static const auto g_message = UnownedStringSlice::fromLiteral("message");
+/* static */ const UnownedStringSlice JSONRPC::jsonRpc = UnownedStringSlice::fromLiteral("jsonrpc");
+/* static */const UnownedStringSlice JSONRPC::jsonRpcVersion = UnownedStringSlice::fromLiteral("2.0");
+/* static */const UnownedStringSlice JSONRPC::id = UnownedStringSlice::fromLiteral("id");
+
 static const auto g_result = UnownedStringSlice::fromLiteral("result");
-static const auto g_data = UnownedStringSlice::fromLiteral("data");
+static const auto g_error = UnownedStringSlice::fromLiteral("error");
+static const auto g_method = UnownedStringSlice::fromLiteral("method");
 
 // Add the fields.
 // TODO(JS): This is a little verbose, and could be improved on with something like
@@ -39,9 +36,10 @@ static const StructRttiInfo _makeJSONRPCErrorResponseRtti()
     JSONRPCErrorResponse obj;
     StructRttiBuilder builder(&obj, "JSONRPCErrorResponse", nullptr);
 
+    builder.addField("jsonrpc", &obj.jsonrpc);
     builder.addField("error", &obj.error);
     builder.addField("data", &obj.data, StructRttiInfo::Flag::Optional);
-    builder.addField("id", &obj.id, combine(StructRttiInfo::Flag::Optional, RttiDefaultValue::MinusOne));
+    builder.addField("id", &obj.id, StructRttiInfo::Flag::Optional);
 
     return builder.make();
 }
@@ -52,9 +50,10 @@ static const StructRttiInfo _makeJSONRPCCallResponseRtti()
     JSONRPCCall obj;
     StructRttiBuilder builder(&obj, "JSONRPCCall", nullptr);
 
+    builder.addField("jsonrpc", &obj.jsonrpc);
     builder.addField("method", &obj.method);
     builder.addField("params", &obj.params, StructRttiInfo::Flag::Optional);
-    builder.addField("id", &obj.id, combine(StructRttiInfo::Flag::Optional, RttiDefaultValue::MinusOne));
+    builder.addField("id", &obj.id, StructRttiInfo::Flag::Optional);
 
     return builder.make();
 }
@@ -65,350 +64,44 @@ static const StructRttiInfo _makeJSONResultResponseResponseRtti()
     JSONResultResponse obj;
     StructRttiBuilder builder(&obj, "JSONResultResponse", nullptr);
 
+    builder.addField("jsonrpc", &obj.jsonrpc);
     builder.addField("result", &obj.result);
-    builder.addField("id", &obj.id, combine(StructRttiInfo::Flag::Optional, RttiDefaultValue::MinusOne));
+    builder.addField("id", &obj.id, StructRttiInfo::Flag::Optional);
 
     return builder.make();
 }
 /* static */const StructRttiInfo JSONResultResponse::g_rttiInfo = _makeJSONResultResponseResponseRtti();
 
-
-/* static */JSONValue JSONRPCUtil::createCall(JSONContainer* container, const UnownedStringSlice& method, JSONValue params, Int id)
-{
-    const Index maxPairs = 4;
-    JSONKeyValue pairs[maxPairs];
-
-    Index i = 0;
-
-    // Version number is a string
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_jsonRpc), container->createString(g_jsonRpcVersion));
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_method), container->createString(method));
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_params), params);
-
-    if (id >= 0)
-    {
-        pairs[i++] = JSONKeyValue::make(container->getKey(g_id), JSONValue::makeInt(id));
-    }
-
-    return container->createObject(pairs, i);
-}
-
-/* static */JSONValue JSONRPCUtil::createCall(JSONContainer* container, const UnownedStringSlice& method, Int id)
-{
-    const Index maxPairs = 3;
-    JSONKeyValue pairs[maxPairs];
-    Index i = 0;
-    // Version number is a string
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_jsonRpc), container->createString(g_jsonRpcVersion));
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_method), container->createString(method));
-
-    if (id >= 0)
-    {
-        pairs[i++] = JSONKeyValue::make(container->getKey(g_id), JSONValue::makeInt(id));
-    }
-
-    return container->createObject(pairs, i);
-}
-
-/* static */JSONValue JSONRPCUtil::createErrorResponse(JSONContainer* container, Index code, const UnownedStringSlice& message, const JSONValue& data, Int id)
-{
-    // Set up the error value
-    JSONValue errorValue;
-    {
-        const Index maxPairs = 2;
-        JSONKeyValue pairs[maxPairs];
-        Index i = 0;
-
-        if (code != 0)
-        {
-            pairs[i++] = JSONKeyValue::make(container->getKey(g_code), JSONValue::makeInt(code));
-        }
-        if (message.getLength() > 0)
-        {
-            pairs[i++] = JSONKeyValue::make(container->getKey(g_message), container->createString(message));
-        }
-        errorValue = container->createObject(pairs, i);
-    }
-
-    const Index maxPairs = 4;
-    JSONKeyValue pairs[maxPairs];
-    Index i = 0;
-
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_jsonRpc), container->createString(g_jsonRpcVersion));
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_error), errorValue);
-
-    if (data.isValid())
-    {
-        pairs[i++] = JSONKeyValue::make(container->getKey(g_data), data);
-    }
-
-    if (id >= 0)
-    {
-        pairs[i++] = JSONKeyValue::make(container->getKey(g_id), JSONValue::makeInt(id));
-    }
-
-    return container->createObject(pairs, i);
-}
-
-
-/* static */JSONValue JSONRPCUtil::createErrorResponse(JSONContainer* container, ErrorCode code, const UnownedStringSlice& message, const JSONValue& data, Int id)
-{
-    return createErrorResponse(container, Index(code), message, data, id);
-}
-
-/* static */JSONValue JSONRPCUtil::createResultResponse(JSONContainer* container, const JSONValue& resultValue, Int id)
+/* static */JSONRPCMessageType JSONRPCUtil::getMessageType(JSONContainer* container, const JSONValue& value)
 {
-    const Index maxPairs = 3;
-    JSONKeyValue pairs[maxPairs];
-    Index i = 0;
-
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_jsonRpc), container->createString(g_jsonRpcVersion));
-    pairs[i++] = JSONKeyValue::make(container->getKey(g_result), resultValue);
-
-    if (id >= 0)
-    {
-        pairs[i++] = JSONKeyValue::make(container->getKey(g_id), JSONValue::makeInt(id));
-    }
-
-    return container->createObject(pairs, i);
-}
-
-/* static */ JSONRPCUtil::ResponseType JSONRPCUtil::getResponseType(JSONContainer* container, const JSONValue& response)
-{
-    if (response.getKind() == JSONValue::Kind::Object)
+    if (value.getKind() == JSONValue::Kind::Object)
     {
         const JSONKey resultKey = container->findKey(g_result);
         const JSONKey errorKey = container->findKey(g_error);
+        const JSONKey methodKey = container->findKey(g_method);
 
-        auto pairs = container->getObject(response);
+        auto pairs = container->getObject(value);
 
         for (const auto& pair : pairs)
         {
             if (pair.key == resultKey)
             {
-                return ResponseType::Result;
+                return JSONRPCMessageType::Result;
             }
             else if (pair.key == errorKey)
             {
-                return ResponseType::Error;
-            }
-        }
-    }
-
-    return ResponseType::Error;
-}
-
-static SlangResult _parseError(JSONContainer* container, const JSONValue& error, JSONRPCUtil::ErrorResponse& out)
-{
-    if (error.getKind() != JSONValue::Kind::Object)
-    {
-        return SLANG_FAIL;
-    }
-    const auto pairs = container->getObject(error);
-
-    const JSONKey messageKey = container->findKey(g_message);
-    const JSONKey codeKey = container->findKey(g_code);
-    const JSONKey dataKey = container->findKey(g_data);
-
-    Int fieldBits = 0;
-
-    for (auto const& pair : pairs)
-    {
-        if (pair.key == messageKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::String)
-            {
-                return SLANG_FAIL;
-            }
-            out.message = container->getString(pair.value);
-            fieldBits |= 0x1;
-        }
-        else if (pair.key == codeKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::Integer)
-            {
-                return SLANG_FAIL;
-            }
-            out.code = Index(pair.value.asInteger());
-            fieldBits |= 0x2;
-        }
-        else if (pair.key == dataKey)
-        {
-            out.data = pair.value;
-            fieldBits |= 0x4;
-        }
-        else
-        {
-            return SLANG_FAIL;
-        }
-    }
-
-    // Check all required fields are set
-    return (fieldBits & 0x3) == 0x3 ? SLANG_OK : SLANG_FAIL;
-}
-
-/* static */SlangResult JSONRPCUtil::parseError(JSONContainer* container, const JSONValue& response, ErrorResponse& out)
-{
-    if (response.getKind() != JSONValue::Kind::Object)
-    {
-        return SLANG_FAIL;
-    }
-
-    const auto pairs = container->getObject(response);
-
-    const JSONKey jsonRpcKey = container->findKey(g_jsonRpc);
-    const JSONKey errorKey = container->findKey(g_error);
-    const JSONKey idKey = container->findKey(g_id);
-
-    Int fieldBits = 0;
-
-    for (auto const& pair : pairs)
-    {
-        if (pair.key == jsonRpcKey)
-        {
-            if (!container->areEqual(pair.value, g_jsonRpcVersion))
-            {
-                return SLANG_FAIL;
-            }
-            fieldBits |= 0x1;
-        }
-        else if (pair.key == errorKey)
-        {
-            // We need to decode the error 
-            SLANG_RETURN_ON_FAIL(_parseError(container, pair.value, out));
-            fieldBits |= 0x2;
-        }
-        else if (pair.key == idKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::Integer)
-            {
-                return SLANG_FAIL;
-            }
-            out.id = Int(pair.value.asInteger());
-            fieldBits |= 0x4;
-        }
-        else
-        {
-            // Unknown key
-            return SLANG_FAIL;
-        }
-    }
-
-    // Check all the required bits are set
-    return ((fieldBits & 0x3) == 0x3) ? SLANG_OK : SLANG_FAIL;
-}
-
-/* static */SlangResult JSONRPCUtil::parseResult(JSONContainer* container, const JSONValue& response, ResultResponse& out)
-{
-    if (response.getKind() != JSONValue::Kind::Object)
-    {
-        return SLANG_FAIL;
-    }
-
-    const auto pairs = container->getObject(response);
-
-    const JSONKey jsonRpcKey = container->findKey(g_jsonRpc);
-    const JSONKey resultKey = container->findKey(g_result);
-    const JSONKey idKey = container->findKey(g_id);
-
-    Int fieldBits = 0;
-
-    for (auto const& pair : pairs)
-    {
-        if (pair.key == jsonRpcKey)
-        {
-            if (!container->areEqual(pair.value, g_jsonRpcVersion))
-            {
-                return SLANG_FAIL;
-            }
-            fieldBits |= 0x1;
-        }
-        else if (pair.key == resultKey)
-        {
-            out.result = pair.value;
-            fieldBits |= 0x2;
-        }
-        else if (pair.key == idKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::Integer)
-            {
-                return SLANG_FAIL;
-            }
-            out.id = Int(pair.value.asInteger());
-            fieldBits |= 0x4;
-        }
-        else
-        {
-            // Unknown key
-            return SLANG_FAIL;
-        }
-    }
-
-    // Check all the required bits are set
-    return ((fieldBits & 0x3) == 0x3) ? SLANG_OK : SLANG_FAIL;
-}
-
-/* static */SlangResult JSONRPCUtil::parseCall(JSONContainer* container, const JSONValue& value, Call& out)
-{
-    if (value.getKind() != JSONValue::Kind::Object)
-    {
-        return SLANG_FAIL;
-    }
-
-    const auto pairs = container->getObject(value);
-
-    const JSONKey jsonRpcKey = container->findKey(g_jsonRpc);
-    const JSONKey methodKey = container->findKey(g_method);
-    const JSONKey paramsKey = container->findKey(g_params);
-    const JSONKey idKey = container->findKey(g_id);
-
-    Int fieldBits = 0;
-
-    for (auto const& pair : pairs)
-    {
-        if (pair.key == jsonRpcKey)
-        {
-            if (!container->areEqual(pair.value, g_jsonRpcVersion))
-            {
-                return SLANG_FAIL;
+                return JSONRPCMessageType::Error;
             }
-            fieldBits |= 0x1;
-        }
-        else if (pair.key == methodKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::String)
-            {
-                return SLANG_FAIL;
-            }
-            out.method = container->getString(pair.value);
-            fieldBits |= 0x2;
-        }
-        else if (pair.key == idKey)
-        {
-            if (pair.value.getKind() != JSONValue::Kind::Integer)
+            else if (pair.key == methodKey)
             {
-                return SLANG_FAIL;
+                return JSONRPCMessageType::Call;
             }
-            out.id = Int(pair.value.asInteger());
-            fieldBits |= 0x4;
-        }
-        else if (pair.key == paramsKey)
-        {
-            out.params = pair.value;
-            fieldBits |= 0x8;
-        }
-        else
-        {
-            // Unknown key
-            return SLANG_FAIL;
         }
     }
 
-    // Check all the required bits are set
-    return ((fieldBits & 0x3) == 0x3) ? SLANG_OK : SLANG_FAIL;
+    return JSONRPCMessageType::Invalid;
 }
 
-
 /* static */SlangResult JSONRPCUtil::parseJSON(const UnownedStringSlice& slice, JSONContainer* container, DiagnosticSink* sink, JSONValue& outValue)
 {
     SourceManager* sourceManager = sink->getSourceManager();
@@ -430,21 +123,50 @@ static SlangResult _parseError(JSONContainer* container, const JSONValue& error,
     return SLANG_OK;
 }
 
-SlangResult JSONRPCUtil::parseJSONAndConsume(HTTPPacketConnection* connection, JSONContainer* container, DiagnosticSink* sink, JSONValue& outValue)
+/* static */SlangResult JSONRPCUtil::convertToNative(JSONContainer* container, const JSONValue& value, DiagnosticSink* sink, const RttiInfo* rttiInfo, void* out)
 {
-    if (!connection->hasContent())
-    {
-        return SLANG_FAIL;
-    }
+    JSONToNativeConverter converter(container, sink);
+    SLANG_RETURN_ON_FAIL(converter.convert(value, rttiInfo, out));
+    return SLANG_OK;
+}
 
-    auto content = connection->getContent();
+/* static */SlangResult JSONRPCUtil::convertToJSON(const RttiInfo* rttiInfo, const void* in, DiagnosticSink* sink, StringBuilder& out)
+{
+    SourceManager* sourceManager = sink->getSourceManager();
+    JSONContainer container(sourceManager);
 
-    UnownedStringSlice text((const char*)content.begin(), content.getCount());
-    SlangResult res = parseJSON(text, container, sink, outValue);
+    NativeToJSONConverter converter(&container, sink);
 
-    // Consume the content
-    connection->consumeContent();
-    return res;
+    JSONValue value;
+    SLANG_RETURN_ON_FAIL(converter.convert(rttiInfo, in, value));
+
+    // Convert into a string
+    JSONWriter writer(JSONWriter::IndentationStyle::Allman);
+    container.traverseRecursively(value, &writer);
+
+    out = writer.getBuilder();
+    return SLANG_OK;
 }
 
+/* static */JSONValue JSONRPCUtil::getId(JSONContainer* container, const JSONValue& root)
+{
+    if (root.getKind() == JSONValue::Kind::Object)
+    {
+        const JSONKey key = container->findKey(JSONRPC::id);
+
+        if (key != JSONKey(0))
+        {
+            auto obj = container->getObject(root);
+            Index index = obj.findFirstIndex([key](const JSONKeyValue& pair) -> bool { return pair.key == key; });
+
+            if (index >= 0)
+            {
+                return obj[index].value;
+            }
+        }
+    }
+    return JSONValue();
+}
+
+
 } // namespace Slang