JSON-RPC test server (#2026)

* #include an absolute path didn't work - because paths were taken to always be relative.

* Use 'Process' to communicate with an command line tool.

* Remove slang-win-stream

* Tidy up windows ProcessUtil.

* First version of BufferedReadStream.

* Windows working IPC for steams.

* Test proxy count option.

* Split Process/ProcessUtil. Process is platform dependant. ProcessUtil are functions that are platform independent.

* First implementation of Unix Process interface.

* Unix process compiles on cygwin.

* Fix typo in unix process.

* Separate unix pipe stream error of invalid access, from pipe availability.

* Fix in standard line extraction.

* Make fd non blocking.

* Fix issues with Windows Process streams.

* Added UnixPipe.

* Some fixes around UnixPipeStream.

* Make a unix stream closed explicit.

* Hack to debug linux process/stream.

* Revert to old linux pipe handling.

* Pass executable path for unit tests.
Split out CommandLine into own source.

* Small improvements in process/command line.

* Check process behavior with crash.

* Make stderr and stdout unbuffered for crash testing.

* Only turn disable buffering in crash test.

* Disable crash test on CI.

* Fix crash on clang/linux.

* Enable crash test.
Remove _appendBuffer as can use StreamUtil functionality.

* Added inital processing for http headers.

* Small improvements to HttpHeader.

* First pass HTTPPacketConnection working on windows.

* Enable other Process communication tests.

* Update comments.

* WIP JSON RPC.

* Add terminate to Process.
Made JSONRPC a Util.

* Small tidy up around HTTPPacketConnection.

* Improve process termination options.

* WIP for test-server.

* Add diagnostics error handling to test-server.

* Improved JSON support.
Parsing/creating JSON-RPC messages.

* WIP JSONRPC parsing.

* First pass RttiInfo support.

* WIP converting between JSON/native types.

* Project files.

* Split out RttiUtil.
Made RttiInfo constuction thread safe.

* WIP RTTI<->JSON.

* Add diagnostics to JSON<->native conversions.

* Make RttiInfo for structs globals. Avoids problem around derived types (like pointers), being able to cause an abort.

* Add pointer support to RTTI.
Fixed some compilation issues on linux.

* Add fixed array support.

* Added Rtti unit test.

* Add rtti unit test.

* Split out quoted/unquoted key handling.
Fix bugs in JSON value/container.
Added JSON native test.

* Make default array allocator use malloc/free.
Remove the new[] handler (doesn't work on visuals studio).

* Fix for linux warning.

* Remove some test code.

* Fix issues on x86 win.

* Fix warning on aarch64.

* Fix some bugs in JSON parsing/handling.
Make Rtti work copy/dtor/ctor struct types.

* Testing JSON<->native with fixed array.
Make makeArrayView explicit if it's just a single value.
Added array type.

* Fix getting arrayView.

* Improve JSON diagnostic name.

* First pass refactor using Rtti for JSON RPC.

* First pass of test server using RTTI/JSON-RPC.

* Added JSONRPCConnection.

* Fix some naming issues.

* First pass of test-server working.

* Added unit test support for JSON-RPC test server.

* Fix compilation issues on linux around template handling.

* Typo fix.

* Fix a bug around SourceLoc lookup with JSONContainer.

* Set the console type to console for ISlangWriters.

* Small improvements to test-server.

* Small improvements in test-server.

* Small fix.

1 files changed, 286 insertions, 0 deletions

diff --git a/source/compiler-core/slang-json-rpc-connection.cpp b/source/compiler-core/slang-json-rpc-connection.cpp
new file mode 100644
index 000000000..20283070b
--- /dev/null
+++ b/source/compiler-core/slang-json-rpc-connection.cpp
@@ -0,0 +1,286 @@
+// slang-json-rpc-connection.cpp
+#include "slang-json-rpc-connection.h"
+
+#include "../core/slang-string-util.h"
+#include "../core/slang-process-util.h"
+
+#include "slang-json-rpc.h"
+#include "slang-json-native.h"
+
+namespace Slang {
+
+SlangResult JSONRPCConnection::init(HTTPPacketConnection* connection, Process* process)
+{
+    m_connection = connection;
+    m_process = process;
+
+    m_sourceManager.initialize(nullptr, nullptr);
+    m_diagnosticSink.init(&m_sourceManager, &JSONLexer::calcLexemeLocation);
+    m_container.setSourceManager(&m_sourceManager);
+
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::initWithStdStreams(Process* process)
+{
+    RefPtr<Stream> stdinStream, stdoutStream;
+
+    Process::getStdStream(Process::StreamType::StdIn, stdinStream);
+    Process::getStdStream(Process::StreamType::StdOut, stdoutStream);
+
+    RefPtr<BufferedReadStream> readStream(new BufferedReadStream(stdinStream));
+
+    RefPtr<HTTPPacketConnection> connection = new HTTPPacketConnection(readStream, stdoutStream);
+    return init(connection, process);
+}
+
+void JSONRPCConnection::clearBuffers()
+{
+    m_sourceManager.reset();
+    m_diagnosticSink.reset();
+    m_container.reset();
+    m_jsonRoot.reset();
+}
+
+bool JSONRPCConnection::isActive()
+{
+    return m_connection->isActive() && (m_process == nullptr || !m_process->isTerminated());
+}
+
+JSONValue JSONRPCConnection::getMessageId()
+{
+    SLANG_ASSERT(hasMessage());
+    return JSONRPCUtil::getId(&m_container, m_jsonRoot);
+}
+
+void JSONRPCConnection::disconnect()
+{
+    if (m_process)
+    {
+        if (!m_process->isTerminated())
+        {
+            if (m_connection)
+            {
+                // Send. If succeeded, wait 
+                if (SLANG_SUCCEEDED(sendCall(UnownedStringSlice::fromLiteral("quit"))))
+                {
+                    // Wait for termination
+                    m_process->waitForTermination(m_terminationTimeOutInMs);
+                }
+            }
+
+            if (!m_process->isTerminated())
+            {
+                // Okay, just try terminating
+                m_process->waitForTermination(m_terminationTimeOutInMs);
+            }
+
+            // Okay just kill it then
+            if (!m_process->isTerminated())
+            {
+                m_process->kill(-1);
+            }
+        }
+        m_process.setNull();
+    }
+
+    m_connection.setNull();
+}
+
+SlangResult JSONRPCConnection::sendRPC(const RttiInfo* rttiInfo, const void* data)
+{
+    // Convert to JSON
+    NativeToJSONConverter converter(&m_container, &m_diagnosticSink);
+    JSONValue value;
+
+    SLANG_RETURN_ON_FAIL(converter.convert(rttiInfo, data, value));
+
+    // Convert to text
+    JSONWriter writer(JSONWriter::IndentationStyle::Allman);
+
+    m_container.traverseRecursively(value, &writer);
+    const StringBuilder& builder = writer.getBuilder();
+    return m_connection->write(builder.getBuffer(), builder.getLength());
+}
+
+SlangResult JSONRPCConnection::sendError(JSONRPC::ErrorCode code)
+{
+    return sendError(code, m_diagnosticSink.outputBuffer.getUnownedSlice());
+}
+
+SlangResult JSONRPCConnection::sendError(JSONRPC::ErrorCode errorCode, const UnownedStringSlice& msg)
+{
+    JSONRPCErrorResponse errorResponse;
+    errorResponse.error.code = Int(errorCode);
+    errorResponse.error.message = msg;
+
+    // TODO(JS):
+    // This is only appropriate if the sendError is for the current input message.
+    // We might want to add function that the client uses, which take the id as a parameter.
+
+    if (m_jsonRoot.isValid())
+    {
+        errorResponse.id = JSONRPCUtil::getId(&m_container, m_jsonRoot);
+    }
+    else
+    {
+        // If we don't have valid json, we set the id to be null per the spec
+        errorResponse.id = JSONValue::makeNull();
+    }
+
+    return sendRPC(&errorResponse);
+}
+
+SlangResult JSONRPCConnection::toNativeOrSendError(const JSONValue& value, const RttiInfo* info, void* dst)
+{
+    m_diagnosticSink.outputBuffer.Clear();
+    if (SLANG_FAILED(JSONRPCUtil::convertToNative(&m_container, value, &m_diagnosticSink, info, dst)))
+    {
+        return sendError(JSONRPC::ErrorCode::InvalidRequest);
+    }
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::sendCall(const UnownedStringSlice& method, const JSONValue& id)
+{
+    JSONRPCCall call;
+    call.id = id;
+    call.method = method;
+
+    SLANG_RETURN_ON_FAIL(sendRPC(&call));
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::sendResult(const RttiInfo* rttiInfo, const void* result, const JSONValue& id)
+{
+    JSONResultResponse response;
+    response.id = id;
+
+    NativeToJSONConverter converter(&m_container, &m_diagnosticSink);
+    SLANG_RETURN_ON_FAIL(converter.convert(rttiInfo, result, response.result));
+
+    // Send the RPC
+    SLANG_RETURN_ON_FAIL(sendRPC(&response));
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::sendCall(const UnownedStringSlice& method, const RttiInfo* argsRttiInfo, const void* args, const JSONValue& id)
+{
+    JSONRPCCall call;
+    call.id = id;
+    call.method = method;
+
+    // Convert the args/params
+    NativeToJSONConverter converter(&m_container, &m_diagnosticSink);
+    SLANG_RETURN_ON_FAIL(converter.convert(argsRttiInfo, args, call.params));
+
+    // Send the RPC
+    SLANG_RETURN_ON_FAIL(sendRPC(&call));
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::waitForResult(Int timeOutInMs)
+{
+    SLANG_RETURN_ON_FAIL(m_connection->waitForResult(timeOutInMs));
+    return tryReadMessage();
+}
+
+SlangResult JSONRPCConnection::tryReadMessage()
+{
+    m_jsonRoot.reset();
+
+    SLANG_RETURN_ON_FAIL(m_connection->update());
+    if (!m_connection->hasContent())
+    {
+        return SLANG_OK;
+    }
+
+    auto content = m_connection->getContent();
+    UnownedStringSlice slice((const char*)content.begin(), content.getCount());
+
+    clearBuffers();
+
+    {
+        const SlangResult res = JSONRPCUtil::parseJSON(slice, &m_container, &m_diagnosticSink, m_jsonRoot);
+
+        // Consume that content/packet
+        m_connection->consumeContent();
+        if (SLANG_FAILED(res))
+        {
+            return sendError(JSONRPC::ErrorCode::ParseError);
+        }
+    }
+
+    return SLANG_OK;
+}
+
+JSONRPCMessageType JSONRPCConnection::getMessageType()
+{
+    return JSONRPCUtil::getMessageType(&m_container, m_jsonRoot);
+}
+
+SlangResult JSONRPCConnection::getMessage(const RttiInfo* rttiInfo, void* out)
+{
+    if (!hasMessage())
+    {
+        return SLANG_FAIL;
+    }
+
+    m_diagnosticSink.outputBuffer.Clear();
+    JSONToNativeConverter converter(&m_container, &m_diagnosticSink);
+
+    // Get the RPC response
+    JSONResultResponse resultResponse;
+    SLANG_RETURN_ON_FAIL(converter.convert(m_jsonRoot, &resultResponse));
+
+    // Convert the result in the response
+    SLANG_RETURN_ON_FAIL(converter.convert(resultResponse.result, rttiInfo, out));
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::getMessageOrSendError(const RttiInfo* rttiInfo, void* out)
+{
+    if (!hasMessage())
+    {
+        return SLANG_FAIL;
+    }
+
+    const auto res = getMessage(rttiInfo, out);
+    if (SLANG_FAILED(res))
+    {
+        return sendError(JSONRPC::ErrorCode::ParseError);
+    }
+    return res;
+}
+
+SlangResult JSONRPCConnection::getRPC(const RttiInfo* rttiInfo, void* out)
+{
+    if (!hasMessage())
+    {
+        return SLANG_FAIL;
+    }
+
+    m_diagnosticSink.outputBuffer.Clear();
+    JSONToNativeConverter converter(&m_container, &m_diagnosticSink);
+
+    // Convert the result in the response
+    SLANG_RETURN_ON_FAIL(converter.convert(m_jsonRoot, rttiInfo, out));
+    return SLANG_OK;
+}
+
+SlangResult JSONRPCConnection::getRPCOrSendError(const RttiInfo* rttiInfo, void* out)
+{
+    if (!hasMessage())
+    {
+        return SLANG_FAIL;
+    }
+
+    const auto res = getRPC(rttiInfo, out);
+    if (SLANG_FAILED(res))
+    {
+        return sendError(JSONRPC::ErrorCode::ParseError);
+    }
+    return res;
+}
+
+} // namespcae Slang