Handle debug-layer messages in a separate channel (#7988)

* Handle debug-layer messages in a separate channel

The Problem (Issue #7343)

The issue was that Vulkan Validation Layer error messages were being mixed into regular test output, causing
potential false positives or negatives. When using -enable-debug-layers true, validation messages would appear in
the same output stream as test results, potentially matching //CHECK: patterns incorrectly.

Example Problem:
- Test expects: //CHECK: 1
- Validation layer prints: VALIDATION ERROR: 1 invalid buffer binding
- Test incorrectly matches the "1" in the error message instead of the actual output

Slang Test Communication Architecture

Execution Modes

Slang has 3 different execution modes controlled by SpawnType:

enum class SpawnType {
    UseSharedLibrary,              // In-process execution
    UseTestServer,                 // Out-of-process via persistent server
    UseFullyIsolatedTestServer,    // Out-of-process via isolated server
    UseExe                         // Direct executable spawn
}

1. In-Process Mode (UseSharedLibrary)

┌─────────────────────────────────────────┐
│              slang-test                 │
│  ┌─────────────┐    ┌─────────────────┐ │
│  │render-test  │    │gfx-unit-test    │ │
│  │unit-test    │    │slangc library   │ │
│  └─────────────┘    └─────────────────┘ │
│           │                   │         │
│           └───► StdWriters ◄──┘         │
│                 (shared)                │
└─────────────────────────────────────────┘
- Communication: Direct function calls, shared memory
- Debug callbacks: Single callback instance in StdWriters
- Used when: Default mode for most tests

2. Out-of-Process Mode (UseTestServer)

┌──────────────────┐    JSON-RPC     ┌──────────────────┐
│   slang-test     │◄──over pipes────┤ test-server.exe  │
│                  │                 │                  │
│  ┌─────────────┐ │                 │ ┌─────────────┐  │
│  │StdWriters   │ │                 │ │render-test  │  │
│  │+debug       │ │                 │ │gfx-unit-test│  │
│  │callback     │ │                 │ │+debug       │  │
│  └─────────────┘ │                 │ │callback     │  │
└──────────────────┘                 │ └─────────────┘  │
                                     └──────────────────┘
- Communication: JSON-RPC over stdin/stdout pipes
- Debug callbacks: Separate instances in each process
- Used when: CI/CD, multi-threaded testing, crash isolation

3. Direct Executable Mode (UseExe)

┌──────────────────┐    pipes       ┌──────────────────┐
│   slang-test     │◄───────────────┤  slangc.exe      │
│                  │                │  other tools     │
└──────────────────┘                └──────────────────┘
- Communication: Standard process pipes (stdout/stderr)
- Debug callbacks: None (external executables)
- Used when: Testing external tools

Communication Mechanisms Deep Dive

JSON-RPC Protocol Over Pipes

The test-server.exe communicates with slang-test using JSON-RPC over stdin/stdout pipes:

// Parent process (slang-test) creates child with pipes
Process* testServerProcess = /* spawn test-server.exe */;

// JSONRPCConnection wraps the pipe communication
JSONRPCConnection connection;
connection.initWithStdStreams(); // Uses stdin/stdout pipes

// Send RPC call
TestServerProtocol::ExecutionResult result;
connection.sendCall("executeTool", &args, &result);

Key Point: The pipes carry structured JSON messages, not raw stdout/stderr. This is what enables clean separation
of different data channels.

Protocol Structure

Your changes extend the ExecutionResult protocol:

struct ExecutionResult {
    String stdOut;      // Regular program output
    String stdError;    // Error messages
    String debugLayer;  // NEW: Debug/validation messages
    int32_t result;
    int32_t returnCode;
};

Your Debug Layer Solution

The Challenge

Memory pointers cannot cross process boundaries. A debugCallback pointer in the parent process is meaningless in
the child process.

The Solution: String-Based Serialization

You solved this by using string capture and serialization:

1. Debug Callback Interface (slang-std-writers.h)

class IDebugCallback {
    virtual void handleMessage(
        DebugMessageType type,
        DebugMessageSource source,
        const char* message) = 0;
};

2. String-Capturing Implementation (slang-support.h)

class CoreDebugCallback : public Slang::IDebugCallback {
    StringBuilder m_buf;  // Captures messages as strings

    void handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) {
        if (type == DebugMessageType::Error) {
            m_buf << message << '\n';  // Serialize to string
        }
    }

    String getString() { return m_buf.toString(); }  // Extract accumulated messages
};

3. Bridge Between RHI and Core (slang-support.h)

class CoreToRHIDebugBridge : public rhi::IDebugCallback {
    Slang::IDebugCallback* m_coreCallback;

    void handleMessage(rhi::DebugMessageType type, rhi::DebugMessageSource source, const char* message) {
        // Convert RHI types to core types and forward
        m_coreCallback->handleMessage(convertType(type), convertSource(source), message);
    }
};

Data Flow: Debug Messages End-to-End

In-Process Mode Flow

GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer → Test Output

Out-of-Process Mode Flow

Child Process:
GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer
                                                              ↓
Parent Process:                                    JSON-RPC Serialization
Test Output ← String Processing ← ExecutionResult.debugLayer ←┘

Step-by-Step Example

1. Test Execution Starts
// In test-server process
CoreDebugCallback debugCallback;
CoreToRHIDebugBridge bridge;
bridge.setCoreCallback(&debugCallback);

// Set up graphics device with debug layers
deviceDesc.debugCallback = &bridge;
2. Graphics API Call Triggers Validation Error
// Inside Vulkan driver (external code)
// Validation layer detects error and calls our callback
bridge.handleMessage(RHI_ERROR, RHI_LAYER, "Invalid buffer binding");
3. Message Capture
// In CoreDebugCallback::handleMessage
m_buf << "Invalid buffer binding\n";  // Stored in string buffer
4. Test Completion & Serialization
// Back in test-server
TestServerProtocol::ExecutionResult result;
result.debugLayer = debugCallback.getString();  // "Invalid buffer binding\n"
result.stdOut = "1";  // Regular test output

// Send via JSON-RPC
connection.sendResult(&result);
5. Parent Process Receives & Separates Output
// In slang-test process
String output = buildTestOutput(result);
// Results in clean separation:
// standard output = {
// 1
// }
// debug layer = {
// Invalid buffer binding
// }

Why This Solution Works

1. Process Isolation: Each process has its own callback objects, no shared pointers
2. String Serialization: Debug messages converted to strings that can cross process boundaries
3. Protocol Extension: Uses existing JSON-RPC infrastructure, just adds new field
4. Clean Separation: Debug messages never mix with stdout/stderr
5. Backward Compatibility: Existing tests unaffected, debug layer field optional

Key Benefits

- Eliminates False Positives: Debug messages can't interfere with //CHECK: patterns
- Better Debugging: Debug messages clearly separated and labeled
- Robust Architecture: Works across all execution modes
- Minimal Changes: Leverages existing communication infrastructure

This elegant solution transforms a fundamental cross-process communication challenge into a simple string
serialization problem, using the existing test server architecture to cleanly separate validation layer messages
from test results.

* Cover the missing slang-test execution path

* format code (#82)

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

---------

Co-authored-by: slangbot <ellieh+slangbot@nvidia.com>
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

1 files changed, 11 insertions, 0 deletions

diff --git a/tools/test-server/test-server-main.cpp b/tools/test-server/test-server-main.cpp
index a2f7f8153..e98be6e5a 100644
--- a/tools/test-server/test-server-main.cpp
+++ b/tools/test-server/test-server-main.cpp
@@ -10,7 +10,10 @@
 #include "../../source/core/slang-string.h"
 #include "../../source/core/slang-test-tool-util.h"
 #include "../../source/core/slang-writer.h"
+#include "../render-test/slang-support.h"
+#include "gfx-unit-test/gfx-test-util.h"
 #include "slang-com-helper.h"
+#include "slang-rhi.h"
 #include "test-server-diagnostics.h"
 #include "unit-test/slang-unit-test.h"
 
@@ -422,6 +425,9 @@ SlangResult TestServer::_executeUnitTest(const JSONRPCCall& call)
     }
 
     TestReporter testReporter;
+    renderer_test::CoreDebugCallback coreDebugCallback;
+    renderer_test::CoreToRHIDebugBridge rhiDebugCallback;
+    rhiDebugCallback.setCoreCallback(&coreDebugCallback);
 
     testModule->setTestReporter(&testReporter);
 
@@ -438,6 +444,7 @@ SlangResult TestServer::_executeUnitTest(const JSONRPCCall& call)
     unitTestContext.enabledApis = RenderApiFlags(args.enabledApis);
     unitTestContext.executableDirectory = m_exeDirectory.getBuffer();
     unitTestContext.enableDebugLayers = args.enableDebugLayers;
+    unitTestContext.debugCallback = &rhiDebugCallback;
 
     auto testCount = testModule->getTestCount();
     SLANG_ASSERT(testIndex >= 0 && testIndex < testCount);
@@ -455,6 +462,7 @@ SlangResult TestServer::_executeUnitTest(const JSONRPCCall& call)
 
     TestServerProtocol::ExecutionResult result;
     result.result = SLANG_OK;
+    result.debugLayer = coreDebugCallback.getString();
 
     if (testReporter.m_failCount > 0)
     {
@@ -508,6 +516,7 @@ SlangResult TestServer::_executeTool(const JSONRPCCall& call)
     StdWriters stdWriters;
     StringBuilder stdOut;
     StringBuilder stdError;
+    renderer_test::CoreDebugCallback debugCallback;
 
     // Make writer/s act as if they are the console.
     RefPtr<StringWriter> stdOutWriter(new StringWriter(&stdOut, WriterFlag::IsConsole));
@@ -515,6 +524,7 @@ SlangResult TestServer::_executeTool(const JSONRPCCall& call)
 
     stdWriters.setWriter(SLANG_WRITER_CHANNEL_STD_ERROR, stdErrorWriter);
     stdWriters.setWriter(SLANG_WRITER_CHANNEL_STD_OUTPUT, stdOutWriter);
+    stdWriters.setDebugCallback(&debugCallback);
 
     // HACK, to make behavior the same as previously
     if (args.toolName == "slangc")
@@ -529,6 +539,7 @@ SlangResult TestServer::_executeTool(const JSONRPCCall& call)
     result.result = funcRes;
     result.stdError = stdError;
     result.stdOut = stdOut;
+    result.debugLayer = debugCallback.getString();
 
     result.returnCode = int32_t(TestToolUtil::getReturnCode(result.result));
     return m_connection->sendResult(&result, id);