Handle debug-layer messages in a separate channel (#7988)

* Handle debug-layer messages in a separate channel

The Problem (Issue #7343)

The issue was that Vulkan Validation Layer error messages were being mixed into regular test output, causing
potential false positives or negatives. When using -enable-debug-layers true, validation messages would appear in
the same output stream as test results, potentially matching //CHECK: patterns incorrectly.

Example Problem:
- Test expects: //CHECK: 1
- Validation layer prints: VALIDATION ERROR: 1 invalid buffer binding
- Test incorrectly matches the "1" in the error message instead of the actual output

Slang Test Communication Architecture

Execution Modes

Slang has 3 different execution modes controlled by SpawnType:

enum class SpawnType {
    UseSharedLibrary,              // In-process execution
    UseTestServer,                 // Out-of-process via persistent server
    UseFullyIsolatedTestServer,    // Out-of-process via isolated server
    UseExe                         // Direct executable spawn
}

1. In-Process Mode (UseSharedLibrary)

┌─────────────────────────────────────────┐
│              slang-test                 │
│  ┌─────────────┐    ┌─────────────────┐ │
│  │render-test  │    │gfx-unit-test    │ │
│  │unit-test    │    │slangc library   │ │
│  └─────────────┘    └─────────────────┘ │
│           │                   │         │
│           └───► StdWriters ◄──┘         │
│                 (shared)                │
└─────────────────────────────────────────┘
- Communication: Direct function calls, shared memory
- Debug callbacks: Single callback instance in StdWriters
- Used when: Default mode for most tests

2. Out-of-Process Mode (UseTestServer)

┌──────────────────┐    JSON-RPC     ┌──────────────────┐
│   slang-test     │◄──over pipes────┤ test-server.exe  │
│                  │                 │                  │
│  ┌─────────────┐ │                 │ ┌─────────────┐  │
│  │StdWriters   │ │                 │ │render-test  │  │
│  │+debug       │ │                 │ │gfx-unit-test│  │
│  │callback     │ │                 │ │+debug       │  │
│  └─────────────┘ │                 │ │callback     │  │
└──────────────────┘                 │ └─────────────┘  │
                                     └──────────────────┘
- Communication: JSON-RPC over stdin/stdout pipes
- Debug callbacks: Separate instances in each process
- Used when: CI/CD, multi-threaded testing, crash isolation

3. Direct Executable Mode (UseExe)

┌──────────────────┐    pipes       ┌──────────────────┐
│   slang-test     │◄───────────────┤  slangc.exe      │
│                  │                │  other tools     │
└──────────────────┘                └──────────────────┘
- Communication: Standard process pipes (stdout/stderr)
- Debug callbacks: None (external executables)
- Used when: Testing external tools

Communication Mechanisms Deep Dive

JSON-RPC Protocol Over Pipes

The test-server.exe communicates with slang-test using JSON-RPC over stdin/stdout pipes:

// Parent process (slang-test) creates child with pipes
Process* testServerProcess = /* spawn test-server.exe */;

// JSONRPCConnection wraps the pipe communication
JSONRPCConnection connection;
connection.initWithStdStreams(); // Uses stdin/stdout pipes

// Send RPC call
TestServerProtocol::ExecutionResult result;
connection.sendCall("executeTool", &args, &result);

Key Point: The pipes carry structured JSON messages, not raw stdout/stderr. This is what enables clean separation
of different data channels.

Protocol Structure

Your changes extend the ExecutionResult protocol:

struct ExecutionResult {
    String stdOut;      // Regular program output
    String stdError;    // Error messages
    String debugLayer;  // NEW: Debug/validation messages
    int32_t result;
    int32_t returnCode;
};

Your Debug Layer Solution

The Challenge

Memory pointers cannot cross process boundaries. A debugCallback pointer in the parent process is meaningless in
the child process.

The Solution: String-Based Serialization

You solved this by using string capture and serialization:

1. Debug Callback Interface (slang-std-writers.h)

class IDebugCallback {
    virtual void handleMessage(
        DebugMessageType type,
        DebugMessageSource source,
        const char* message) = 0;
};

2. String-Capturing Implementation (slang-support.h)

class CoreDebugCallback : public Slang::IDebugCallback {
    StringBuilder m_buf;  // Captures messages as strings

    void handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) {
        if (type == DebugMessageType::Error) {
            m_buf << message << '\n';  // Serialize to string
        }
    }

    String getString() { return m_buf.toString(); }  // Extract accumulated messages
};

3. Bridge Between RHI and Core (slang-support.h)

class CoreToRHIDebugBridge : public rhi::IDebugCallback {
    Slang::IDebugCallback* m_coreCallback;

    void handleMessage(rhi::DebugMessageType type, rhi::DebugMessageSource source, const char* message) {
        // Convert RHI types to core types and forward
        m_coreCallback->handleMessage(convertType(type), convertSource(source), message);
    }
};

Data Flow: Debug Messages End-to-End

In-Process Mode Flow

GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer → Test Output

Out-of-Process Mode Flow

Child Process:
GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer
                                                              ↓
Parent Process:                                    JSON-RPC Serialization
Test Output ← String Processing ← ExecutionResult.debugLayer ←┘

Step-by-Step Example

1. Test Execution Starts
// In test-server process
CoreDebugCallback debugCallback;
CoreToRHIDebugBridge bridge;
bridge.setCoreCallback(&debugCallback);

// Set up graphics device with debug layers
deviceDesc.debugCallback = &bridge;
2. Graphics API Call Triggers Validation Error
// Inside Vulkan driver (external code)
// Validation layer detects error and calls our callback
bridge.handleMessage(RHI_ERROR, RHI_LAYER, "Invalid buffer binding");
3. Message Capture
// In CoreDebugCallback::handleMessage
m_buf << "Invalid buffer binding\n";  // Stored in string buffer
4. Test Completion & Serialization
// Back in test-server
TestServerProtocol::ExecutionResult result;
result.debugLayer = debugCallback.getString();  // "Invalid buffer binding\n"
result.stdOut = "1";  // Regular test output

// Send via JSON-RPC
connection.sendResult(&result);
5. Parent Process Receives & Separates Output
// In slang-test process
String output = buildTestOutput(result);
// Results in clean separation:
// standard output = {
// 1
// }
// debug layer = {
// Invalid buffer binding
// }

Why This Solution Works

1. Process Isolation: Each process has its own callback objects, no shared pointers
2. String Serialization: Debug messages converted to strings that can cross process boundaries
3. Protocol Extension: Uses existing JSON-RPC infrastructure, just adds new field
4. Clean Separation: Debug messages never mix with stdout/stderr
5. Backward Compatibility: Existing tests unaffected, debug layer field optional

Key Benefits

- Eliminates False Positives: Debug messages can't interfere with //CHECK: patterns
- Better Debugging: Debug messages clearly separated and labeled
- Robust Architecture: Works across all execution modes
- Minimal Changes: Leverages existing communication infrastructure

This elegant solution transforms a fundamental cross-process communication challenge into a simple string
serialization problem, using the existing test server architecture to cleanly separate validation layer messages
from test results.

* Cover the missing slang-test execution path

* format code (#82)

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

---------

Co-authored-by: slangbot <ellieh+slangbot@nvidia.com>
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

2 files changed, 59 insertions, 19 deletions

diff --git a/tools/render-test/render-test-main.cpp b/tools/render-test/render-test-main.cpp
index 23035280c..af0415bd0 100644
--- a/tools/render-test/render-test-main.cpp
+++ b/tools/render-test/render-test-main.cpp
@@ -4,6 +4,7 @@
 
 #include "../../source/core/slang-test-tool-util.h"
 #include "../source/core/slang-io.h"
+#include "../source/core/slang-std-writers.h"
 #include "../source/core/slang-string-util.h"
 #include "core/slang-token-reader.h"
 #include "options.h"
@@ -1322,23 +1323,6 @@ static void renderDocBeginFrame() {}
 static void renderDocEndFrame() {}
 #endif
 
-class StdWritersDebugCallback : public rhi::IDebugCallback
-{
-public:
-    Slang::StdWriters* writers;
-    virtual SLANG_NO_THROW void SLANG_MCALL handleMessage(
-        rhi::DebugMessageType type,
-        rhi::DebugMessageSource source,
-        const char* message) override
-    {
-        SLANG_UNUSED(source);
-        if (type == rhi::DebugMessageType::Error)
-        {
-            writers->getOut().print("%s\n", message);
-        }
-    }
-};
-
 static SlangResult _innerMain(
     Slang::StdWriters* stdWriters,
     SlangSession* session,
@@ -1456,8 +1440,8 @@ static SlangResult _innerMain(
         }
     }
 
-    StdWritersDebugCallback debugCallback;
-    debugCallback.writers = stdWriters;
+    renderer_test::CoreToRHIDebugBridge debugCallback;
+    debugCallback.setCoreCallback(stdWriters->getDebugCallback());
 
     // Use the profile name set on options if set
     input.profile = options.profileName.getLength() ? options.profileName : input.profile;
diff --git a/tools/render-test/slang-support.h b/tools/render-test/slang-support.h
index 916166c3a..51a2c8de1 100644
--- a/tools/render-test/slang-support.h
+++ b/tools/render-test/slang-support.h
@@ -1,6 +1,7 @@
 // slang-support.h
 #pragma once
 
+#include "core/slang-std-writers.h"
 #include "options.h"
 #include "shader-input-layout.h"
 #include "slang.h"
@@ -10,6 +11,61 @@
 namespace renderer_test
 {
 
+/// Bridge from core debug callback to RHI debug callback
+/// This allows core callbacks to receive messages from RHI systems
+/// TODO: We should replace rhi::IDebugCallback with Slang::IDebugCallback.
+class CoreToRHIDebugBridge : public rhi::IDebugCallback
+{
+public:
+    void setCoreCallback(Slang::IDebugCallback* coreCallback) { m_coreCallback = coreCallback; }
+
+    virtual SLANG_NO_THROW void SLANG_MCALL handleMessage(
+        rhi::DebugMessageType type,
+        rhi::DebugMessageSource source,
+        const char* message) override
+    {
+        if (m_coreCallback)
+        {
+            // Convert RHI types to core types
+            Slang::DebugMessageType coreType = static_cast<Slang::DebugMessageType>(type);
+            Slang::DebugMessageSource coreSource = static_cast<Slang::DebugMessageSource>(source);
+            m_coreCallback->handleMessage(coreType, coreSource, message);
+        }
+    }
+
+private:
+    Slang::IDebugCallback* m_coreCallback = nullptr;
+};
+
+/// Core debug callback that captures debug messages in a string buffer
+class CoreDebugCallback : public Slang::IDebugCallback
+{
+public:
+    virtual SLANG_NO_THROW void SLANG_MCALL handleMessage(
+        Slang::DebugMessageType type,
+        Slang::DebugMessageSource source,
+        const char* message) override
+    {
+        SLANG_UNUSED(source);
+
+        // Only capture error messages
+        if (type == Slang::DebugMessageType::Error)
+        {
+            m_buf << message;
+            if (message[strlen(message) - 1] != '\n')
+            {
+                m_buf << '\n';
+            }
+        }
+    }
+
+    void clear() { m_buf.clear(); }
+    Slang::String getString() { return m_buf.toString(); }
+
+private:
+    Slang::StringBuilder m_buf;
+};
+
 struct ShaderCompileRequest
 {
     struct SourceInfo