Handle debug-layer messages in a separate channel (#7988)

* Handle debug-layer messages in a separate channel

The Problem (Issue #7343)

The issue was that Vulkan Validation Layer error messages were being mixed into regular test output, causing
potential false positives or negatives. When using -enable-debug-layers true, validation messages would appear in
the same output stream as test results, potentially matching //CHECK: patterns incorrectly.

Example Problem:
- Test expects: //CHECK: 1
- Validation layer prints: VALIDATION ERROR: 1 invalid buffer binding
- Test incorrectly matches the "1" in the error message instead of the actual output

Slang Test Communication Architecture

Execution Modes

Slang has 3 different execution modes controlled by SpawnType:

enum class SpawnType {
    UseSharedLibrary,              // In-process execution
    UseTestServer,                 // Out-of-process via persistent server
    UseFullyIsolatedTestServer,    // Out-of-process via isolated server
    UseExe                         // Direct executable spawn
}

1. In-Process Mode (UseSharedLibrary)

┌─────────────────────────────────────────┐
│              slang-test                 │
│  ┌─────────────┐    ┌─────────────────┐ │
│  │render-test  │    │gfx-unit-test    │ │
│  │unit-test    │    │slangc library   │ │
│  └─────────────┘    └─────────────────┘ │
│           │                   │         │
│           └───► StdWriters ◄──┘         │
│                 (shared)                │
└─────────────────────────────────────────┘
- Communication: Direct function calls, shared memory
- Debug callbacks: Single callback instance in StdWriters
- Used when: Default mode for most tests

2. Out-of-Process Mode (UseTestServer)

┌──────────────────┐    JSON-RPC     ┌──────────────────┐
│   slang-test     │◄──over pipes────┤ test-server.exe  │
│                  │                 │                  │
│  ┌─────────────┐ │                 │ ┌─────────────┐  │
│  │StdWriters   │ │                 │ │render-test  │  │
│  │+debug       │ │                 │ │gfx-unit-test│  │
│  │callback     │ │                 │ │+debug       │  │
│  └─────────────┘ │                 │ │callback     │  │
└──────────────────┘                 │ └─────────────┘  │
                                     └──────────────────┘
- Communication: JSON-RPC over stdin/stdout pipes
- Debug callbacks: Separate instances in each process
- Used when: CI/CD, multi-threaded testing, crash isolation

3. Direct Executable Mode (UseExe)

┌──────────────────┐    pipes       ┌──────────────────┐
│   slang-test     │◄───────────────┤  slangc.exe      │
│                  │                │  other tools     │
└──────────────────┘                └──────────────────┘
- Communication: Standard process pipes (stdout/stderr)
- Debug callbacks: None (external executables)
- Used when: Testing external tools

Communication Mechanisms Deep Dive

JSON-RPC Protocol Over Pipes

The test-server.exe communicates with slang-test using JSON-RPC over stdin/stdout pipes:

// Parent process (slang-test) creates child with pipes
Process* testServerProcess = /* spawn test-server.exe */;

// JSONRPCConnection wraps the pipe communication
JSONRPCConnection connection;
connection.initWithStdStreams(); // Uses stdin/stdout pipes

// Send RPC call
TestServerProtocol::ExecutionResult result;
connection.sendCall("executeTool", &args, &result);

Key Point: The pipes carry structured JSON messages, not raw stdout/stderr. This is what enables clean separation
of different data channels.

Protocol Structure

Your changes extend the ExecutionResult protocol:

struct ExecutionResult {
    String stdOut;      // Regular program output
    String stdError;    // Error messages
    String debugLayer;  // NEW: Debug/validation messages
    int32_t result;
    int32_t returnCode;
};

Your Debug Layer Solution

The Challenge

Memory pointers cannot cross process boundaries. A debugCallback pointer in the parent process is meaningless in
the child process.

The Solution: String-Based Serialization

You solved this by using string capture and serialization:

1. Debug Callback Interface (slang-std-writers.h)

class IDebugCallback {
    virtual void handleMessage(
        DebugMessageType type,
        DebugMessageSource source,
        const char* message) = 0;
};

2. String-Capturing Implementation (slang-support.h)

class CoreDebugCallback : public Slang::IDebugCallback {
    StringBuilder m_buf;  // Captures messages as strings

    void handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) {
        if (type == DebugMessageType::Error) {
            m_buf << message << '\n';  // Serialize to string
        }
    }

    String getString() { return m_buf.toString(); }  // Extract accumulated messages
};

3. Bridge Between RHI and Core (slang-support.h)

class CoreToRHIDebugBridge : public rhi::IDebugCallback {
    Slang::IDebugCallback* m_coreCallback;

    void handleMessage(rhi::DebugMessageType type, rhi::DebugMessageSource source, const char* message) {
        // Convert RHI types to core types and forward
        m_coreCallback->handleMessage(convertType(type), convertSource(source), message);
    }
};

Data Flow: Debug Messages End-to-End

In-Process Mode Flow

GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer → Test Output

Out-of-Process Mode Flow

Child Process:
GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer
                                                              ↓
Parent Process:                                    JSON-RPC Serialization
Test Output ← String Processing ← ExecutionResult.debugLayer ←┘

Step-by-Step Example

1. Test Execution Starts
// In test-server process
CoreDebugCallback debugCallback;
CoreToRHIDebugBridge bridge;
bridge.setCoreCallback(&debugCallback);

// Set up graphics device with debug layers
deviceDesc.debugCallback = &bridge;
2. Graphics API Call Triggers Validation Error
// Inside Vulkan driver (external code)
// Validation layer detects error and calls our callback
bridge.handleMessage(RHI_ERROR, RHI_LAYER, "Invalid buffer binding");
3. Message Capture
// In CoreDebugCallback::handleMessage
m_buf << "Invalid buffer binding\n";  // Stored in string buffer
4. Test Completion & Serialization
// Back in test-server
TestServerProtocol::ExecutionResult result;
result.debugLayer = debugCallback.getString();  // "Invalid buffer binding\n"
result.stdOut = "1";  // Regular test output

// Send via JSON-RPC
connection.sendResult(&result);
5. Parent Process Receives & Separates Output
// In slang-test process
String output = buildTestOutput(result);
// Results in clean separation:
// standard output = {
// 1
// }
// debug layer = {
// Invalid buffer binding
// }

Why This Solution Works

1. Process Isolation: Each process has its own callback objects, no shared pointers
2. String Serialization: Debug messages converted to strings that can cross process boundaries
3. Protocol Extension: Uses existing JSON-RPC infrastructure, just adds new field
4. Clean Separation: Debug messages never mix with stdout/stderr
5. Backward Compatibility: Existing tests unaffected, debug layer field optional

Key Benefits

- Eliminates False Positives: Debug messages can't interfere with //CHECK: patterns
- Better Debugging: Debug messages clearly separated and labeled
- Robust Architecture: Works across all execution modes
- Minimal Changes: Leverages existing communication infrastructure

This elegant solution transforms a fundamental cross-process communication challenge into a simple string
serialization problem, using the existing test server architecture to cleanly separate validation layer messages
from test results.

* Cover the missing slang-test execution path

* format code (#82)

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

---------

Co-authored-by: slangbot <ellieh+slangbot@nvidia.com>
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

1 files changed, 25 insertions, 0 deletions

diff --git a/source/core/slang-std-writers.h b/source/core/slang-std-writers.h
index bb67d9d9c..bb38d12e0 100644
--- a/source/core/slang-std-writers.h
+++ b/source/core/slang-std-writers.h
@@ -7,6 +7,27 @@
 namespace Slang
 {
 
+enum class DebugMessageType
+{
+    Info,
+    Warning,
+    Error
+};
+
+enum class DebugMessageSource
+{
+    Layer,
+    Driver,
+    Slang
+};
+
+class IDebugCallback
+{
+public:
+    virtual SLANG_NO_THROW void SLANG_MCALL
+    handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) = 0;
+};
+
 /* Holds standard writers for the channels */
 class StdWriters : public RefObject
 {
@@ -14,6 +35,9 @@ public:
     ISlangWriter* getWriter(SlangWriterChannel chan) const { return m_writers[chan]; }
     void setWriter(SlangWriterChannel chan, ISlangWriter* writer) { m_writers[chan] = writer; }
 
+    IDebugCallback* getDebugCallback() const { return m_debugCallback; }
+    void setDebugCallback(IDebugCallback* callback) { m_debugCallback = callback; }
+
     /// Flush all the set writers
     void flushWriters();
 
@@ -42,6 +66,7 @@ public:
 
 protected:
     ComPtr<ISlangWriter> m_writers[SLANG_WRITER_CHANNEL_COUNT_OF];
+    IDebugCallback* m_debugCallback = nullptr;
 
     static StdWriters* s_singleton;
 };