Handle debug-layer messages in a separate channel (#7988)

* Handle debug-layer messages in a separate channel

The Problem (Issue #7343)

The issue was that Vulkan Validation Layer error messages were being mixed into regular test output, causing
potential false positives or negatives. When using -enable-debug-layers true, validation messages would appear in
the same output stream as test results, potentially matching //CHECK: patterns incorrectly.

Example Problem:
- Test expects: //CHECK: 1
- Validation layer prints: VALIDATION ERROR: 1 invalid buffer binding
- Test incorrectly matches the "1" in the error message instead of the actual output

Slang Test Communication Architecture

Execution Modes

Slang has 3 different execution modes controlled by SpawnType:

enum class SpawnType {
    UseSharedLibrary,              // In-process execution
    UseTestServer,                 // Out-of-process via persistent server
    UseFullyIsolatedTestServer,    // Out-of-process via isolated server
    UseExe                         // Direct executable spawn
}

1. In-Process Mode (UseSharedLibrary)

┌─────────────────────────────────────────┐
│              slang-test                 │
│  ┌─────────────┐    ┌─────────────────┐ │
│  │render-test  │    │gfx-unit-test    │ │
│  │unit-test    │    │slangc library   │ │
│  └─────────────┘    └─────────────────┘ │
│           │                   │         │
│           └───► StdWriters ◄──┘         │
│                 (shared)                │
└─────────────────────────────────────────┘
- Communication: Direct function calls, shared memory
- Debug callbacks: Single callback instance in StdWriters
- Used when: Default mode for most tests

2. Out-of-Process Mode (UseTestServer)

┌──────────────────┐    JSON-RPC     ┌──────────────────┐
│   slang-test     │◄──over pipes────┤ test-server.exe  │
│                  │                 │                  │
│  ┌─────────────┐ │                 │ ┌─────────────┐  │
│  │StdWriters   │ │                 │ │render-test  │  │
│  │+debug       │ │                 │ │gfx-unit-test│  │
│  │callback     │ │                 │ │+debug       │  │
│  └─────────────┘ │                 │ │callback     │  │
└──────────────────┘                 │ └─────────────┘  │
                                     └──────────────────┘
- Communication: JSON-RPC over stdin/stdout pipes
- Debug callbacks: Separate instances in each process
- Used when: CI/CD, multi-threaded testing, crash isolation

3. Direct Executable Mode (UseExe)

┌──────────────────┐    pipes       ┌──────────────────┐
│   slang-test     │◄───────────────┤  slangc.exe      │
│                  │                │  other tools     │
└──────────────────┘                └──────────────────┘
- Communication: Standard process pipes (stdout/stderr)
- Debug callbacks: None (external executables)
- Used when: Testing external tools

Communication Mechanisms Deep Dive

JSON-RPC Protocol Over Pipes

The test-server.exe communicates with slang-test using JSON-RPC over stdin/stdout pipes:

// Parent process (slang-test) creates child with pipes
Process* testServerProcess = /* spawn test-server.exe */;

// JSONRPCConnection wraps the pipe communication
JSONRPCConnection connection;
connection.initWithStdStreams(); // Uses stdin/stdout pipes

// Send RPC call
TestServerProtocol::ExecutionResult result;
connection.sendCall("executeTool", &args, &result);

Key Point: The pipes carry structured JSON messages, not raw stdout/stderr. This is what enables clean separation
of different data channels.

Protocol Structure

Your changes extend the ExecutionResult protocol:

struct ExecutionResult {
    String stdOut;      // Regular program output
    String stdError;    // Error messages
    String debugLayer;  // NEW: Debug/validation messages
    int32_t result;
    int32_t returnCode;
};

Your Debug Layer Solution

The Challenge

Memory pointers cannot cross process boundaries. A debugCallback pointer in the parent process is meaningless in
the child process.

The Solution: String-Based Serialization

You solved this by using string capture and serialization:

1. Debug Callback Interface (slang-std-writers.h)

class IDebugCallback {
    virtual void handleMessage(
        DebugMessageType type,
        DebugMessageSource source,
        const char* message) = 0;
};

2. String-Capturing Implementation (slang-support.h)

class CoreDebugCallback : public Slang::IDebugCallback {
    StringBuilder m_buf;  // Captures messages as strings

    void handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) {
        if (type == DebugMessageType::Error) {
            m_buf << message << '\n';  // Serialize to string
        }
    }

    String getString() { return m_buf.toString(); }  // Extract accumulated messages
};

3. Bridge Between RHI and Core (slang-support.h)

class CoreToRHIDebugBridge : public rhi::IDebugCallback {
    Slang::IDebugCallback* m_coreCallback;

    void handleMessage(rhi::DebugMessageType type, rhi::DebugMessageSource source, const char* message) {
        // Convert RHI types to core types and forward
        m_coreCallback->handleMessage(convertType(type), convertSource(source), message);
    }
};

Data Flow: Debug Messages End-to-End

In-Process Mode Flow

GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer → Test Output

Out-of-Process Mode Flow

Child Process:
GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer
                                                              ↓
Parent Process:                                    JSON-RPC Serialization
Test Output ← String Processing ← ExecutionResult.debugLayer ←┘

Step-by-Step Example

1. Test Execution Starts
// In test-server process
CoreDebugCallback debugCallback;
CoreToRHIDebugBridge bridge;
bridge.setCoreCallback(&debugCallback);

// Set up graphics device with debug layers
deviceDesc.debugCallback = &bridge;
2. Graphics API Call Triggers Validation Error
// Inside Vulkan driver (external code)
// Validation layer detects error and calls our callback
bridge.handleMessage(RHI_ERROR, RHI_LAYER, "Invalid buffer binding");
3. Message Capture
// In CoreDebugCallback::handleMessage
m_buf << "Invalid buffer binding\n";  // Stored in string buffer
4. Test Completion & Serialization
// Back in test-server
TestServerProtocol::ExecutionResult result;
result.debugLayer = debugCallback.getString();  // "Invalid buffer binding\n"
result.stdOut = "1";  // Regular test output

// Send via JSON-RPC
connection.sendResult(&result);
5. Parent Process Receives & Separates Output
// In slang-test process
String output = buildTestOutput(result);
// Results in clean separation:
// standard output = {
// 1
// }
// debug layer = {
// Invalid buffer binding
// }

Why This Solution Works

1. Process Isolation: Each process has its own callback objects, no shared pointers
2. String Serialization: Debug messages converted to strings that can cross process boundaries
3. Protocol Extension: Uses existing JSON-RPC infrastructure, just adds new field
4. Clean Separation: Debug messages never mix with stdout/stderr
5. Backward Compatibility: Existing tests unaffected, debug layer field optional

Key Benefits

- Eliminates False Positives: Debug messages can't interfere with //CHECK: patterns
- Better Debugging: Debug messages clearly separated and labeled
- Robust Architecture: Works across all execution modes
- Minimal Changes: Leverages existing communication infrastructure

This elegant solution transforms a fundamental cross-process communication challenge into a simple string
serialization problem, using the existing test server architecture to cleanly separate validation layer messages
from test results.

* Cover the missing slang-test execution path

* format code (#82)

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

---------

Co-authored-by: slangbot <ellieh+slangbot@nvidia.com>
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

1 files changed, 1 insertions, 28 deletions

diff --git a/tools/gfx-unit-test/gfx-test-util.cpp b/tools/gfx-unit-test/gfx-test-util.cpp
index b5a39a4f0..2e3efe09f 100644
--- a/tools/gfx-unit-test/gfx-test-util.cpp
+++ b/tools/gfx-unit-test/gfx-test-util.cpp
@@ -15,33 +15,6 @@ using Slang::ComPtr;
 
 namespace gfx_test
 {
-class DebugPrinter : public rhi::IDebugCallback
-{
-public:
-    virtual SLANG_NO_THROW void SLANG_MCALL handleMessage(
-        rhi::DebugMessageType type,
-        rhi::DebugMessageSource source,
-        const char* message) override
-    {
-        static const char* kTypeStrings[] = {"INFO", "WARN", "ERROR"};
-        static const char* kSourceStrings[] = {"Layer", "Driver", "Slang"};
-        if (type == rhi::DebugMessageType::Error)
-        {
-            fprintf(
-                stderr,
-                "[%s] (%s) %s\n",
-                kTypeStrings[int(type)],
-                kSourceStrings[int(source)],
-                message);
-            fflush(stderr);
-        }
-    }
-    static DebugPrinter* getInstance()
-    {
-        static DebugPrinter instance;
-        return &instance;
-    }
-};
 
 void diagnoseIfNeeded(slang::IBlob* diagnosticsBlob)
 {
@@ -278,7 +251,7 @@ Slang::ComPtr<IDevice> createTestingDevice(
     if (context->enableDebugLayers)
     {
         deviceDesc.enableValidation = context->enableDebugLayers;
-        deviceDesc.debugCallback = DebugPrinter::getInstance();
+        deviceDesc.debugCallback = context->debugCallback;
     }
 
     D3D12DeviceExtendedDesc extDesc = {};