Handle debug-layer messages in a separate channel (#7988)

* Handle debug-layer messages in a separate channel

The Problem (Issue #7343)

The issue was that Vulkan Validation Layer error messages were being mixed into regular test output, causing
potential false positives or negatives. When using -enable-debug-layers true, validation messages would appear in
the same output stream as test results, potentially matching //CHECK: patterns incorrectly.

Example Problem:
- Test expects: //CHECK: 1
- Validation layer prints: VALIDATION ERROR: 1 invalid buffer binding
- Test incorrectly matches the "1" in the error message instead of the actual output

Slang Test Communication Architecture

Execution Modes

Slang has 3 different execution modes controlled by SpawnType:

enum class SpawnType {
    UseSharedLibrary,              // In-process execution
    UseTestServer,                 // Out-of-process via persistent server
    UseFullyIsolatedTestServer,    // Out-of-process via isolated server
    UseExe                         // Direct executable spawn
}

1. In-Process Mode (UseSharedLibrary)

┌─────────────────────────────────────────┐
│              slang-test                 │
│  ┌─────────────┐    ┌─────────────────┐ │
│  │render-test  │    │gfx-unit-test    │ │
│  │unit-test    │    │slangc library   │ │
│  └─────────────┘    └─────────────────┘ │
│           │                   │         │
│           └───► StdWriters ◄──┘         │
│                 (shared)                │
└─────────────────────────────────────────┘
- Communication: Direct function calls, shared memory
- Debug callbacks: Single callback instance in StdWriters
- Used when: Default mode for most tests

2. Out-of-Process Mode (UseTestServer)

┌──────────────────┐    JSON-RPC     ┌──────────────────┐
│   slang-test     │◄──over pipes────┤ test-server.exe  │
│                  │                 │                  │
│  ┌─────────────┐ │                 │ ┌─────────────┐  │
│  │StdWriters   │ │                 │ │render-test  │  │
│  │+debug       │ │                 │ │gfx-unit-test│  │
│  │callback     │ │                 │ │+debug       │  │
│  └─────────────┘ │                 │ │callback     │  │
└──────────────────┘                 │ └─────────────┘  │
                                     └──────────────────┘
- Communication: JSON-RPC over stdin/stdout pipes
- Debug callbacks: Separate instances in each process
- Used when: CI/CD, multi-threaded testing, crash isolation

3. Direct Executable Mode (UseExe)

┌──────────────────┐    pipes       ┌──────────────────┐
│   slang-test     │◄───────────────┤  slangc.exe      │
│                  │                │  other tools     │
└──────────────────┘                └──────────────────┘
- Communication: Standard process pipes (stdout/stderr)
- Debug callbacks: None (external executables)
- Used when: Testing external tools

Communication Mechanisms Deep Dive

JSON-RPC Protocol Over Pipes

The test-server.exe communicates with slang-test using JSON-RPC over stdin/stdout pipes:

// Parent process (slang-test) creates child with pipes
Process* testServerProcess = /* spawn test-server.exe */;

// JSONRPCConnection wraps the pipe communication
JSONRPCConnection connection;
connection.initWithStdStreams(); // Uses stdin/stdout pipes

// Send RPC call
TestServerProtocol::ExecutionResult result;
connection.sendCall("executeTool", &args, &result);

Key Point: The pipes carry structured JSON messages, not raw stdout/stderr. This is what enables clean separation
of different data channels.

Protocol Structure

Your changes extend the ExecutionResult protocol:

struct ExecutionResult {
    String stdOut;      // Regular program output
    String stdError;    // Error messages
    String debugLayer;  // NEW: Debug/validation messages
    int32_t result;
    int32_t returnCode;
};

Your Debug Layer Solution

The Challenge

Memory pointers cannot cross process boundaries. A debugCallback pointer in the parent process is meaningless in
the child process.

The Solution: String-Based Serialization

You solved this by using string capture and serialization:

1. Debug Callback Interface (slang-std-writers.h)

class IDebugCallback {
    virtual void handleMessage(
        DebugMessageType type,
        DebugMessageSource source,
        const char* message) = 0;
};

2. String-Capturing Implementation (slang-support.h)

class CoreDebugCallback : public Slang::IDebugCallback {
    StringBuilder m_buf;  // Captures messages as strings

    void handleMessage(DebugMessageType type, DebugMessageSource source, const char* message) {
        if (type == DebugMessageType::Error) {
            m_buf << message << '\n';  // Serialize to string
        }
    }

    String getString() { return m_buf.toString(); }  // Extract accumulated messages
};

3. Bridge Between RHI and Core (slang-support.h)

class CoreToRHIDebugBridge : public rhi::IDebugCallback {
    Slang::IDebugCallback* m_coreCallback;

    void handleMessage(rhi::DebugMessageType type, rhi::DebugMessageSource source, const char* message) {
        // Convert RHI types to core types and forward
        m_coreCallback->handleMessage(convertType(type), convertSource(source), message);
    }
};

Data Flow: Debug Messages End-to-End

In-Process Mode Flow

GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer → Test Output

Out-of-Process Mode Flow

Child Process:
GPU Driver → RHI Debug Callback → Core Debug Callback → String Buffer
                                                              ↓
Parent Process:                                    JSON-RPC Serialization
Test Output ← String Processing ← ExecutionResult.debugLayer ←┘

Step-by-Step Example

1. Test Execution Starts
// In test-server process
CoreDebugCallback debugCallback;
CoreToRHIDebugBridge bridge;
bridge.setCoreCallback(&debugCallback);

// Set up graphics device with debug layers
deviceDesc.debugCallback = &bridge;
2. Graphics API Call Triggers Validation Error
// Inside Vulkan driver (external code)
// Validation layer detects error and calls our callback
bridge.handleMessage(RHI_ERROR, RHI_LAYER, "Invalid buffer binding");
3. Message Capture
// In CoreDebugCallback::handleMessage
m_buf << "Invalid buffer binding\n";  // Stored in string buffer
4. Test Completion & Serialization
// Back in test-server
TestServerProtocol::ExecutionResult result;
result.debugLayer = debugCallback.getString();  // "Invalid buffer binding\n"
result.stdOut = "1";  // Regular test output

// Send via JSON-RPC
connection.sendResult(&result);
5. Parent Process Receives & Separates Output
// In slang-test process
String output = buildTestOutput(result);
// Results in clean separation:
// standard output = {
// 1
// }
// debug layer = {
// Invalid buffer binding
// }

Why This Solution Works

1. Process Isolation: Each process has its own callback objects, no shared pointers
2. String Serialization: Debug messages converted to strings that can cross process boundaries
3. Protocol Extension: Uses existing JSON-RPC infrastructure, just adds new field
4. Clean Separation: Debug messages never mix with stdout/stderr
5. Backward Compatibility: Existing tests unaffected, debug layer field optional

Key Benefits

- Eliminates False Positives: Debug messages can't interfere with //CHECK: patterns
- Better Debugging: Debug messages clearly separated and labeled
- Robust Architecture: Works across all execution modes
- Minimal Changes: Leverages existing communication infrastructure

This elegant solution transforms a fundamental cross-process communication challenge into a simple string
serialization problem, using the existing test server architecture to cleanly separate validation layer messages
from test results.

* Cover the missing slang-test execution path

* format code (#82)

Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

---------

Co-authored-by: slangbot <ellieh+slangbot@nvidia.com>
Co-authored-by: slangbot <186143334+slangbot@users.noreply.github.com>

1 files changed, 40 insertions, 0 deletions

diff --git a/tools/slang-test/slang-test-main.cpp b/tools/slang-test/slang-test-main.cpp
index 54ec1a842..97c63c392 100644
--- a/tools/slang-test/slang-test-main.cpp
+++ b/tools/slang-test/slang-test-main.cpp
@@ -24,6 +24,7 @@
 #include "../../source/compiler-core/slang-downstream-compiler.h"
 #include "../../source/compiler-core/slang-language-server-protocol.h"
 #include "../../source/compiler-core/slang-nvrtc-compiler.h"
+#include "../render-test/slang-support.h"
 #include "directory-util.h"
 #include "options.h"
 #include "parse-diagnostic-util.h"
@@ -844,6 +845,9 @@ Result spawnAndWaitSharedLibrary(
     {
         StringBuilder stdErrorString;
         StringBuilder stdOutString;
+        renderer_test::CoreDebugCallback coreDebugCallback;
+        renderer_test::CoreToRHIDebugBridge rhiDebugBridge;
+        rhiDebugBridge.setCoreCallback(&coreDebugCallback);
 
         // Say static so not released
         StringWriter stdError(&stdErrorString, WriterFlag::IsConsole | WriterFlag::IsStatic);
@@ -854,6 +858,7 @@ Result spawnAndWaitSharedLibrary(
         StdWriters stdWriters;
         stdWriters.setWriter(SLANG_WRITER_CHANNEL_STD_ERROR, &stdError);
         stdWriters.setWriter(SLANG_WRITER_CHANNEL_STD_OUTPUT, &stdOut);
+        stdWriters.setDebugCallback(&coreDebugCallback);
 
         if (exeName == "slangc" || exeName == "slangi")
         {
@@ -876,6 +881,7 @@ Result spawnAndWaitSharedLibrary(
 
         outRes.standardError = stdErrorString;
         outRes.standardOutput = stdOutString;
+        outRes.debugLayer = coreDebugCallback.getString();
 
         outRes.resultCode = (int)TestToolUtil::getReturnCode(res);
 
@@ -1013,6 +1019,7 @@ static Result _executeRPC(
     outRes.resultCode = exeRes.returnCode;
     outRes.standardError = exeRes.stdError;
     outRes.standardOutput = exeRes.stdOut;
+    outRes.debugLayer = exeRes.debugLayer;
 
     return SLANG_OK;
 }
@@ -1530,6 +1537,7 @@ String getOutput(const ExecuteResult& exeRes)
 
     String standardOuptut = exeRes.standardOutput;
     String standardError = exeRes.standardError;
+    String debugLayer = exeRes.debugLayer;
 
     // We construct a single output string that captures the results
     StringBuilder actualOutputBuilder;
@@ -1540,6 +1548,12 @@ String getOutput(const ExecuteResult& exeRes)
     actualOutputBuilder.append("}\nstandard output = {\n");
     actualOutputBuilder.append(standardOuptut);
     actualOutputBuilder.append("}\n");
+    if (debugLayer.getLength() > 0)
+    {
+        actualOutputBuilder.append("debug layer = {\n");
+        actualOutputBuilder.append(debugLayer);
+        actualOutputBuilder.append("}\n");
+    }
 
     return actualOutputBuilder.produceString();
 }
@@ -3262,6 +3276,7 @@ static TestResult _runHLSLComparisonTest(
 
     String standardOutput = exeRes.standardOutput;
     String standardError = exeRes.standardError;
+    String debugLayer = exeRes.debugLayer;
 
     // We construct a single output string that captures the results
     StringBuilder actualOutputBuilder;
@@ -3272,6 +3287,12 @@ static TestResult _runHLSLComparisonTest(
     actualOutputBuilder.append("}\nstandard output = {\n");
     actualOutputBuilder.append(standardOutput);
     actualOutputBuilder.append("}\n");
+    if (debugLayer.getLength() > 0)
+    {
+        actualOutputBuilder.append("debug layer = {\n");
+        actualOutputBuilder.append(debugLayer);
+        actualOutputBuilder.append("}\n");
+    }
 
     String actualOutput = actualOutputBuilder.produceString();
 
@@ -3339,6 +3360,7 @@ TestResult doGLSLComparisonTestRun(
 
     String standardOuptut = exeRes.standardOutput;
     String standardError = exeRes.standardError;
+    String debugLayer = exeRes.debugLayer;
 
     // We construct a single output string that captures the results
     StringBuilder outputBuilder;
@@ -3349,6 +3371,12 @@ TestResult doGLSLComparisonTestRun(
     outputBuilder.append("}\nstandard output = {\n");
     outputBuilder.append(standardOuptut);
     outputBuilder.append("}\n");
+    if (debugLayer.getLength() > 0)
+    {
+        outputBuilder.append("debug layer = {\n");
+        outputBuilder.append(debugLayer);
+        outputBuilder.append("}\n");
+    }
 
     String outputPath = outputStem + outputKind;
     String output = outputBuilder.produceString();
@@ -3748,6 +3776,7 @@ TestResult doRenderComparisonTestRun(
 
     String standardOutput = exeRes.standardOutput;
     String standardError = exeRes.standardError;
+    String debugLayer = exeRes.debugLayer;
 
     // We construct a single output string that captures the results
     StringBuilder outputBuilder;
@@ -3758,6 +3787,12 @@ TestResult doRenderComparisonTestRun(
     outputBuilder.append("}\nstandard output = {\n");
     outputBuilder.append(standardOutput);
     outputBuilder.append("}\n");
+    if (debugLayer.getLength() > 0)
+    {
+        outputBuilder.append("debug layer = {\n");
+        outputBuilder.append(debugLayer);
+        outputBuilder.append("}\n");
+    }
 
     String outputPath = outputStem + outputKind;
     String output = outputBuilder.produceString();
@@ -4733,12 +4768,17 @@ static SlangResult runUnitTestModule(
     if (!testModule)
         return SLANG_FAIL;
 
+    renderer_test::CoreDebugCallback coreDebugCallback;
+    renderer_test::CoreToRHIDebugBridge rhiDebugBridge;
+    rhiDebugBridge.setCoreCallback(&coreDebugCallback);
+
     UnitTestContext unitTestContext;
     unitTestContext.slangGlobalSession = context->getSession();
     unitTestContext.workDirectory = "";
     unitTestContext.enabledApis = context->options.enabledApis;
     unitTestContext.enableDebugLayers = context->options.enableDebugLayers;
     unitTestContext.executableDirectory = context->exeDirectoryPath.getBuffer();
+    unitTestContext.debugCallback = &rhiDebugBridge;
 
     auto testCount = testModule->getTestCount();