summaryrefslogtreecommitdiffstats
path: root/source/core/slang-byte-encode-util.h
diff options
context:
space:
mode:
authorCopilot <198982749+Copilot@users.noreply.github.com>2025-09-29 13:27:21 +0800
committerGitHub <noreply@github.com>2025-09-29 13:27:21 +0800
commitd3093deb49d1088ede6c3dcd418575bfa4bcd732 (patch)
tree4171f8aa31d8552dd717c32d153426613ca67bd8 /source/core/slang-byte-encode-util.h
parent99de1a6203f676955f80de8c93c0f56c91d4ca96 (diff)
Fix segfault when shader entry points return resource types (#8434)
The Slang compiler was segfaulting when trying to compile shaders that return resource types (like `Texture2D`, `RWTexture2D`, `SamplerState`, etc.) from entry point functions. This occurred because there was missing validation that should reject such invalid return types before they reach IR generation. For example, this code would cause a segfault: ```slang StructuredBuffer<Texture2D<int>> skyLight; [shader("compute")] Texture2D<int> computeMain(uint3 threadID : SV_DispatchThreadID) { return skyLight[threadID.x]; } ``` ## Root Cause The issue was in the entry point validation logic in `validateEntryPoint()`. While there was a TODO comment indicating that return type validation should be performed, it was never implemented. The compiler would accept the invalid shader code and attempt to process it during IR lowering, where resource types as return values are not properly handled, leading to a segmentation fault. ## Solution 1. **Added robust validation**: Modified `validateEntryPoint()` in `slang-check-shader.cpp` to use the existing `SemanticsVisitor::getTypeTags()` functionality to check for invalid return types by detecting `TypeTag::Opaque` and `TypeTag::Unsized` bits. This leverages the existing type analysis infrastructure that comprehensively handles: - Direct resource types (Texture2D, RWTexture2D, SamplerState, etc.) - Structs containing resource-typed fields (through type tag propagation) - Nested structures and complex type hierarchies - Arrays and other composite types 2. **Added diagnostic message**: Uses existing diagnostic `entryPointCannotReturnResourceType` (error 38010) that provides a clear error message explaining why resource types cannot be returned from shader entry points 3. **Updated existing tests**: Modified existing tests to match the updated validation behavior ## Result Instead of a segfault, users now get a clear, actionable error message: ``` error 38010: entry point 'computeMain' cannot return type 'Texture2D<int>' that contains resource types ``` The fix properly handles all resource types including `Texture2D`, `RWTexture2D`, `SamplerState`, and others, while preserving the ability to compile valid shaders that return simple data types. Fixes #6438. <!-- START COPILOT CODING AGENT TIPS --> --- 💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: expipiplus1 <857308+expipiplus1@users.noreply.github.com> Co-authored-by: Ellie Hermaszewska <ellieh@nvidia.com> Co-authored-by: csyonghe <2652293+csyonghe@users.noreply.github.com>
Diffstat (limited to 'source/core/slang-byte-encode-util.h')
0 files changed, 0 insertions, 0 deletions