diff options
| author | Copilot <198982749+Copilot@users.noreply.github.com> | 2025-09-29 13:27:21 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-09-29 13:27:21 +0800 |
| commit | d3093deb49d1088ede6c3dcd418575bfa4bcd732 (patch) | |
| tree | 4171f8aa31d8552dd717c32d153426613ca67bd8 /source/slang/slang-vm-bytecode.cpp | |
| parent | 99de1a6203f676955f80de8c93c0f56c91d4ca96 (diff) | |
Fix segfault when shader entry points return resource types (#8434)
The Slang compiler was segfaulting when trying to compile shaders that
return resource types (like `Texture2D`, `RWTexture2D`, `SamplerState`,
etc.) from entry point functions. This occurred because there was
missing validation that should reject such invalid return types before
they reach IR generation.
For example, this code would cause a segfault:
```slang
StructuredBuffer<Texture2D<int>> skyLight;
[shader("compute")]
Texture2D<int> computeMain(uint3 threadID : SV_DispatchThreadID)
{
return skyLight[threadID.x];
}
```
## Root Cause
The issue was in the entry point validation logic in
`validateEntryPoint()`. While there was a TODO comment indicating that
return type validation should be performed, it was never implemented.
The compiler would accept the invalid shader code and attempt to process
it during IR lowering, where resource types as return values are not
properly handled, leading to a segmentation fault.
## Solution
1. **Added robust validation**: Modified `validateEntryPoint()` in
`slang-check-shader.cpp` to use the existing
`SemanticsVisitor::getTypeTags()` functionality to check for invalid
return types by detecting `TypeTag::Opaque` and `TypeTag::Unsized` bits.
This leverages the existing type analysis infrastructure that
comprehensively handles:
- Direct resource types (Texture2D, RWTexture2D, SamplerState, etc.)
- Structs containing resource-typed fields (through type tag
propagation)
- Nested structures and complex type hierarchies
- Arrays and other composite types
2. **Added diagnostic message**: Uses existing diagnostic
`entryPointCannotReturnResourceType` (error 38010) that provides a clear
error message explaining why resource types cannot be returned from
shader entry points
3. **Updated existing tests**: Modified existing tests to match the
updated validation behavior
## Result
Instead of a segfault, users now get a clear, actionable error message:
```
error 38010: entry point 'computeMain' cannot return type 'Texture2D<int>' that contains resource types
```
The fix properly handles all resource types including `Texture2D`,
`RWTexture2D`, `SamplerState`, and others, while preserving the ability
to compile valid shaders that return simple data types.
Fixes #6438.
<!-- START COPILOT CODING AGENT TIPS -->
---
💡 You can make Copilot smarter by setting up custom instructions,
customizing its development environment and configuring Model Context
Protocol (MCP) servers. Learn more [Copilot coding agent
tips](https://gh.io/copilot-coding-agent-tips) in the docs.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: expipiplus1 <857308+expipiplus1@users.noreply.github.com>
Co-authored-by: Ellie Hermaszewska <ellieh@nvidia.com>
Co-authored-by: csyonghe <2652293+csyonghe@users.noreply.github.com>
Diffstat (limited to 'source/slang/slang-vm-bytecode.cpp')
0 files changed, 0 insertions, 0 deletions
