Pointer layout support (#2930)

* WIP looking at reflection with pointers.

* Added GetPointerLayout.

* Initial test via reflection with layout of ptr type.

* WIP handles ptrs to types that have layout that hasn't been completed.

* Move tests to ptr.

* WIP try to take into account lowering correctly between AggTypeDecl and Type, but doesn't quite work.

* WIP a different path to handling recursive lowering problem with Ptr.

* Fix issues with reflection output.

* Small tidy.

* Fix for infinite recursion issue.

* Lower IRPointerTypeLayout

* Working with generics.
Has a hack to work around Layout around Ptr in IR.
The reflection around the generic - the name isn't much use, it should probably have the generic parameters, but that would require getName to do something more sophisticated.

* Fix issue around calling finishOuterGenerics to early.

* Remove feature/ptr test.

* Fix type legalization being an infinite loop with Ptr self referencing.

* Disable the pointer self reference test because produces an infintie loop on emit.

* Fixed comment based on review.

* Fix for issue with emit and pointers causing infinite recursion.

1 files changed, 13 insertions, 0 deletions

diff --git a/source/slang/slang-parameter-binding.cpp b/source/slang/slang-parameter-binding.cpp
index f6637b13c..cd238f623 100644
--- a/source/slang/slang-parameter-binding.cpp
+++ b/source/slang/slang-parameter-binding.cpp
@@ -1997,6 +1997,18 @@ static RefPtr<TypeLayout> processEntryPointVaryingParameter(
     else if (const auto textureType = as<TextureType>(type)) { return nullptr;  }
     else if(const auto samplerStateType = as<SamplerStateType>(type)) { return nullptr;  }
     else if(const auto constantBufferType = as<ConstantBufferType>(type)) { return nullptr;  }
+    else if (auto ptrType = as<PtrType>(type))
+    {
+        SLANG_ASSERT(ptrType->astNodeType == ASTNodeType::PtrType);
+
+        // Work out the layout for the value/target type
+        auto valueTypeLayout = processEntryPointVaryingParameter(context, ptrType->getValueType(), state, varLayout);
+
+        RefPtr<PointerTypeLayout> ptrTypeLayout = new PointerTypeLayout();
+        ptrTypeLayout->valueTypeLayout = valueTypeLayout;
+
+        return ptrTypeLayout;
+    }
     // Catch declaration-reference types late in the sequence, since
     // otherwise they will include all of the above cases...
     else if( auto declRefType = as<DeclRefType>(type) )
@@ -2136,6 +2148,7 @@ static RefPtr<TypeLayout> processEntryPointVaryingParameter(
             SLANG_UNEXPECTED("unhandled type kind");
         }
     }
+    
     // If we ran into an error in checking the user's code, then skip this parameter
     else if( const auto errorType = as<ErrorType>(type) )
     {