Support for [__extern] attribute (#1111)

* Added RiffReadHelper

* Move type to fourCC in Chunk simplifies some code.

* Make MemoryArena able to track external blocks.
Allow ownership of Data to vary.
Changed IR serialization to use moved allocations to avoid copies.

As it turns out all of the array writes could use unowned data, but doing so requires the IRData to stay in scope longer than IRSerialData, which it does at the moment - but perhaps needs better naming or a control for the feature.

* Write out slang-module container.

* WIP on -r option.
Loading modules - with -r.

* Making the serialized-module run (without using imported module).

* Split compiling module from the test.

* Separate module compilation with a function working.

* Remove serialization test as not used.

* Fix warning on gcc.

* Updated test to have types across module boundary.

* Allow entry point declaration.
A test that tries to build with just an entry point declaration and a module.

* Try to make link work with multiple modules.

* Multi module linking first pass working.

* Multi module test working with -module-name option

* Added feature to repro manifest of approximation of command line that was used.

* Use isDefinition - for determining to add decorations to entry point lowering.

* Added support for repo-file-system.h
More precise control of CacheFileSystem.
Allow RelativeFileSystem to strip paths optionally.
Use canonical paths in PathInfo cache.
Fix bug in -D options for command line output of StateSerailizeUtil

* Add missing slang-options.h

* Fix bug in bit slang-state-serialize.cpp with bit removal.

* Added documentation around -repro-file-system
Added spLoadReproAsFileSystem function.

* Fix warning.

* spAddLibraryReference

* * Add support for slang-lib extension
* Container output when using -no-codegen option

* Use the m_containerFormat to determine if the module container is constructed.
Store the result in a blob. This allows for potential access via the API.
Write the blob if a filename is set.
Use m_ prefix for container variables.

* Added spGetContainerCode.
Made spGetCompileRequestCode work.

* * Put obfuscateCode on linkage
* Remove obfuscation from variable names - as can be achieved by either stripping and/or removing NameHintDecorations at lowering
* Remove name hints being added during lowering
* Add stripping of SourceLoc location in strip phase

* Hashing of linkage import/export names.

* Do final strip in emitEntryPoint, removes any remaining SourceLoc.

* Support for [__extern] to mark struct/function that are defined elsewhere.

* Allow adding extern to any decl.

* Use ExternAtrtibute to apply import decoration, rather than use an ir extern decoration.

* Added a test for [__extern]

* Improved comment around [__extern]

9 files changed, 112 insertions, 4 deletions

diff --git a/source/slang/core.meta.slang b/source/slang/core.meta.slang
index 3b94c4e05..763bd4a78 100644
--- a/source/slang/core.meta.slang
+++ b/source/slang/core.meta.slang
@@ -1318,3 +1318,7 @@ attribute_syntax [format(format : String)] : FormatAttribute;
 __attributeTarget(Decl)
 attribute_syntax [allow(diagnostic: String)] : AllowAttribute;
 
+// Linking
+__attributeTarget(Decl)
+attribute_syntax [__extern] : ExternAttribute;
+
diff --git a/source/slang/core.meta.slang.h b/source/slang/core.meta.slang.h
index cc8b796e0..7cb729722 100644
--- a/source/slang/core.meta.slang.h
+++ b/source/slang/core.meta.slang.h
@@ -1349,3 +1349,7 @@ SLANG_RAW("\n")
 SLANG_RAW("__attributeTarget(Decl)\n")
 SLANG_RAW("attribute_syntax [allow(diagnostic: String)] : AllowAttribute;\n")
 SLANG_RAW("\n")
+SLANG_RAW("// Linking\n")
+SLANG_RAW("__attributeTarget(Decl)\n")
+SLANG_RAW("attribute_syntax [__extern] : ExternAttribute;\n")
+SLANG_RAW("\n")
diff --git a/source/slang/slang-ir-link.cpp b/source/slang/slang-ir-link.cpp
index 2d497158d..88f385548 100644
--- a/source/slang/slang-ir-link.cpp
+++ b/source/slang/slang-ir-link.cpp
@@ -979,6 +979,12 @@ bool isBetterForTarget(
     IRInst*         newVal,
     IRInst*         oldVal)
 {
+    // Anything is better than nothing..
+    if (oldVal == nullptr)
+    {
+        return true;
+    }
+
     String targetName = getTargetName(context);
 
     // For right now every declaration might have zero or more
@@ -1000,7 +1006,7 @@ bool isBetterForTarget(
     //     (A and B and C) or (A and D) or (E) or (F and G) ...
     //
     // A code generation target would then consist of a
-    // conjunction of invidual tags:
+    // conjunction of individual tags:
     //
     //    (HLSL and SM_4_0 and Vertex and ...)
     //
@@ -1012,6 +1018,7 @@ bool isBetterForTarget(
     // of the other's.
 
     auto newLevel = getTargetSpecialiationLevel(newVal, targetName);
+    
     auto oldLevel = getTargetSpecialiationLevel(oldVal, targetName);
     if(newLevel != oldLevel)
         return UInt(newLevel) > UInt(oldLevel);
@@ -1182,18 +1189,23 @@ IRInst* cloneGlobalValueWithLinkage(
 
     // We will try to track the "best" declaration we can find.
     //
-    // Generally, one declaration wil lbe better than another if it is
+    // Generally, one declaration will be better than another if it is
     // more specialized for the chosen target. Otherwise, we simply favor
     // definitions over declarations.
     //
-    IRInst* bestVal = sym->irGlobalValue;
-    for( auto ss = sym->nextWithSameName; ss; ss = ss->nextWithSameName )
+    IRInst* bestVal = nullptr;
+    for(IRSpecSymbol* ss = sym; ss; ss = ss->nextWithSameName )
     {
         IRInst* newVal = ss->irGlobalValue;
         if(isBetterForTarget(context, newVal, bestVal))
             bestVal = newVal;
     }
 
+    if (!bestVal)
+    {
+        return nullptr;
+    }
+
     // Check if we've already cloned this value, for the case where
     // we didn't have an original value (just a name), but we've
     // now found a representative value.
diff --git a/source/slang/slang-lower-to-ir.cpp b/source/slang/slang-lower-to-ir.cpp
index e8e2ff61a..f50bd8be1 100644
--- a/source/slang/slang-lower-to-ir.cpp
+++ b/source/slang/slang-lower-to-ir.cpp
@@ -422,6 +422,22 @@ bool isFromStdLib(Decl* decl)
 
 bool isImportedDecl(IRGenContext* context, Decl* decl)
 {
+    // If the declaration has the extern attribute then it must be imported
+    // from another module
+    //
+    // The [__extern] attribute is a very special case feature (aka "a hack") that allows a symbol to be declared
+    // as if it is part of the current module for AST purposes, but then expects to be imported from another IR module.
+    // For that linkage to work, both the exporting and importing modules must have the same name (which would
+    // usually indicate that they are the same module).
+    //
+    // Note that in practice for matching during linking uses the fully qualified name - including module name.
+    // Thus using extern __attribute isn't useful for symbols that are imported via `import`, only symbols
+    // that notionally come from the same module but are split into separate compilations (as can be done with -module-name)
+    if (decl->FindModifier<ExternAttribute>())
+    {
+        return true;
+    }
+
     ModuleDecl* moduleDecl = findModuleDecl(decl);
     if (!moduleDecl)
         return false;
diff --git a/source/slang/slang-modifier-defs.h b/source/slang/slang-modifier-defs.h
index 5ac61b991..fb3292e23 100644
--- a/source/slang/slang-modifier-defs.h
+++ b/source/slang/slang-modifier-defs.h
@@ -460,3 +460,8 @@ END_SYNTAX_CLASS()
 SYNTAX_CLASS(AllowAttribute, Attribute)
     FIELD_INIT(DiagnosticInfo const*, diagnostic, nullptr)
 END_SYNTAX_CLASS()
+
+
+// A `[__extern]` attribute, which indicates that a function/type is defined externally
+//
+SIMPLE_SYNTAX_CLASS(ExternAttribute, Attribute)
diff --git a/tests/serialization/extern/extern-test.slang b/tests/serialization/extern/extern-test.slang
new file mode 100644
index 000000000..0079a3527
--- /dev/null
+++ b/tests/serialization/extern/extern-test.slang
@@ -0,0 +1,26 @@
+// serialized-module-entry-point-test.slang
+
+//TEST:COMPILE: -module-name module -no-codegen tests/serialization/extern/module-a.slang -o tests/serialization/extern/module-a.slang-lib
+//TEST:COMPILE: -module-name module -no-codegen tests/serialization/extern/module-b.slang -o tests/serialization/extern/module-b.slang-lib
+//TEST:COMPARE_COMPUTE_EX: -xslang -module-name -xslang module -slang -compute -xslang -r -xslang tests/serialization/extern/module-a.slang-lib -xslang -r -xslang tests/serialization/extern/module-b.slang-lib 
+
+//TEST_INPUT:ubuffer(data=[0 0 0 0 ], stride=4):dxbinding(0),glbinding(0),out,name outputBuffer
+RWStructuredBuffer<int> outputBuffer;
+
+// Declare the type exists
+[__extern] struct Thing {};
+// A mechanism to make a Thing without knowing the specific fields.
+[__extern] Thing makeThing(int a, int b);
+
+[__extern] int doSomething(Thing a, Thing b);
+
+[numthreads(4, 1, 1)]
+void computeMain(uint3 dispatchThreadID : SV_DispatchThreadID)
+{
+    int index = int(dispatchThreadID.x);
+
+    Thing a = makeThing(index + 1, index + 2);
+    Thing b = makeThing(-index , index * 2);
+    
+    outputBuffer[index] = doSomething(a, b);
+}
\ No newline at end of file
diff --git a/tests/serialization/extern/extern-test.slang.expected.txt b/tests/serialization/extern/extern-test.slang.expected.txt
new file mode 100644
index 000000000..e326b6740
--- /dev/null
+++ b/tests/serialization/extern/extern-test.slang.expected.txt
@@ -0,0 +1,4 @@
+5
+B
+11
+17
diff --git a/tests/serialization/extern/module-a.slang b/tests/serialization/extern/module-a.slang
new file mode 100644
index 000000000..81816a213
--- /dev/null
+++ b/tests/serialization/extern/module-a.slang
@@ -0,0 +1,23 @@
+//TEST_IGNORE_FILE:
+
+// module-a.slang
+
+struct Thing
+{
+    int a; 
+    int b;
+};
+
+Thing makeThing(int a, int b)
+{
+    Thing thing;
+    thing.a = a;
+    thing.b = b;
+    return thing;
+}
+
+int foo(Thing thing) 
+{ 
+    return thing.a + thing.b * 2; 
+}
+
diff --git a/tests/serialization/extern/module-b.slang b/tests/serialization/extern/module-b.slang
new file mode 100644
index 000000000..20371f156
--- /dev/null
+++ b/tests/serialization/extern/module-b.slang
@@ -0,0 +1,14 @@
+//TEST_IGNORE_FILE:
+
+// module-b.slang
+
+// This looks like a definition (and it is) but with [__extern] it's definition will be replaced at link time with a defintion
+[__extern] struct Thing {};
+[__extern] int foo(Thing thing);
+
+int doSomething(Thing a, Thing b)
+{
+    return foo(a) + foo(b);
+}
+
+