Reading artifact hierarchy from file system interface (#2787)

* #include an absolute path didn't work - because paths were taken to always be relative.

* WIP simplifying artifact interface.

* Use ContainedKind.

* Remove LazyCastableList.
Use ContainedKind for find.

* Remove ICastableList.

* Remove need for ICastableList.

* Remove IArtifactContainer.

* Small fixes.

* Small improvements around Artifact.

* Make explicit find is for *representations* that can cast.
Fix bug in handling casting in lookup.

* Made associated items artifacts too.

* Small fixes.

* Small improvements around writing a container.

* WIP artifact container format.

* Make the root a special case.

* Special case if the artifact doesn't have children/associated.

* First pass handling of interpretting a file system into artifact hierarchy.

* Explain the final structure. Make the file system available.

* Remove addArtifact from IArtifact interface - means will be compatible with previous version.

* Rename function to get compile result as a filesytem.

5 files changed, 434 insertions, 60 deletions

diff --git a/source/compiler-core/slang-artifact-container-util.cpp b/source/compiler-core/slang-artifact-container-util.cpp
index a5cbf1ff1..29e2e736e 100644
--- a/source/compiler-core/slang-artifact-container-util.cpp
+++ b/source/compiler-core/slang-artifact-container-util.cpp
@@ -3,79 +3,76 @@
 
 #include "slang-artifact-util.h"
 #include "slang-artifact-desc-util.h"
+#include "slang-artifact-representation-impl.h"
 
 #include "../core/slang-file-system.h"
 #include "../core/slang-io.h"
 #include "../core/slang-zip-file-system.h"
 #include "../core/slang-castable.h"
+#include "../core/slang-string-slice-pool.h"
 
 namespace Slang {
 
-/* What is the structure we want here?
+/* 
+Artifact file structure
+=======================
 
-Lets say we have an artifact with "blah.spv". One approach would be to take the base name and so end up with
+There is many ways this could work, with different trade offs. The approach taken here is 
+to make *every* artifact be a directory. There are two special case directories "associated" and "children",
+which hold the artifacts associated/chidren artifacts. 
+
+So for example if we have 
 
 ```
-blah/
-blah/blah.spv
-blah/associated/...
-blah/children/...
+thing.spv
+  associated
+    diagnostics
 ```
 
-This is relatively simple and certainly pretty simple to implement. 
-
-This doesn't hold additional information - such as what might be held in the ArtifactDesc. We could use .dsc 
-to describe that. We could also remove the kernels name, as we already have that encoded in the directory name. 
-So 
+It will become
 
 ```
-blah/
-blah/bin.spv
-blah/desc.json
-blah/associated/...
-blah/children/...
+thing.spv
+associated/0/diagnostics.diag
 ```
 
-What happens if we have data that *doesn't* have a name? We can just produce names by incrementing some counter.
-
-This works but seems a little weird, in that the name of the output kernel is not reflected. We could just use an extension
-that we use for directories for this additional data. Note that it would be "nice" to make this work with just the basename,
-but there is a problem with files that don't have an extension (as seen with executables on unix-like scenarios).
-
 ```
-blah.spv
-blah.artifact/
-blah.artifact/desc.json
-blah.artifact/associated
-blah.artifact/... can just hold children directly
+somemodule
+  associated
+    diagnostics
+    0a0a0a.map
+    0b0b0b.map
 ```
-If we just hold children directly then their names could clash with standard names. 
-
-We might want an extension such that we know it is an artifact. 
-
-Other considerations
 
-* Wanting to be able to easily combine containers (by hand if necessary in a file system)
-  * Doing so implies we don't want some kind of global, or directory manifest.
-* Wanting to easily be able to 
+Becomes
 
-Given these constraints it seems like a good combination is
+```
+somemodule.slang-module
+associated/0/diagnostics
+associated/0a0a0a/0a0a0a.map
+associated/0b0b0b/0b0b0b.map
+```
 
-* Everything associated with an artifact is in a directory on it's own
-* The name of the item should probably be stored as is in the directory
-* Any ArtifactDesc or additional information should be held in a file in the directory
-* Children and associated items should be stored in their own directories (which will follow the same mechanisms recursively)
+That is a little verbose, but if the associated artifacts have children/associated, then things still work.
 
 ```
-blah/
-blah/blah.spv
-blah/desc.json
-blah/associated/...
-blah/children/...
+container
+  a.spv 
+    associated
+        diagnostics
+  b.dxil
+    associated
+        diagnostics
+        sourcemap
 ```
 
-Perhaps we want to special case for scenarios where we have artifact "leaves" (ie where they have no associated, or children)
-in that case placing in a directory all on it's own is perhaps not needed.
+```
+a/a.spv
+a/associated/0/diagnostics.diagnostics
+b/b.spv
+b/associated/0/diagnostics.diagnostics
+b/associated/1/sourcemap.map
+```
 
 */
 
@@ -259,23 +256,196 @@ SlangResult ArtifactContainerWriter::write(IArtifact* artifact)
     String baseName;
     SLANG_RETURN_ON_FAIL(getBaseName(artifact, baseName));
 
-    // Special case if there are no children/associated, we can just write into the same directory
-    if (artifact->getChildren().count == 0 &&
-        artifact->getAssociated().count == 0)
+    // We don't special case if the artifact contains no children/associated.
+    // We always create a directory for all artifacts. This makes it more verbose,
+    // but simplifies things, because *generally* an artifact including it's children/associated
+    // meta data is all contained in a single directory
+
     {
-        // I guess this assumes the name doesn't clash with any name we already have
+        Scope artifactScope;
+        SLANG_RETURN_ON_FAIL(artifactScope.pushAndRequireDirectory(this, baseName));
         SLANG_RETURN_ON_FAIL(writeInDirectory(artifact, baseName));
     }
+
+    return SLANG_OK;
+}
+
+/* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ArtifactContainerParser !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! */
+
+struct FileSystemContents
+{
+    // The first entry is always the root path
+
+    struct IndexRange
+    {
+        SLANG_FORCE_INLINE Index getCount() const { return endIndex - startIndex; }
+        
+        SLANG_FORCE_INLINE Index begin() const { return startIndex; }
+        SLANG_FORCE_INLINE Index end() const { return endIndex; }
+
+        void set(Index inStart, Index inEnd) { startIndex = inStart; endIndex = inEnd; }
+
+        static IndexRange make(Index inStart, Index inEnd) { IndexRange range; range.set(inStart, inEnd); return range; }
+
+        Index startIndex;
+        Index endIndex;
+    };
+
+    struct Entry
+    {
+        bool isFile() const { return range.startIndex < 0; }
+        bool isDirectory() const { return range.startIndex >= 0; }
+
+        void setDirectory() { range.set(0, 0); }
+        void setFile() { range.set(-1, -1); }
+
+        void setType(SlangPathType type) { (type == SLANG_PATH_TYPE_FILE) ? setFile() : setDirectory(); }
+
+        void setDirectoryRange(Index inStartIndex, Index inEndIndex)
+        {
+            SLANG_ASSERT(inEndIndex >= inStartIndex);
+            SLANG_ASSERT(isDirectory());
+            range.set(inStartIndex, inEndIndex);
+        }
+
+        Index parentDirectoryIndex = -1;                ///< The directory this entry is in. -1 is root.
+        UnownedStringSlice name;                        ///< Name of this entry
+        IndexRange range = IndexRange::make(-1, -1);    ///< Default to file
+    };
+    
+    void clear()
+    {
+        m_pool.clear();
+        m_entries.clear();
+    }
+    
+    IndexRange getContentsRange(Index index) const { return m_entries[index].range; }
+    
+    ConstArrayView<Entry> getContents(Index index) const { return getContents(m_entries[index]); } 
+    ConstArrayView<Entry> getContents(const Entry& entry) const
+    {
+        return entry.range.getCount() ? 
+            makeConstArrayView(m_entries.getBuffer() + entry.range.startIndex, entry.range.getCount()) :
+            makeConstArrayView<Entry>(nullptr, 0);
+    }
+     
+    void appendPath(Index entryIndex, StringBuilder& buf);
+
+    SlangResult find(ISlangFileSystemExt* fileSyste, const UnownedStringSlice& path);
+
+    FileSystemContents():
+        m_pool(StringSlicePool::Style::Default)
+    {
+        clear();
+    }
+
+    static void _add(SlangPathType pathType, const char* name, void* userData)
+    {
+        FileSystemContents* contents = (FileSystemContents*)userData;
+
+        Entry entry;
+        
+        entry.parentDirectoryIndex = contents->m_currentParent;
+        entry.name = contents->m_pool.addAndGetSlice(name);
+        entry.setType(pathType);
+
+        contents->m_entries.add(entry);
+    }
+
+    Index m_currentParent = -1;             ///< Convenience for adding entries when using enumerate
+
+    StringSlicePool m_pool;                 ///< Holds strings
+    List<Entry> m_entries;                  ///< The entries
+};
+
+void FileSystemContents::appendPath(Index entryIndex, StringBuilder& buf)
+{
+    const auto& entry = m_entries[entryIndex];
+    if (entry.parentDirectoryIndex >= 0)
+    {
+        // If there is a parent recurse to append that first
+        appendPath(entry.parentDirectoryIndex, buf);
+    }
+
+    // If the buffer is non zero, we need to add a separator
+    if (buf.getLength() > 0)
+    {
+        buf.appendChar('/');
+    }
+
+    buf.append(entry.name);
+}
+
+SlangResult FileSystemContents::find(ISlangFileSystemExt* fileSystem, const UnownedStringSlice& inPath)
+{
+    clear();
+
+    StringBuilder currentPath;
+    currentPath.append(inPath);
+
+    // If there is no name, just go with .
+    const char* checkPath = currentPath.getLength() ? currentPath.getBuffer() : ".";
+
+    SlangPathType pathType;
+    SLANG_RETURN_ON_FAIL(fileSystem->getPathType(checkPath, &pathType));
+
+    if (pathType == SLANG_PATH_TYPE_FILE)
+    {
+        Entry directoryEntry;
+        directoryEntry.parentDirectoryIndex = -1;
+        directoryEntry.name = m_pool.addAndGetSlice(Path::getParentDirectory(inPath));
+        directoryEntry.range.set(1, 2);
+        SLANG_ASSERT(directoryEntry.isDirectory());
+
+        m_entries.add(directoryEntry);
+
+        Entry entry;
+        entry.parentDirectoryIndex = 0;
+        entry.name = m_pool.addAndGetSlice(Path::getFileName(inPath));
+        SLANG_ASSERT(entry.isFile());
+
+        m_entries.add(entry);
+    }
     else
     {
-        Scope artifactScope;
-        SLANG_RETURN_ON_FAIL(artifactScope.pushAndRequireDirectory(this, baseName));
-        SLANG_RETURN_ON_FAIL(writeInDirectory(artifact, baseName));
+        Entry directoryEntry;
+        directoryEntry.setDirectory();
+        
+        directoryEntry.name = m_pool.addAndGetSlice(inPath);
+        m_entries.add(directoryEntry);
+
+        for (Index i = 0; i < m_entries.getCount(); ++i)
+        {
+            const Entry entry = m_entries[i];
+
+            if (entry.isDirectory())
+            {
+                // Clear the current path
+                currentPath.Clear();
+
+                appendPath(i, currentPath);
+
+                // Makes all the items added have this set as their parent
+                m_currentParent = i;
+
+                const auto startIndex = m_entries.getCount();
+
+                const char*const path = currentPath.getLength() ? currentPath.getBuffer() : ".";
+
+                const auto res = fileSystem->enumeratePathContents(path, _add, this);
+                
+                m_entries[i].setDirectoryRange(startIndex, m_entries.getCount());
+            
+                SLANG_RETURN_ON_FAIL(res);
+            }
+        }
     }
 
     return SLANG_OK;
 }
 
+/* !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ArtifactContainerUtil !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! */
+
 /* static */SlangResult ArtifactContainerUtil::writeContainer(IArtifact* artifact, const String& defaultFileName, ISlangMutableFileSystem* fileSystem)
 {
     ArtifactContainerWriter writer(fileSystem);
@@ -407,7 +577,7 @@ static SlangResult _remove(ISlangMutableFileSystem* fileSystem, const String& pa
 
         // Okay we can now write out the zip
         SLANG_RETURN_ON_FAIL(osFileSystem->saveFileBlob(fileName.getBuffer(), blob));
-
+        
         return SLANG_OK;
     }
     else if (ext == toSlice("dir"))
@@ -433,14 +603,192 @@ static SlangResult _remove(ISlangMutableFileSystem* fileSystem, const String& pa
     return SLANG_OK;
 }
 
+struct ArtifactContainerReader
+{
+    SlangResult read(ISlangFileSystemExt* fileSystem, ComPtr<IArtifact>& outArtifact);
+
+        /// A directory that contains multiple artifact directories
+    SlangResult _readContainerDirectory(Index directoryIndex, IArtifact::ContainedKind kind, IArtifact* container);
+        /// A directory that holds a single  
+    SlangResult _readArtifactDirectory(Index directoryIndex, ComPtr<IArtifact>& outArtifact);
+
+    SlangResult _readFile(Index fileIndex, ComPtr<IArtifact>& outArtifact);
+
+    FileSystemContents m_contents;
+    ISlangFileSystemExt* m_fileSystem;
+};
+
+SlangResult ArtifactContainerReader::read(ISlangFileSystemExt* fileSystem, ComPtr<IArtifact>& outArtifact)
+{
+    m_fileSystem = fileSystem;
+    m_contents.find(fileSystem, toSlice(""));
+
+    return _readArtifactDirectory(0, outArtifact);
+}
+
+
+SlangResult ArtifactContainerReader::_readFile(Index fileIndex, ComPtr<IArtifact>& outArtifact)
+{
+    outArtifact.setNull();
+    
+    const auto& entry = m_contents.m_entries[fileIndex];
+    SLANG_ASSERT(entry.isFile());
+
+    ArtifactDesc desc;
+
+    auto ext = Path::getPathExt(entry.name);
+    if (ext.getLength() == 0)
+    {
+        // I guess we'll assume it's an executable for now. We should use some kind of associated information/manifest 
+        // probly
+        desc = ArtifactDesc::make(ArtifactKind::Executable, ArtifactPayload::HostCPU);
+    }
+    else
+    {
+        desc = ArtifactDescUtil::getDescFromPath(entry.name);
+    }
+
+    // Don't know what this is. 
+    if (desc.kind == ArtifactKind::Unknown ||
+        desc.kind == ArtifactKind::Invalid)
+    {
+        return SLANG_OK;
+    }
+
+    // I guess I can just make an artifact for this
+    auto artifact = ArtifactUtil::createArtifact(desc);
+
+    if (entry.name.getLength())
+    {
+        // We can set the name on the artifact if set
+        // We know it's 0 terminated, because all names are in the pool 
+        // and therefore have to have 0 termination
+        artifact->setName(entry.name.begin());
+    }
+
+    StringBuilder path;
+    m_contents.appendPath(fileIndex, path);
+
+    IExtFileArtifactRepresentation* rep = new ExtFileArtifactRepresentation(path.getUnownedSlice(), m_fileSystem);
+    artifact->addRepresentation(rep);
+ 
+    outArtifact = artifact;
+    return SLANG_OK;
+}
+
+SlangResult ArtifactContainerReader::_readContainerDirectory(Index directoryIndex, IArtifact::ContainedKind kind, IArtifact* containerArtifact)
+{
+    // This directory only contains other directories which are artifacts
+    // Files are ignored 
+
+    auto indexRange = m_contents.getContentsRange(directoryIndex);
+
+    for (Index i = indexRange.startIndex; i < indexRange.endIndex; ++i)
+    {
+        const auto& entry = m_contents.m_entries[i];
+
+        // We ignore files
+        if (entry.isFile())
+        {
+            continue;
+        }
+
+        ComPtr<IArtifact> artifact;
+
+        SLANG_RETURN_ON_FAIL(_readArtifactDirectory(i, artifact));
+        
+        if (artifact)
+        {
+            switch (kind)
+            {
+                case IArtifact::ContainedKind::Associated:         containerArtifact->addAssociated(artifact); break;
+                case IArtifact::ContainedKind::Children:           containerArtifact->addChild(artifact); break;
+                default: SLANG_ASSERT(!"Can't add artifact to this kind"); return SLANG_FAIL;
+            }
+        }
+    }
+
+    return SLANG_OK;
+}
+
+SlangResult ArtifactContainerReader::_readArtifactDirectory(Index directoryIndex, ComPtr<IArtifact>& outArtifact)
+{
+    auto indexRange = m_contents.getContentsRange(directoryIndex);
+
+    Index childrenIndex = -1;
+    Index associatedIndex = -1;
+
+    ComPtr<IArtifact> artifact;
+
+    // Look for files
+    for (Index i = indexRange.startIndex; i < indexRange.endIndex; ++i)
+    {
+        const auto& entry = m_contents.m_entries[i];
+        if (entry.isFile())
+        {
+            ComPtr<IArtifact> readArtifact;
+            SLANG_RETURN_ON_FAIL(_readFile(i, readArtifact));
+            
+            if (readArtifact)
+            {
+                if (artifact)
+                {
+                    // We can only have one artifact in the directory
+                    return SLANG_FAIL;
+                }
+                artifact = readArtifact;
+            }
+        }
+        else if (entry.isDirectory())
+        {
+            if (entry.name == toSlice("associated"))
+            {
+                associatedIndex = i;
+            }
+            else if (entry.name == toSlice("children"))
+            {
+                childrenIndex = i;
+            }
+        }
+    }
+
+    // If we didn't find an artifact so far
+    if (!artifact)
+    {
+        // If we have children/associated we can assume it's a container
+        if (childrenIndex >= 0 || associatedIndex >= 0)
+        {
+            artifact = ArtifactUtil::createArtifact(ArtifactDesc::make(ArtifactKind::Container, ArtifactPayload::Unknown));
+            artifact->setName(m_contents.m_entries[directoryIndex].name.begin());
+        }
+        else
+        {
+            // Didn't find anything
+            return SLANG_OK;
+        }
+    }
+
+    if (childrenIndex >= 0)
+    {
+        SLANG_RETURN_ON_FAIL(_readContainerDirectory(childrenIndex, IArtifact::ContainedKind::Children, artifact));
+    }
+    if (associatedIndex >= 0)
+    {
+        SLANG_RETURN_ON_FAIL(_readContainerDirectory(associatedIndex, IArtifact::ContainedKind::Associated, artifact));
+    }
+
+    outArtifact = artifact;
+    return SLANG_OK;
+}
+
 /* static */SlangResult ArtifactContainerUtil::readContainer(ISlangFileSystemExt* fileSystem, ComPtr<IArtifact>& outArtifact)
 {
-    SLANG_UNUSED(fileSystem);
     SLANG_UNUSED(outArtifact);
 
-    // We traverse the file system creating the artifacts
+    ArtifactContainerReader reader;
+    SLANG_RETURN_ON_FAIL(reader.read(fileSystem, outArtifact));
 
-    return SLANG_E_NOT_IMPLEMENTED;
+    return SLANG_OK;
 }
 
 } // namespace Slang
diff --git a/source/core/slang-string-slice-pool.h b/source/core/slang-string-slice-pool.h
index 765119360..15d204468 100644
--- a/source/core/slang-string-slice-pool.h
+++ b/source/core/slang-string-slice-pool.h
@@ -64,6 +64,11 @@ public:
         /// Add a string
     Handle add(const String& string) { return add(string.getUnownedSlice()); }
 
+        /// Add and get the result as a slice
+    Slice addAndGetSlice(const Slice& slice) { return getSlice(add(slice)); }
+    Slice addAndGetSlice(const char* chars) { return getSlice(add(chars)); }
+    Slice addAndGetSlice(const String& string) { return getSlice(add(string)); }
+
         /// Returns true if found
     bool findOrAdd(const Slice& slice, Handle& outHandle);
 
diff --git a/source/slang/slang-compiler.cpp b/source/slang/slang-compiler.cpp
index 1a4a12dfe..43a6d238e 100644
--- a/source/slang/slang-compiler.cpp
+++ b/source/slang/slang-compiler.cpp
@@ -1853,7 +1853,6 @@ namespace Slang
         {
         case ContainerFormat::SlangModule:
         {
-
             OwnedMemoryStream stream(FileAccess::Write);
             SlangResult res = writeContainerToStream(&stream);
             if (SLANG_FAILED(res))
@@ -1869,7 +1868,6 @@ namespace Slang
             auto containerBlob = ListBlob::moveCreate(blobData);
 
             m_containerArtifact = Artifact::create(ArtifactDesc::make(Artifact::Kind::CompileBinary, ArtifactPayload::SlangIR, ArtifactStyle::Unknown));
-
             m_containerArtifact->addRepresentationUnknown(containerBlob);
 
             return res;
diff --git a/source/slang/slang-compiler.h b/source/slang/slang-compiler.h
index 52d154e5c..0c366f485 100755
--- a/source/slang/slang-compiler.h
+++ b/source/slang/slang-compiler.h
@@ -2594,6 +2594,7 @@ namespace Slang
         virtual SLANG_NO_THROW SlangResult SLANG_MCALL getTargetCodeBlob(int targetIndex, ISlangBlob** outBlob) SLANG_OVERRIDE;
         virtual SLANG_NO_THROW SlangResult SLANG_MCALL getTargetHostCallable(int targetIndex, ISlangSharedLibrary** outSharedLibrary) SLANG_OVERRIDE;
         virtual SLANG_NO_THROW void const* SLANG_MCALL getCompileRequestCode(size_t* outSize) SLANG_OVERRIDE;
+        virtual SLANG_NO_THROW ISlangMutableFileSystem* SLANG_MCALL getCompileRequestResultAsFileSystem() SLANG_OVERRIDE;
         virtual SLANG_NO_THROW SlangResult SLANG_MCALL getContainerCode(ISlangBlob** outBlob) SLANG_OVERRIDE;
         virtual SLANG_NO_THROW SlangResult SLANG_MCALL loadRepro(ISlangFileSystem* fileSystem, const void* data, size_t size) SLANG_OVERRIDE;
         virtual SLANG_NO_THROW SlangResult SLANG_MCALL saveRepro(ISlangBlob** outBlob) SLANG_OVERRIDE;
@@ -2632,6 +2633,8 @@ namespace Slang
             /// Where the container is stored. This is calculated as part of compile if m_containerFormat is set to
             /// a supported format. 
         ComPtr<IArtifact> m_containerArtifact;
+            /// Holds the container as a file system
+        ComPtr<ISlangMutableFileSystem> m_containerFileSystem;
 
             // Path to output container to
         String m_containerOutputPath;
diff --git a/source/slang/slang.cpp b/source/slang/slang.cpp
index 597e27bb9..4904abe03 100644
--- a/source/slang/slang.cpp
+++ b/source/slang/slang.cpp
@@ -13,6 +13,9 @@
 #include "../compiler-core/slang-artifact-desc-util.h"
 #include "../compiler-core/slang-artifact-util.h"
 #include "../compiler-core/slang-artifact-associated-impl.h"
+#include "../compiler-core/slang-artifact-container-util.h"
+
+#include "../core/slang-memory-file-system.h"
 
 #include "slang-module-library.h"
 
@@ -5234,6 +5237,23 @@ char const* EndToEndCompileRequest::getEntryPointSource(int entryPointIndex)
     return (char const*)getEntryPointCode(entryPointIndex, nullptr);
 }
 
+ISlangMutableFileSystem* EndToEndCompileRequest::getCompileRequestResultAsFileSystem()
+{
+    if (!m_containerFileSystem)
+    {
+        if (m_containerArtifact)
+        {
+            ComPtr<ISlangMutableFileSystem> fileSystem(new MemoryFileSystem);
+            if (SLANG_SUCCEEDED(ArtifactContainerUtil::writeContainer(m_containerArtifact, "", fileSystem)))
+            {
+                m_containerFileSystem.swap(fileSystem);
+            }
+        }
+    }
+
+    return m_containerFileSystem;
+}
+
 void const* EndToEndCompileRequest::getCompileRequestCode(size_t* outSize)
 {
     if (m_containerArtifact)