Standard library save/loadable (#1592)

* #include an absolute path didn't work - because paths were taken to always be relative.

* Fix handling of access modifiers inside type definition.

* Fix access problem for AST node.
Make dumping produce a single function with switch, to potentially make available without Dump specific access.

* WIP on serialization design doc.

* Remove project references to previously generated files.

* More docs on serialization design.

* Improve serialization documentation.
Remove unused function from IRSerialReader.

* Small fixes around naming. Remove long comment from slang-serialize.h - as covered in serialization.md

* Remove long comment in slang-serialize.h as covered in serialization.md

* More information about doing replacements on read for AST and problems surrounding.

* Typo fix.

* Spelling fixes.

* Value serialize.

* Value types with inheritence.

* Use value reflection serial conversion for more AST types

* Use automatic serialization on more of AST.

* Get the types via decltype, simplifies what the extractor has to do.

* Update the serialization.md for the value serialization.

* Small doc improvements.

* Update project.

* Remove ImportExternalDecl type
Added addImportSymbol and ImportSymbol type
Fixed bug in container which meant it wouldn't read back AST module

* Because of change of how imports and handled, store objects as SerialPointers.

* First pass symbol lookup from mangled names.

* Cache current module looked up from mangled name.

* Fix SourceLoc bug.
Improve comments.

* Added diagnostic on mangled symbol not being found

* Fix typo.

* WIP serializing stdlib.

* WIP serializing stdlib in.

* Fix problem serializing arrays that hold data that is already serialized.

* Remove clash of names in MagicTypeModifier.

* Make conversion from char to String explicit.
Fix reference count issue with SerialReader.

* Add code to save/load stdlib.

* Use return code to avoid warning -                 SerialContainerUtil::write(module, options, &stream))

* Make all String numeric ctors explicit.
Added isChar to UnownedStringSlice.
Added operator== for UnownedStringSlice to String to avoid need to convert to String and allocate.

* Add error check to readAllText.

* tabs -> spaces on String.h

* tab -> spaces String.cpp

* Remove msg for StringBuilder, just build inplace for exceptions.

* Check SerialClasses - for name clashes.
Renamed Modifier::name as Modifier::keywordName

* Handling of extensions when deserializing AST - updating the moduleDecl->mapTypeToCandidateExtensions

Co-authored-by: Tim Foley <tim.foley.is@gmail.com>

1 files changed, 20 insertions, 6 deletions

diff --git a/source/slang/slang-serialize.h b/source/slang/slang-serialize.h
index a48a7e216..ff402b35c 100644
--- a/source/slang/slang-serialize.h
+++ b/source/slang/slang-serialize.h
@@ -246,7 +246,9 @@ public:
     const List<SerialPointer>& getObjects() const { return m_objects; }
 
         /// Add an object to be kept in scope
-    void addScope(const RefObject* obj) { m_scope.add(obj); }
+    void addScopeWithoutAddRef(const RefObject* obj) { m_scope.add(obj); }
+        /// Add obj with a reference
+    void addScope(const RefObject* obj) { const_cast<RefObject*>(obj)->addReference(); m_scope.add(obj); }
 
         /// Used for attaching extra objects necessary for serializing
     SerialExtraObjects& getExtraObjects() { return m_extraObjects; }
@@ -324,9 +326,21 @@ public:
     SerialIndex writeObject(const NodeBase* ptr);
     SerialIndex writeObject(const RefObject* ptr);
 
+        /// Add an array - may need to convert to serialized format
     template <typename T>
     SerialIndex addArray(const T* in, Index count);
 
+    template <typename NATIVE_TYPE>
+    /// Add an array where all the elements are already in serialized format (ie there is no need to do a conversion)
+    SerialIndex addSerialArray(const void* elements, Index elementCount)
+    {
+        typedef SerialTypeInfo<NATIVE_TYPE> TypeInfo;
+        return addSerialArray(sizeof(typename TypeInfo::SerialType), SerialTypeInfo<NATIVE_TYPE>::SerialAlignment, elements, elementCount);
+    }
+
+        /// Add an array where all the elements are already in serialized format (ie there is no need to do a conversion)
+    SerialIndex addSerialArray(size_t elementSize, size_t alignment, const void* elements, Index elementCount);
+
         /// Add the string
     SerialIndex addString(const UnownedStringSlice& slice) { return _addStringSlice(SerialTypeKind::String, m_sliceMap, slice); }
     SerialIndex addString(const String& in);
@@ -362,8 +376,6 @@ protected:
 
     SerialIndex _addStringSlice(SerialTypeKind typeKind, SliceMap& sliceMap, const UnownedStringSlice& slice);
 
-    SerialIndex _addArray(size_t elementSize, size_t alignment, const void* elements, Index elementCount);
-
     SerialIndex _add(const void* nativePtr, SerialInfo::Entry* entry)
     {
         m_entries.add(entry);
@@ -399,7 +411,7 @@ SerialIndex SerialWriter::addArray(const T* in, Index count)
     if (std::is_same<T, ElementSerialType>::value)
     {
         // If they are the same we can just write out
-        return _addArray(sizeof(T), SLANG_ALIGN_OF(ElementSerialType), in, count);
+        return addSerialArray(sizeof(T), SLANG_ALIGN_OF(ElementSerialType), in, count);
     }
     else
     {
@@ -411,7 +423,7 @@ SerialIndex SerialWriter::addArray(const T* in, Index count)
         {
             ElementTypeInfo::toSerial(this, &in[i], &work[i]);
         }
-        return _addArray(sizeof(ElementSerialType), SLANG_ALIGN_OF(ElementSerialType), work.getBuffer(), count);
+        return addSerialArray(sizeof(ElementSerialType), SLANG_ALIGN_OF(ElementSerialType), work.getBuffer(), count);
     }
 }
 
@@ -478,7 +490,6 @@ struct SerialClass
 class SerialClasses : public RefObject
 {
 public:
-
         /// Will add it's own copy into m_classesByType
         /// In process will calculate alignment, offset etc for fields
         /// NOTE! the super set, *must* be an already added to this SerialClasses
@@ -492,6 +503,9 @@ public:
         /// Returns true if this cls is *owned* by this SerialClasses
     bool isOwned(const SerialClass* cls) const;
 
+        /// Returns true if the SerialClasses structure appears ok
+    bool isOk() const;
+
         /// Get a serial class based on its type/subType
     const SerialClass* getSerialClass(SerialTypeKind typeKind, SerialSubType subType) const
     {