StringUtil::extractLine (#989)

* Added extractLine line parsing to StringUtil. Use for matching lines instead of calcLines. calcLines uses extractLine to extract lines.
Fixed problems found in output of some tests- due to how a how final line is handled. Now a final line has a \r or \n\r combination, but nothing else after it, it is considered the last line (not the line after it).

* Use StringUtil::extractLine in slang-generate.

* Improved comment on extractLine

* Remove test code from StringUtil::extractLine

* Made StringUtil::extractLine act as if line terminators are 'separators'.
Added unit-test-string.cpp - to check behavior.

* Adding LineParser - not entirely necessary, but slightly easier to use.

* Improved LineParser::Iterator end testing.
Added improved tests for LineParser.

* Move line comparson after termination case - to fix problem with gcc release build.

* Make UnownedStringSlice handle comparison when begin is nullptr - as it uses memcmp and passing nullptr to memcmp is undefined, leading to optimizer being able to do some unfortunate optimizations on gcc.

1 files changed, 7 insertions, 34 deletions

diff --git a/source/slang/slang-source-loc.cpp b/source/slang/slang-source-loc.cpp
index faa7e77c3..e6882ff55 100644
--- a/source/slang/slang-source-loc.cpp
+++ b/source/slang/slang-source-loc.cpp
@@ -212,44 +212,17 @@ const List<uint32_t>& SourceFile::getLineBreakOffsets()
     if (m_lineBreakOffsets.getCount() == 0)
     {
         UnownedStringSlice content = getContent();
-
-        char const* begin = content.begin();
-        char const* end = content.end();
-
-        char const* cursor = begin;
-
-        // Treat the beginning of the file as a line break
-        m_lineBreakOffsets.add(0);
-
-        while (cursor != end)
+        char const* contentBegin = content.begin();
+      
+        while (true)
         {
-            int c = *cursor++;
-            switch (c)
+            auto line = StringUtil::extractLine(content);
+            if (line.begin() == nullptr)
             {
-                case '\r': case '\n':
-                {
-                    // When we see a line-break character we need
-                    // to record the line break, but we also need
-                    // to deal with the annoying issue of encodings,
-                    // where a multi-byte sequence might encode
-                    // the line break.
-
-                    // Check to make sure that the EOF hasn't been reached.
-                    if (cursor != end)
-                    {
-                        int d = *cursor;
-                        if ((c ^ d) == ('\r' ^ '\n'))
-                            cursor++;
-                    }
-
-                    m_lineBreakOffsets.add(uint32_t(cursor - begin));
-                    break;
-                }
-                default:
-                    break;
+                break;
             }
+            m_lineBreakOffsets.add(uint32_t(line.begin() - contentBegin));
         }
-
         // Note that we do *not* treat the end of the file as a line
         // break, because otherwise we would report errors like
         // "end of file inside string literal" with a line number