AppVeyor: Run tests as part of AppVeyor builds

This includes a bunch of related changes:

- `slang-test`
  - Add a notion of an "output mode" that specifies whether we output to console (the default), or invoke the apprpriate AppVeyor command to update test status
  - Add a notion of test categories, so that tests can be tagged with categories, and then we can invoke only those tets in a given category, or choose to *exclude* tests with specific categories
  - Allow the `OSProcessSpawner` to look up an executable by "path" (meaning a full path is expected) or by "name" (meaning it should be allowed to look in the current directory, `PATH` environment variable, etc.). This was important to make sure that I can run `appveyor` without having to know its absolute path.

- AppVeyor configuration
  - Change badge to reflect new build account for organization (rather than a single-user account)
  - Remove attempt to set AppVeyor build version in a clever way, since it breaks links from GitHub to AppVeyor
  - Change order or configurations in the build matrix to front-load the Release build (which has the main tests)
  - Turn on `fast_finish` flag so we don't have to wait as long for failed builds
  - Turn on `parallel` builds
  - Set `verbosity: minimal` to avoid getting build spew about Xamarin stuff I'm not using
  - Add custom `test_script` to invoke `test.bat`
    - Sets the test category based on teh build configuration, so we don't run the full test suite on every input.

- `test.bat`
  - Allow for `-platform` and `-configuration` arguments
  - Rewrute a platform of `Win32` over to `x86` to match how the output directories are named
  - Futz around with how the directories are being passed along to work around annoying `.bat` file quoting behavior (I still don't get how batch files work)

- Tests
  - Mark a bunch of tests as `smoke` tests
  - Mark the relevant tests as `render` tests
    (these get filtered out for AppVeyor builds)

15 files changed, 467 insertions, 81 deletions

diff --git a/README.md b/README.md
index 63608bda6..33bff684c 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # Slang
 
-[![AppVeyor build status](https://ci.appveyor.com/api/projects/status/kt9ch5niwkslk5p4/branch/master?svg=true)](https://ci.appveyor.com/project/tangent-vector/slang/branch/master)
+[![AppVeyor build status](https://ci.appveyor.com/api/projects/status/3jptgsry13k6wdwp/branch/master?svg=true)](https://ci.appveyor.com/project/shader-slang/slang/branch/master)
 
 Slang is a library for compiling real-time shader code.
 It can be used with either existing HLSL or GLSL code, or with code written directly in Slang.
diff --git a/appveyor.yml b/appveyor.yml
index 9052ec9b3..132bffb6b 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -1,34 +1,3 @@
-
-# Try to set AppVeyord build "version" to something reasonable,
-# based on what we are trying to build.
-init:
-  - ps: |
-      if ($env:APPVEYOR_REPO_TAG -eq "true")
-      {
-          # If we are building from a tag in the repository,
-          # then use the tag name as the start of the build version
-          $versionName = "$($env:APPVEYOR_REPO_TAG_NAME.TrimStart("v"))"
-      }
-      elseif($env:APPVEYOR_PULL_REQUEST_NUMBER)
-      {
-          # Otherwise, if we are building for a pull request,
-          # then use the pull request number to name our build version
-          $versionName = "pr$env:APPVEYOR_PULL_REQUEST_NUMBER"
-      }
-      else
-      {
-          # Otherwise, just set a build name based on the
-          # branch that we are building against
-          $versionName = "$env:APPVEYOR_REPO_BRANCH"
-      }
-      # Finally, we set the AppVeyor build version to use our version
-      # name with the build number appended.
-      #
-      # The string "appveyor" is being included before the build number
-      # in case we need to differentiate builds being made with different
-      # continuous integration providers.
-      Update-AppveyorBuild -Version "$versionName+appveyor.$env:APPVEYOR_BUILD_NUMBER"
-
 # Our solution file is currently set up for VS2015
 image: Visual Studio 2015
 
@@ -42,17 +11,48 @@ install:
 platform:
   - Win32
   - x64
+
 configuration:
-  - Debug
   - Release
+  - Debug
+
+# In the interests of time, go ahead and immediately fail a build
+# if any job fails, rather than keep on building to discover the
+# full set of failures.
+matrix:
+  fast_finish: true
 
 # MSBUILD should ideally be able to find our solution file
 # automatically, but it seems to get confused, so we specify
 # the file name to use here.
 build:
   project: slang.sln
+  parallel: true
+  verbosity: minimal
+
+# Testing
 
-# TODO: need to invoke our test script as part of the build
+# We only run tests on the Release build for now, just to speed
+# up the build process.
+#
+# TODO: We should really define different levels of tests, and
+# at least run some "smoke" tests across all builds.
+
+test_script:
+  - ps: |
+      if ($env:CONFIGURATION -eq "Debug")
+      {
+        $testCategory = "smoke"
+      }
+      elseif($env:PLATFORM -eq "x64")
+      {
+        $testCategory = "full"
+      }
+      else
+      {
+        $testCategory = "quick"
+      }
+      .\test.bat -platform %PLATFORM% -configuration %CONFIGURATION% -appveyor -category $testCategory
 
 # TODO: need to figure out what we want to package for deployment
 
diff --git a/test.bat b/test.bat
index b4398b991..38378a2e3 100644
--- a/test.bat
+++ b/test.bat
@@ -16,19 +16,40 @@ if "%1"=="-release" (
 	shift
 	goto :ARGLOOP
 )
+if "%1"=="-platform" (
+	set SLANG_TEST_PLATFORM=%2
+	shift
+	shift
+	goto :ARGLOOP
+)
+if "%1"=="-configuration" (
+	set SLANG_TEST_CONFIG=%2
+	shift
+	shift
+	goto :ARGLOOP
+)
 
 :: When done with arguments, we'll just fall through here
 
+:: Set root directory to the directory where `test.bat` resides
+:: (which should be the root of the source tree)
 SET "SLANG_TEST_ROOT=%~dp0"
 
+:: If  platform and configuration haven't been set, then set
+:: them to default values.
 IF "%SLANG_TEST_PLATFORM%" == "" ( SET "SLANG_TEST_PLATFORM=x86" )
 IF "%SLANG_TEST_CONFIG%" == "" ( SET "SLANG_TEST_CONFIG=Debug" )
 
-set "SLANG_TEST_BIN_DIR=%SLANG_TEST_ROOT%bin\%SLANG_TEST_PLATFORM%\%SLANG_TEST_CONFIG%\\"
+:: If the user specified a platform of "Win32" swap that to "x86"
+:: to match how we are generating our output directories.
+IF "%SLANG_TEST_PLATFORM%"=="Win32" ( Set "SLANG_TEST_PLATFORM=x86" )
+
+:: Establish the directory where the binaries to be tested reside
+set "SLANG_TEST_BIN_DIR=%SLANG_TEST_ROOT%bin\%SLANG_TEST_PLATFORM%\%SLANG_TEST_CONFIG%\"
 
 :: ensure that any built tools are visible
 SET "PATH=%PATH%;%SLANG_TEST_BIN_DIR%"
 
-:: TODO: ensure that everything is built?
+:: TODO: Maybe we should actually invoke `msbuild` to make sure all the code is up to date?
 
-"%SSLANG_TEST_BIN_DIR%slang-test.exe" --bindir "%SLANG_TEST_BIN_DIR%" %*
+"%SLANG_TEST_BIN_DIR%slang-test.exe" -bindir "%SLANG_TEST_BIN_DIR%\" %*
diff --git a/tests/bindings/targets-and-uavs-structure.hlsl b/tests/bindings/targets-and-uavs-structure.hlsl
index dcc053253..4ec64fb75 100644
--- a/tests/bindings/targets-and-uavs-structure.hlsl
+++ b/tests/bindings/targets-and-uavs-structure.hlsl
@@ -1,4 +1,4 @@
-//TEST:COMPARE_HLSL: -target dxbc-assembly -profile ps_5_0 -entry main
+//TEST(smoke):COMPARE_HLSL: -target dxbc-assembly -profile ps_5_0 -entry main
 
 // Handle the case where the fragment shader output is
 // defined a structure, and the semantics are on the sub-fields
diff --git a/tests/diagnostics/call-argument-type.slang b/tests/diagnostics/call-argument-type.slang
index d9663147f..b4f7e7477 100644
--- a/tests/diagnostics/call-argument-type.slang
+++ b/tests/diagnostics/call-argument-type.slang
@@ -1,4 +1,4 @@
-//TEST:SIMPLE:
+//TEST(smoke):SIMPLE:
 // call function with wrong argument type
 
 struct A {};
diff --git a/tests/glsl/sascha-willems/tessellation/pntriangles.tese b/tests/glsl/sascha-willems/tessellation/pntriangles.tese
index 0187ad781..5faf27f9d 100644
--- a/tests/glsl/sascha-willems/tessellation/pntriangles.tese
+++ b/tests/glsl/sascha-willems/tessellation/pntriangles.tese
@@ -1,4 +1,4 @@
-//TEST:COMPARE_GLSL:
+//TEST(smoke):COMPARE_GLSL:
 #version 450
 
 #extension GL_ARB_separate_shader_objects : enable
diff --git a/tests/hlsl/dxsdk/AdaptiveTessellationCS40/Render.hlsl b/tests/hlsl/dxsdk/AdaptiveTessellationCS40/Render.hlsl
index b98b870da..c106c46e7 100644
--- a/tests/hlsl/dxsdk/AdaptiveTessellationCS40/Render.hlsl
+++ b/tests/hlsl/dxsdk/AdaptiveTessellationCS40/Render.hlsl
@@ -1,4 +1,4 @@
-//TEST:COMPARE_HLSL: -profile vs_4_0 -entry RenderBaseVS -profile ps_4_0 -entry RenderPS -target dxbc-assembly
+//TEST(smoke):COMPARE_HLSL: -profile vs_4_0 -entry RenderBaseVS -profile ps_4_0 -entry RenderPS -target dxbc-assembly
 //--------------------------------------------------------------------------------------
 // File: Render.hlsl
 //
diff --git a/tests/preprocessor/define-function-like.slang b/tests/preprocessor/define-function-like.slang
index f1dd9caa4..fa8294077 100644
--- a/tests/preprocessor/define-function-like.slang
+++ b/tests/preprocessor/define-function-like.slang
@@ -1,4 +1,4 @@
-//TEST:SIMPLE:
+//TEST(smoke):SIMPLE:
 // support for function-like macros
 
 #define FOO(x) 1.0 + x
diff --git a/tests/reflection/resource-in-cbuffer.hlsl b/tests/reflection/resource-in-cbuffer.hlsl
index 956387587..9ab127363 100644
--- a/tests/reflection/resource-in-cbuffer.hlsl
+++ b/tests/reflection/resource-in-cbuffer.hlsl
@@ -1,4 +1,4 @@
-//TEST:SIMPLE:-profile ps_4_0 -target reflection-json
+//TEST(smoke):SIMPLE:-profile ps_4_0 -target reflection-json
 
 // Confirm that we can generate reflection
 // information for resources nested inside
diff --git a/tests/render/cross-compile0.hlsl b/tests/render/cross-compile0.hlsl
index b482035d1..d300e2bd5 100644
--- a/tests/render/cross-compile0.hlsl
+++ b/tests/render/cross-compile0.hlsl
@@ -1,4 +1,4 @@
-//TEST:COMPARE_HLSL_GLSL_RENDER:
+//TEST(smoke,render):COMPARE_HLSL_GLSL_RENDER:
 
 // This is a basic test case for cross-compilation behavior.
 //
diff --git a/tests/render/render0.hlsl b/tests/render/render0.hlsl
index 3ecd582f3..5e04c7d63 100644
--- a/tests/render/render0.hlsl
+++ b/tests/render/render0.hlsl
@@ -1,4 +1,4 @@
-//TEST:COMPARE_HLSL_RENDER:
+//TEST(smoke,render):COMPARE_HLSL_RENDER:
 // Starting with a basic test for the ability to render stuff...
 
 cbuffer Uniforms
diff --git a/tests/rewriter/error0.hlsl b/tests/rewriter/error0.hlsl
index dc3e84fda..10957e9e5 100644
--- a/tests/rewriter/error0.hlsl
+++ b/tests/rewriter/error0.hlsl
@@ -1,4 +1,4 @@
-//TEST:COMPARE_HLSL: -no-checking -target dxbc-assembly -profile ps_4_0 -entry main
+//TEST(smoke):COMPARE_HLSL: -no-checking -target dxbc-assembly -profile ps_4_0 -entry main
 
 // We need to confirm that when there is an error in
 // the input code, we allow the downstream compiler
diff --git a/tools/slang-test/main.cpp b/tools/slang-test/main.cpp
index ed419f946..4940228fe 100644
--- a/tools/slang-test/main.cpp
+++ b/tools/slang-test/main.cpp
@@ -22,6 +22,20 @@ using namespace CoreLib::IO;
 #include <stdlib.h>
 #include <stdarg.h>
 
+enum OutputMode
+{
+    // Default mode is to write test results to the console
+    kOutputMode_Default = 0,
+
+    // When running under AppVeyor contiuous integration, we
+    // need to output test results in a way that the AppVeyor
+    // environment can pick up and display.
+    kOutputMode_AppVeyor,
+};
+
+struct TestCategory;
+TestCategory* findTestCategory(String const& name);
+
 struct Options
 {
     char const* appName = "slang-test";
@@ -37,6 +51,15 @@ struct Options
 
     // force generation of baselines for HLSL tests
     bool generateHLSLBaselines = false;
+
+    // kind of output to generate
+    OutputMode outputMode = kOutputMode_Default;
+
+    // Only run tests that match one of the given categories
+    Dictionary<TestCategory*, TestCategory*> includeCategories;
+
+    // Exclude test taht match one these categories
+    Dictionary<TestCategory*, TestCategory*> excludeCategories;
 };
 Options options;
 
@@ -74,7 +97,7 @@ void parseOptions(int* argc, char** argv)
             break;
         }
 
-        if( strcmp(arg, "--bindir") == 0 )
+        if( strcmp(arg, "-bindir") == 0 )
         {
             if( argCursor == argEnd )
             {
@@ -99,6 +122,56 @@ void parseOptions(int* argc, char** argv)
         {
             // Assumed to be handle by .bat file that called us
         }
+        else if( strcmp(arg, "-configuration") == 0 )
+        {
+            if( argCursor == argEnd )
+            {
+                fprintf(stderr, "error: expected operand for '%s'\n", arg);
+                exit(1);
+            }
+            argCursor++;
+            // Assumed to be handle by .bat file that called us
+        }
+        else if( strcmp(arg, "-platform") == 0 )
+        {
+            if( argCursor == argEnd )
+            {
+                fprintf(stderr, "error: expected operand for '%s'\n", arg);
+                exit(1);
+            }
+            argCursor++;
+            // Assumed to be handle by .bat file that called us
+        }
+        else if( strcmp(arg, "-appveyor") == 0 )
+        {
+            options.outputMode = kOutputMode_AppVeyor;
+        }
+        else if( strcmp(arg, "-category") == 0 )
+        {
+            if( argCursor == argEnd )
+            {
+                fprintf(stderr, "error: expected operand for '%s'\n", arg);
+                exit(1);
+            }
+            auto category = findTestCategory(*argCursor++);
+            if(category)
+            {
+                options.includeCategories.Add(category, category);
+            }
+        }
+        else if( strcmp(arg, "-exclude") == 0 )
+        {
+            if( argCursor == argEnd )
+            {
+                fprintf(stderr, "error: expected operand for '%s'\n", arg);
+                exit(1);
+            }
+            auto category = findTestCategory(*argCursor++);
+            if(category)
+            {
+                options.excludeCategories.Add(category, category);
+            }
+        }
         else
         {
             fprintf(stderr, "unknown option '%s'\n", arg);
@@ -231,11 +304,53 @@ String collectRestOfLine(char const** ioCursor)
     return getString(textBegin, textEnd);
 }
 
+// A category that a test can be tagged with
+struct TestCategory
+{
+    // The name of the category, from the user perspective
+    String name;
+
+    // The logical "super-category" of this category
+    TestCategory* parent;
+
+    // A list of categories that we explcitly want to exclude
+    List<TestCategory*> prohibitedCategories;
+};
+
+Dictionary<String, TestCategory*> testCategories;
+TestCategory* defaultTestCategory;
+
+TestCategory* addTestCategory(String const& name, TestCategory* parent)
+{
+    TestCategory* category = new TestCategory();
+    category->name = name;
+
+    category->parent = parent;
+
+    testCategories.Add(name, category);
+
+    return category;
+}
+
+TestCategory* findTestCategory(String const& name)
+{
+    TestCategory* category = nullptr;
+    if( !testCategories.TryGetValue(name, category) )
+    {
+        error("unknown test category name '%s'\n", name.Buffer());
+        return nullptr;
+    }
+    return category;
+}
+
 // Optiosn for a particular test
 struct TestOptions
 {
     String command;
     List<String> args;
+
+    // The categories that this test was assigned to
+    List<TestCategory*> categories;
 };
 
 // Information on tests to run for a particular file
@@ -250,7 +365,71 @@ TestResult gatherTestOptions(
 {
     char const* cursor = *ioCursor;
 
-    // Start by scanning for the sub-command name:
+    TestOptions testOptions;
+
+    // Right after the `TEST` keyword, the user may specify
+    // one or more categories for the test.
+    if(*cursor == '(')
+    {
+        cursor++;
+        // optional test category
+        skipHorizontalSpace(&cursor);
+        char const* categoryStart = cursor;
+        for(;;)
+        {
+            switch( *cursor )
+            {
+            default:
+                cursor++;
+                continue;
+
+            case ',':
+            case ')':
+                {
+                    char const* categoryEnd = cursor;
+                    cursor++;
+
+                    auto categoryName = getString(categoryStart, categoryEnd);
+                    TestCategory* category = findTestCategory(categoryName);
+
+                    if(!category)
+                    {
+                        return kTestResult_Fail;
+                    }
+
+                    testOptions.categories.Add(category);
+
+                    if( *categoryEnd == ',' )
+                    {
+                        skipHorizontalSpace(&cursor);
+                        categoryStart = cursor;
+                        continue;
+                    }
+                }
+                break;
+        
+            case 0: case '\r': case '\n':
+                return kTestResult_Fail;
+            }
+
+            break;
+        }
+    }
+
+    // If no categories were specified, then add the default category
+    if(testOptions.categories.Count() == 0)
+    {
+        testOptions.categories.Add(defaultTestCategory);
+    }
+
+    if(*cursor == ':')
+        cursor++;
+    else
+    {
+        return kTestResult_Fail;
+    }
+
+    // Next scan for a sub-command name
     char const* commandStart = cursor;
     for(;;)
     {
@@ -270,12 +449,16 @@ TestResult gatherTestOptions(
         break;
     }
     char const* commandEnd = cursor;
-    if(*cursor == ':')
-        cursor++;
 
-    TestOptions testOptions;
     testOptions.command = getString(commandStart, commandEnd);
 
+    if(*cursor == ':')
+        cursor++;
+    else
+    {
+        return kTestResult_Fail;
+    }
+
     // Now scan for arguments. For now we just assume that
     // any whitespace separation indicates a new argument
     // (we don't support quoting)
@@ -344,14 +527,14 @@ TestResult gatherTestsForFile(
         skipHorizontalSpace(&cursor);
 
         // Look for a pattern that matches what we want
-        if(match(&cursor, "//TEST:"))
+        if(match(&cursor, "//TEST_IGNORE_FILE"))
         {
-            if(gatherTestOptions(&cursor, testList) != kTestResult_Pass)
-                return kTestResult_Fail;
+            return kTestResult_Ignored;
         }
-        else if(match(&cursor, "//TEST_IGNORE_FILE"))
+        else if(match(&cursor, "//TEST"))
         {
-            return kTestResult_Ignored;
+            if(gatherTestOptions(&cursor, testList) != kTestResult_Pass)
+                return kTestResult_Fail;
         }
         else
         {
@@ -372,6 +555,7 @@ OSError spawnAndWait(String	testPath, OSProcessSpawner& spawner)
     OSError err = spawner.spawnAndWaitForCompletion();
     if (err != kOSError_None)
     {
+//        fprintf(stderr, "failed to run test '%S'\n", testPath.ToWString());
         error("failed to run test '%S'", testPath.ToWString());
     }
     return err;
@@ -426,7 +610,7 @@ TestResult runSimpleTest(TestInput& input)
 
     OSProcessSpawner spawner;
 
-    spawner.pushExecutableName(String(options.binDir) + "slangc.exe");
+    spawner.pushExecutablePath(String(options.binDir) + "slangc.exe");
     spawner.pushArgument(filePath999);
 
     for( auto arg : input.testOptions->args )
@@ -485,7 +669,7 @@ TestResult generateHLSLBaseline(TestInput& input)
     auto outputStem = input.outputStem;
 
     OSProcessSpawner spawner;
-    spawner.pushExecutableName(String(options.binDir) + "slangc.exe");
+    spawner.pushExecutablePath(String(options.binDir) + "slangc.exe");
     spawner.pushArgument(filePath999);
 
     for( auto arg : input.testOptions->args )
@@ -531,7 +715,7 @@ TestResult runHLSLComparisonTest(TestInput& input)
 
     OSProcessSpawner spawner;
 
-    spawner.pushExecutableName(String(options.binDir) + "slangc.exe");
+    spawner.pushExecutablePath(String(options.binDir) + "slangc.exe");
     spawner.pushArgument(filePath999);
 
     for( auto arg : input.testOptions->args )
@@ -621,7 +805,7 @@ TestResult doGLSLComparisonTestRun(
 
     OSProcessSpawner spawner;
 
-    spawner.pushExecutableName(String(options.binDir) + "slangc.exe");
+    spawner.pushExecutablePath(String(options.binDir) + "slangc.exe");
     spawner.pushArgument(filePath999);
 
     if( langDefine )
@@ -710,7 +894,7 @@ TestResult doRenderComparisonTestRun(TestInput& input, char const* langOption, c
 
     OSProcessSpawner spawner;
 
-    spawner.pushExecutableName(String(options.binDir) + "render-test.exe");
+    spawner.pushExecutablePath(String(options.binDir) + "render-test.exe");
     spawner.pushArgument(filePath999);
 
     for( auto arg : input.testOptions->args )
@@ -913,6 +1097,150 @@ struct TestContext
     int failedTestCount;
 };
 
+// deal with the fallout of a test having completed, whether
+// passed or failed or who-knows-what.
+void handleTestResult(
+    TestContext*    context,
+    String const&   testName,
+    TestResult      testResult)
+{
+    switch( testResult )
+    {
+    case kTestResult_Fail:
+        context->failedTestCount++;
+        break;
+
+    case kTestResult_Pass:
+        context->passedTestCount++;
+        break;
+
+    case kTestResult_Ignored:
+        // Note that we don't currently add ignored tests into
+        // the totals, which is kind of inaccurate.
+        break;
+
+    default:
+        assert(!"unexpected");
+        break;
+    }
+
+//    printf("OUTPUT_MODE: %d\n", options.outputMode);
+    switch( options.outputMode )
+    {
+    case kOutputMode_Default:
+        {
+            char const* resultString = "UNEXPECTED";
+            switch( testResult )
+            {
+            case kTestResult_Fail:      resultString = "FAILED";  break;
+            case kTestResult_Pass:      resultString = "passed";  break;
+            case kTestResult_Ignored:   resultString = "ignored"; break;
+            default:
+                assert(!"unexpected");
+                break;
+            }
+
+            printf("%s test: '%S'\n", resultString, testName.ToWString());
+        }
+        break;
+
+    case kOutputMode_AppVeyor:
+        {
+            char const* resultString = "None";
+            switch( testResult )
+            {
+            case kTestResult_Fail:      resultString = "Failed";  break;
+            case kTestResult_Pass:      resultString = "Passed";  break;
+            case kTestResult_Ignored:   resultString = "Ignored"; break;
+            default:
+                assert(!"unexpected");
+                break;
+            }
+
+
+            OSProcessSpawner spawner;
+            spawner.pushExecutableName("appveyor");
+            spawner.pushArgument("AddTest");
+            spawner.pushArgument(testName);
+            spawner.pushArgument("-FileName");
+            // TODO: this isn't actually a file name in all cases
+            spawner.pushArgument(testName);
+            spawner.pushArgument("-Framework");
+            spawner.pushArgument("slang-test");
+            spawner.pushArgument("-Outcome");
+            spawner.pushArgument(resultString);
+
+            auto err = spawner.spawnAndWaitForCompletion();
+
+            if( err != kOSError_None )
+            {
+                error("failed to add appveyor test results for '%S'\n", testName.ToWString());
+
+#if 0
+                fprintf(stderr, "[%d] TEST RESULT: %s {%d} {%s} {%s}\n", err, spawner.commandLine_.Buffer(),
+                    spawner.getResultCode(),
+                    spawner.getStandardOutput().begin(),
+                    spawner.getStandardError().begin());
+#endif
+            }
+        }
+        break;
+
+    default:
+        assert(!"unexpected");
+        break;
+    }
+}
+
+bool testCategoryMatches(
+    TestCategory*   sub,
+    TestCategory*   sup)
+{
+    auto ss = sub;
+    while(ss)
+    {
+        if(ss == sup)
+            return true;
+
+        ss = ss->parent;
+    }
+    return false;
+}
+
+bool testCategoryMatches(
+    TestCategory*                           categoryToMatch,
+    Dictionary<TestCategory*, TestCategory*> categorySet)
+{
+    for( auto item : categorySet )
+    {
+        if(testCategoryMatches(categoryToMatch, item.Value))
+            return true;
+    }
+    return false;
+}
+
+bool testPassesCategoryMask(
+    TestContext*        context,
+    TestOptions const&  test)
+{
+    // Don't include a test we should filter out
+    for( auto testCategory : test.categories )
+    {
+        if(testCategoryMatches(testCategory, options.excludeCategories))
+            return false;
+    }
+
+    // Otherwise inclue any test the user asked for
+    for( auto testCategory : test.categories )
+    {
+        if(testCategoryMatches(testCategory, options.includeCategories))
+            return true;
+    }
+
+    // skip by default
+    return false;
+}
+
 void runTestsOnFile(
     TestContext*    context,
     String          filePath)
@@ -929,10 +1257,7 @@ void runTestsOnFile(
     // Note cases where a test file exists, but we found nothing to run
     if( testList.tests.Count() == 0 )
     {
-        context->totalTestCount++;
-        context->failedTestCount++;
-
-        printf("FAILED test: '%S' (no test commands found)\n", filePath.ToWString());
+        handleTestResult(context, filePath, kTestResult_Ignored);
         return;
     }
 
@@ -940,9 +1265,16 @@ void runTestsOnFile(
     int subTestCount = 0;
     for( auto& tt : testList.tests )
     {
+        int subTestIndex = subTestCount++;
+
+        // Check that the test passes our current category mask
+        if(!testPassesCategoryMask(context, tt))
+        {
+            continue;
+        }
+
         context->totalTestCount++;
 
-        int subTestIndex = subTestCount++;
 
         String outputStem = filePath;
         if(subTestIndex != 0)
@@ -951,22 +1283,9 @@ void runTestsOnFile(
         }
 
         TestResult result = runTest(filePath, outputStem, tt, testList);
-        if(result == kTestResult_Ignored)
-            return;
 
-        if (result == kTestResult_Pass)
-        {
-            printf("passed");
-            context->passedTestCount++;
-        }
-        else
-        {
-            printf("FAILED");
-            context->failedTestCount++;
-        }
+        handleTestResult(context, outputStem, result);
 
-        printf(" test: '%S'", outputStem.ToWString());
-        printf("\n");
     }
 }
 
@@ -1038,8 +1357,37 @@ int main(
     int		argc,
     char**	argv)
 {
+    // Set up our test categories here
+
+    auto fullTestCategory = addTestCategory("full", nullptr);
+
+    auto quickTestCategory = addTestCategory("quick", fullTestCategory);
+
+    auto smokeTestCategory = addTestCategory("smoke", quickTestCategory);
+
+    auto renderTestCategory = addTestCategory("render", fullTestCategory);
+
+    // An un-categorized test will always belong to the `full` category
+    defaultTestCategory = fullTestCategory;
+
+    //
+
+
     parseOptions(&argc, argv);
 
+    if( options.includeCategories.Count() == 0 )
+    {
+        options.includeCategories.Add(fullTestCategory, fullTestCategory);
+    }
+
+    // Exclude rendering tests when building under AppVeyor.
+    //
+    // TODO: this is very ad hoc, and we should do something cleaner.
+    if( options.outputMode == kOutputMode_AppVeyor )
+    {
+        options.excludeCategories.Add(renderTestCategory, renderTestCategory);
+    }
+
     TestContext context = { 0 };
 
     // Enumerate test files according to policy
diff --git a/tools/slang-test/os.cpp b/tools/slang-test/os.cpp
index 8f5172ae6..4bc3d8e6d 100644
--- a/tools/slang-test/os.cpp
+++ b/tools/slang-test/os.cpp
@@ -205,6 +205,15 @@ void OSProcessSpawner::pushExecutableName(
 {
     executableName_ = executableName;
     commandLine_.Append(executableName);
+    isExecutablePath_ = false;
+}
+
+void OSProcessSpawner::pushExecutablePath(
+    CoreLib::Basic::String executablePath)
+{
+    executableName_ = executablePath;
+    commandLine_.Append(executablePath);
+    isExecutablePath_ = true;
 }
 
 void OSProcessSpawner::pushArgument(
@@ -305,7 +314,7 @@ OSError OSProcessSpawner::spawnAndWaitForCompletion()
 
     // `CreateProcess` requires write access to this, for some reason...
     BOOL success = CreateProcessW(
-        executableName_.ToWString(),
+        isExecutablePath_ ? executableName_.ToWString() : nullptr,
         (LPWSTR)commandLine_.ToString().ToWString(),
         nullptr,
         nullptr,
diff --git a/tools/slang-test/os.h b/tools/slang-test/os.h
index 2996001e7..5471e7e1a 100644
--- a/tools/slang-test/os.h
+++ b/tools/slang-test/os.h
@@ -129,6 +129,11 @@ struct OSProcessSpawner
     void pushExecutableName(
         CoreLib::Basic::String executableName);
 
+    // Set the executable name for the process to be spawned.
+    // Note: this call must be made before any arguments are pushed.
+    void pushExecutablePath(
+        CoreLib::Basic::String executablePath);
+
     // Append an argument for the process to be spawned.
     void pushArgument(
         CoreLib::Basic::String argument);
@@ -155,6 +160,9 @@ struct OSProcessSpawner
 #ifdef WIN32
     CoreLib::Basic::String executableName_;
     CoreLib::Basic::StringBuilder commandLine_;
+
+    // Is the executable specified by path, rather than just by name?
+    bool isExecutablePath_;
 #else
 #endif
 };