SY-3690: Refactor Integration Test Infrastructure (#2099)

Large-scale refactor of the integration test framework and CI pipeline.

The monolithic `test_conductor.py` (~1400 lines) has been decomposed into focused,
single-responsibility modules:

- **config_client.py** — Discovers, loads, filters, and expands test configurations
  from `*_tests.json` files, including parameter matrix expansion and test class
  discovery
- **execution_client.py** — Manages test lifecycle (setup, run, teardown) with
  sequencing modes (sequential, random, asynchronous) and thread pool management
- **telemetry_client.py** — Streams real-time test metrics (pass/fail counts, timing)
  to Synnax channels
- **report_client.py** — Generates JSON and human-readable test result reports
- **log_client.py** — Structured logging with optional Synnax channel output
- **models.py** — Shared data models (TestResult, TestStatus, etc.)
- **target_filter.py** — Flexible test filtering with substring matching at file,
  sequence, and case levels. Supports comma-separated file targets (e.g.,
  `driver,control,latency`)

The `TestCase` base class has been simplified — Synnax client connection management
moved into the framework, and dead code removed. All ~50 test case files updated
with minor import changes.

Integration tests now run concurrently across multiple runners instead of sequentially
on a single runner per OS:

- **New reusable workflow** (`test.integration.worker.yaml`) — Handles the full test
  lifecycle for a single matrix entry: artifact download, process cleanup, core startup,
  test execution, and artifact upload
- **Matrix strategy** — Tests split into 3 groups (arc, console, driver) running in
  parallel on both Windows and Ubuntu (6 concurrent runners)
- **Single-source-of-truth matrix** (`validate_test_coverage.sh`) — Defines the test
  groups, validates all `*_tests.json` files are covered, and outputs the matrix JSON
  for GitHub Actions
- **Custom flags support** — When the `FLAGS` workflow input is provided, the matrix
  collapses to a single worker per OS, passing flags directly to test-conductor
  (e.g., `uv run tc console/lifecycle` or `uv run tc -f channel_life`)

- Added retry mechanism for flaky tests
- Integrated structured logging to Synnax channels
- Improved failure output artifacts

Author: sy-nico

.github/workflows/test.integration.worker.yaml

@@ -0,0 +1,137 @@
+name: Integration Test Run
+
+on:
+  workflow_call:
+    inputs:
+      test_matrix:
+        description: "JSON test matrix (e.g., from validate_test_coverage.sh)"
+        type: string
+        required: true
+      ref_run_id:
+        description: "Reference run ID for downloading build artifacts"
+        type: string
+        required: true
+      os:
+        description: "Runner OS label (e.g., 'windows-latest', 'ubuntu-latest')"
+        type: string
+        required: true
+      flags:
+        description: "Additional flags to pass to test-conductor"
+        type: string
+        required: false
+        default: ""
+
+jobs:
+  run:
+    name: ${{ matrix.name }}
+    timeout-minutes: 30
+    runs-on:
+      - self-hosted
+      - ${{ inputs.os }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJSON(inputs.test_matrix) }}
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v6
+        with:
+          clean: false
+
+      - name: Force Quit Existing Synnax Processes (Windows)
+        if: runner.os == 'Windows'
+        shell: cmd
+        run: integration/scripts/KillSynnaxProcessesWindows.cmd
+
+      - name: Force Quit Existing Synnax Processes (Unix)
+        if: runner.os == 'Linux'
+        continue-on-error: true
+        run: integration/scripts/kill_synnax_processes_unix.sh
+
+      - name: Download and Setup Artifacts (Windows)
+        if: runner.os == 'Windows'
+        shell: cmd
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REF_RUN_ID: ${{ inputs.ref_run_id }}
+        run: integration/scripts/DownloadArtifactsWindows.cmd
+
+      - name: Download and Setup Artifacts (Unix)
+        if: runner.os == 'Linux'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REF_RUN_ID: ${{ inputs.ref_run_id }}
+        run: integration/scripts/download_artifacts_unix.sh
+
+      - name: Update Submodules
+        run: git submodule update --init --recursive
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version-file: integration/pyproject.toml
+
+      - name: Install Playwright (Windows)
+        if: runner.os == 'Windows'
+        shell: cmd
+        working-directory: integration
+        run: uv run playwright install --with-deps
+
+      - name: Install Playwright (Unix)
+        if: runner.os == 'Linux'
+        working-directory: integration
+        run: uv run playwright install --with-deps
+
+      - name: Start Core (Windows)
+        if: runner.os == 'Windows'
+        timeout-minutes: 1
+        shell: powershell
+        env:
+          SYNNAX_LICENSE_KEY: ${{ secrets.SYNNAX_LICENSE_KEY }}
+        run: integration/scripts/StartCoreWindows.ps1
+
+      - name: Start Core (Unix)
+        if: runner.os == 'Linux'
+        timeout-minutes: 1
+        env:
+          SYNNAX_LICENSE_KEY: ${{ secrets.SYNNAX_LICENSE_KEY }}
+        run: integration/scripts/start_core_unix.sh
+
+      - name: Test Conductor (Windows)
+        if: runner.os == 'Windows'
+        id: test-conductor-win
+        timeout-minutes: 30
+        shell: cmd
+        working-directory: integration
+        env:
+          PYTHONIOENCODING: utf-8
+        run: >-
+          uv run test-conductor --name tc_win ${{ matrix.target || inputs.flags }}
+
+      - name: Test Conductor (Unix)
+        if: runner.os == 'Linux'
+        id: test-conductor-unix
+        timeout-minutes: 30
+        working-directory: integration
+        run: >-
+          uv run test-conductor --name tc_unix ${{ matrix.target || inputs.flags }}
+
+      - name: Upload Test Results
+        uses: actions/upload-artifact@v6
+        if: always()
+        with:
+          name: test-results-${{ inputs.os }}-${{ matrix.name }}
+          path: integration/tests/results/*
+          retention-days: 7
+
+      - name: Upload Core Logs
+        uses: actions/upload-artifact@v6
+        if: ${{ failure() || cancelled() }}
+        with:
+          name:
+            core-logs-${{ inputs.os }}-${{ matrix.name }}-attempt-${{ github.run_attempt
+            }}
+          path: ~/synnax-data/synnax-core.log
+          retention-days: 7

.github/workflows/test.integration.yaml

@@ -12,8 +12,9 @@ on:
     paths:
       - .bazeliskrc
       - .bazelrc
-      - .github/workflows/test.integration.yaml
       - .github/workflows/build.synnax.yaml
+      - .github/workflows/test.integration.yaml
+      - .github/workflows/test.integration.worker.yaml
       - alamos/go/**
       - alamos/py/**
       - alamos/ts/**
@@ -90,6 +91,9 @@ jobs:
         ${{ steps.update-outputs.outputs.REF_RUN_ID ||
         steps.set-outputs.outputs.REF_RUN_ID }}
       FLAGS: ${{ steps.set-outputs.outputs.FLAGS }}
+      TEST_MATRIX:
+        ${{ steps.set-outputs.outputs.CUSTOM_MATRIX ||
+        steps.test-matrix.outputs.TEST_MATRIX }}
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v6
@@ -116,6 +120,10 @@ jobs:
         run: |
           echo "FLAGS=${FLAGS}" >> $GITHUB_OUTPUT
           echo "SKIP_BUILD=${SKIP_BUILD}" >> $GITHUB_OUTPUT
+          if [ -n "${FLAGS}" ]; then
+            SAFE_NAME=$(echo "${FLAGS}" | sed 's/[^a-zA-Z0-9_-]/_/g')
+            echo "CUSTOM_MATRIX={\"include\":[{\"name\":\"${SAFE_NAME}\",\"target\":\"\"}]}" >> $GITHUB_OUTPUT
+          fi
           if [ "${SKIP_BUILD}" = "true" ]; then
             if [ -z "${REF_RUN_ID}" ]; then
               echo "Empty REF_RUN_ID. Setting SKIP_BUILD to false."
@@ -145,6 +153,11 @@ jobs:
             echo "REF_RUN_ID=${{ github.run_id }}" >> $GITHUB_OUTPUT
           fi
 
+      - name: Validate Test Coverage
+        if: ${{ steps.set-outputs.outputs.FLAGS == '' }}
+        id: test-matrix
+        run: integration/scripts/validate_test_coverage.sh
+
   build:
     needs: setup
     if: needs.setup.outputs.SKIP_BUILD == 'false'
@@ -163,146 +176,30 @@ jobs:
       version: ${{ needs.setup.outputs.VERSION }}
     secrets: inherit
 
-  deploy-and-test-windows:
-    name: Deploy and Test (windows)
-    timeout-minutes: 30
-    runs-on:
-      - self-hosted
-      - windows-latest
+  test-windows:
+    name: Test (windows)
     needs: [setup, build]
     if:
       ${{ !cancelled() && needs.setup.result == 'success' && (needs.build.result ==
       'success' || needs.build.result == 'skipped') }}
-    steps:
-      - name: Checkout Repository
-        uses: actions/checkout@v6
-        with:
-          clean: false
-
-      - name: Force Quit Existing Synnax Processes (Windows)
-        shell: cmd
-        run: integration/scripts/KillSynnaxProcessesWindows.cmd
-
-      - name: Download and Setup Windows Artifacts
-        shell: cmd
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REF_RUN_ID: ${{ needs.setup.outputs.REF_RUN_ID }}
-        run: integration/scripts/DownloadArtifactsWindows.cmd
-
-      - name: Update Submodules
-        run: git submodule update --init --recursive
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v7
-
-      - name: Set up Python
-        uses: actions/setup-python@v6
-        with:
-          python-version-file: integration/pyproject.toml
-
-      - name: Install Playwright
-        shell: cmd
-        working-directory: integration
-        run: uv run playwright install --with-deps
-
-      - name: Start Core
-        timeout-minutes: 1
-        shell: powershell
-        env:
-          SYNNAX_LICENSE_KEY: ${{ secrets.SYNNAX_LICENSE_KEY }}
-        run: integration/scripts/StartCoreWindows.ps1
-
-      - name: Test Conductor
-        id: test-conductor
-        timeout-minutes: 30
-        shell: cmd
-        working-directory: integration
-        env:
-          PYTHONIOENCODING: utf-8
-        run: uv run test-conductor --name tc_win ${{ needs.setup.outputs.FLAGS }}
-
-      - name: Upload Test Results
-        uses: actions/upload-artifact@v6
-        if: always()
-        with:
-          name: test-results-windows
-          path: integration/tests/results/*
-          retention-days: 7
-
-      - name: Upload Core Logs
-        uses: actions/upload-artifact@v6
-        if: ${{ failure() || cancelled() }}
-        with:
-          name: core-logs-windows-attempt-${{ github.run_attempt }}
-          path: ~/synnax-data/synnax-core.log
-          retention-days: 7
+    uses: ./.github/workflows/test.integration.worker.yaml
+    with:
+      test_matrix: ${{ needs.setup.outputs.TEST_MATRIX }}
+      ref_run_id: ${{ needs.setup.outputs.REF_RUN_ID }}
+      os: windows-latest
+      flags: ${{ needs.setup.outputs.FLAGS }}
+    secrets: inherit
 
-  deploy-and-test-ubuntu:
-    name: Deploy and Test (ubuntu)
-    timeout-minutes: 30
-    runs-on:
-      - self-hosted
-      - ubuntu-latest
+  test-ubuntu:
+    name: Test (ubuntu)
     needs: [setup, build]
     if:
       ${{ !cancelled() && needs.setup.result == 'success' && (needs.build.result ==
       'success' || needs.build.result == 'skipped') }}
-    steps:
-      - name: Checkout Repository
-        uses: actions/checkout@v6
-        with:
-          clean: false
-
-      - name: Force Quit Existing Synnax Processes
-        continue-on-error: true
-        run: integration/scripts/kill_synnax_processes_unix.sh
-
-      - name: Download and Setup Linux Artifacts
-        env:
-          GH_TOKEN: ${{ github.token }}
-          REF_RUN_ID: ${{ needs.setup.outputs.REF_RUN_ID }}
-        run: integration/scripts/download_artifacts_unix.sh
-
-      - name: Update Submodules
-        run: git submodule update --init --recursive
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v7
-
-      - name: Set up Python
-        uses: actions/setup-python@v6
-        with:
-          python-version-file: integration/pyproject.toml
-
-      - name: Install Playwright
-        working-directory: integration
-        run: uv run playwright install --with-deps
-
-      - name: Start Core
-        timeout-minutes: 1
-        env:
-          SYNNAX_LICENSE_KEY: ${{ secrets.SYNNAX_LICENSE_KEY }}
-        run: integration/scripts/start_core_unix.sh
-
-      - name: Test Conductor
-        id: test-conductor
-        timeout-minutes: 30
-        working-directory: integration
-        run: uv run test-conductor --name tc_unix ${{ needs.setup.outputs.FLAGS }}
-
-      - name: Upload Test Results
-        uses: actions/upload-artifact@v6
-        if: always()
-        with:
-          name: test-results-linux
-          path: integration/tests/results/*
-          retention-days: 7
-
-      - name: Upload Core Logs
-        uses: actions/upload-artifact@v6
-        if: ${{ failure() || cancelled() }}
-        with:
-          name: core-logs-linux-attempt-${{ github.run_attempt }}
-          path: ~/synnax-data/synnax-core.log
-          retention-days: 7
+    uses: ./.github/workflows/test.integration.worker.yaml
+    with:
+      test_matrix: ${{ needs.setup.outputs.TEST_MATRIX }}
+      ref_run_id: ${{ needs.setup.outputs.REF_RUN_ID }}
+      os: ubuntu-latest
+      flags: ${{ needs.setup.outputs.FLAGS }}
+    secrets: inherit