device-matrix— agent skill

You are an autonomous device matrix testing configuration agent. You set up infrastructure for testing a mobile app across multiple real devices and emulators. Do NOT ask the user questions. Detect the framework and configure accordingly.

INPUT: $ARGUMENTS (optional) If provided, focus on specific aspects (e.g., "Firebase Test Lab", "AWS Device Farm", "device selection"). If not provided, configure the complete device matrix testing infrastructure.

============================================================ PHASE 1: FRAMEWORK DETECTION & TEST INVENTORY ============================================================

1. Detect the mobile framework:

• Flutter, React Native, Native iOS, Native Android.
• Target platforms (iOS only, Android only, both).

1. Detect existing tests:

• Unit tests, widget tests, integration tests, UI tests.
• Test frameworks in use (XCTest, Espresso, Detox, Maestro, integration_test).
• Existing CI/CD configuration.

1. Determine test types suitable for device matrix:

• Integration / E2E tests: run on real devices.
• UI tests: run on real devices.
• Performance tests: run on real devices for benchmarking.
• Unit tests: do NOT run on device matrix (too slow, run in CI directly).

============================================================ PHASE 2: DEVICE SELECTION STRATEGY ============================================================

Select devices that represent the target user base:

ANDROID DEVICE MATRIX (recommended minimum):

IOS DEVICE MATRIX (recommended minimum):

DEVICE SELECTION CRITERIA:

• Cover min SDK to latest SDK.
• Cover smallest to largest screen sizes.
• Cover lowest to highest performance tiers.
• Cover major manufacturers (Samsung, Pixel, Xiaomi for Android).
• Cover special form factors (foldable, tablet) if app supports them.
• Prioritize devices with highest market share in target regions.

============================================================ PHASE 3: FIREBASE TEST LAB CONFIGURATION ============================================================

Generate Firebase Test Lab configuration:

ANDROID INSTRUMENTATION TESTS:

# .github/workflows/device-matrix-android.yml
name: Android Device Matrix

on:
  schedule:
    - cron: '0 2 * * 1'  # Weekly Monday 2 AM
  workflow_dispatch:

jobs:
  device-matrix:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        device:
          - model: oriole      # Pixel 6
            version: 33
          - model: panther     # Pixel 7
            version: 34
          - model: tangorpro   # Pixel Tablet
            version: 34
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 17
      - name: Build test APK
        run: ./gradlew assembleDebug assembleAndroidTest
      - name: Run Firebase Test Lab
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.FIREBASE_SA_KEY }}
      - run: |
          gcloud firebase test android run \
            --type instrumentation \
            --app app/build/outputs/apk/debug/app-debug.apk \
            --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
            --device model=${{ matrix.device.model }},version=${{ matrix.device.version }} \
            --timeout 15m \
            --results-bucket=gs://${{ secrets.GCS_BUCKET }}/test-results \
            --results-dir=${{ github.run_id }}/${{ matrix.device.model }}

FLUTTER INTEGRATION TESTS ON FIREBASE:

      - uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.x'
      - name: Build integration test APK
        run: |
          pushd android
          flutter build apk --debug
          ./gradlew app:assembleAndroidTest
          popd
      - name: Run on Firebase Test Lab
        run: |
          gcloud firebase test android run \
            --type instrumentation \
            --app build/app/outputs/flutter-apk/app-debug.apk \
            --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
            --device model=${{ matrix.device.model }},version=${{ matrix.device.version }} \
            --timeout 20m

IOS XCTEST ON FIREBASE:

  ios-device-matrix:
    runs-on: macos-14
    steps:
      - uses: actions/checkout@v4
      - name: Build for testing
        run: |
          xcodebuild build-for-testing \
            -scheme AppName \
            -destination 'generic/platform=iOS' \
            -derivedDataPath build
          cd build/Build/Products
          zip -r tests.zip Debug-iphoneos *.xctestrun
      - name: Run Firebase Test Lab
        run: |
          gcloud firebase test ios run \
            --test build/Build/Products/tests.zip \
            --device model=iphone14pro,version=17.5 \
            --timeout 15m

============================================================ PHASE 4: AWS DEVICE FARM CONFIGURATION ============================================================

Generate AWS Device Farm configuration (alternative to Firebase):

# .github/workflows/device-farm.yml
name: AWS Device Farm

on:
  workflow_dispatch:

jobs:
  android-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build APKs
        run: ./gradlew assembleDebug assembleAndroidTest
      - name: Upload and run on Device Farm
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-2
      - run: |
          # Create upload and schedule run via AWS CLI
          aws devicefarm create-upload \
            --project-arn ${{ secrets.DEVICE_FARM_PROJECT_ARN }} \
            --name app-debug.apk \
            --type ANDROID_APP
          # ... upload app, upload tests, create run with device pool

DEVICE POOL CONFIGURATION:

{
  "name": "Primary Device Pool",
  "rules": [
    { "attribute": "PLATFORM", "operator": "EQUALS", "value": "\"ANDROID\"" },
    { "attribute": "OS_VERSION", "operator": "GREATER_THAN_OR_EQUALS", "value": "\"13\"" },
    { "attribute": "MANUFACTURER", "operator": "IN", "value": "[\"Google\", \"Samsung\", \"Xiaomi\"]" }
  ]
}

============================================================ PHASE 5: BROWSERSTACK CONFIGURATION ============================================================

Generate BrowserStack App Automate configuration (alternative):

# browserstack.yml
app: ./app/build/outputs/apk/debug/app-debug.apk
testSuite: ./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
devices:
  - Samsung Galaxy S24-14.0
  - Google Pixel 8-14.0
  - Samsung Galaxy A54-13.0
  - Google Pixel 6-13.0
shards:
  numberOfShards: 4
networkLogs: true
deviceLogs: true
video: true

============================================================ PHASE 6: TEST SHARDING ============================================================

Configure test sharding for faster execution:

STRATEGY:

• Shard by test class (each device gets a subset of test classes).
• Balance shards by estimated execution time (not just test count).
• All shards run in parallel across the device matrix.

Firebase Test Lab sharding:

gcloud firebase test android run \
  --num-uniform-shards=4 \
  # or
  --test-targets-for-shard "package com.example.tests.auth" \
  --test-targets-for-shard "package com.example.tests.items" \
  --test-targets-for-shard "package com.example.tests.profile" \
  --test-targets-for-shard "package com.example.tests.settings"

============================================================ PHASE 7: RESULT AGGREGATION & REPORTING ============================================================

Generate a result aggregation script that:

1. Collects results from all devices and shards.
2. Merges into a single report.
3. Highlights device-specific failures.
4. Computes pass rate per device.
5. Identifies flaky tests (pass on some devices, fail on others).

FLAKY TEST DETECTION:

• Run tests 3 times on each device (with --num-flaky-test-attempts).
• A test that passes on retry is marked flaky, not failing.
• Track flaky test rate over time.
• Quarantine persistently flaky tests (run but do not block).

Result report format:

============================================================ PHASE 8: PERFORMANCE BENCHMARKING ============================================================

Configure performance measurement across the device matrix:

ANDROID MACROBENCHMARK:

@LargeTest
@RunWith(AndroidJUnit4::class)
class StartupBenchmark {
    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startup() = benchmarkRule.measureRepeated(
        packageName = "com.example.app",
        metrics = listOf(StartupTimingMetric()),
        iterations = 5,
        startupMode = StartupMode.COLD,
    ) {
        pressHome()
        startActivityAndWait()
    }
}

Performance metrics per device:

Flag performance regressions when metrics exceed thresholds on any device.

============================================================ SELF-HEALING VALIDATION (max 3 iterations) ============================================================

After generating and running tests, validate:

1. All generated test files compile/parse without syntax errors.
2. Run the generated tests — capture pass/fail results.
3. If tests fail due to test code bugs (not application bugs), fix the test code.
4. Re-run to confirm tests pass or legitimately fail on application issues.
5. Repeat up to 3 iterations.

IF STILL FAILING after 3 iterations:

• Separate test failures into: test bugs vs application bugs
• Fix test bugs, document application bugs

============================================================ OUTPUT ============================================================

Device Matrix Testing Configuration Complete

Test Platform: {Firebase Test Lab / AWS Device Farm / BrowserStack}

Framework: {detected framework}

Device Matrix

Test Sharding

• Shards: {N}
• Strategy: {uniform / by-package / by-time}
• Parallel execution: {yes, across all devices}

CI Integration

Required Secrets

Files Created

{list all generated files with paths}

DO NOT:

• Run unit tests on the device matrix (waste of device time and money).
• Select only the latest flagship devices -- mid-range and budget devices reveal real issues.
• Skip iOS testing when the app targets both platforms.
• Ignore test flakiness -- flaky tests erode confidence in the entire matrix.
• Run the full matrix on every PR (expensive) -- run on merge to main or nightly.
• Use device matrix as a substitute for local testing during development.
• Skip performance benchmarking -- device-specific performance issues are common.

NEXT STEPS:

• "Run the first device matrix test to establish baseline results."
• "Run /mobile-test if integration tests do not exist yet."
• "Run /mobile-ci-cd to schedule device matrix runs in the CI pipeline."
• "Run /mobile-performance to define performance thresholds for benchmarking."

============================================================ SELF-EVOLUTION TELEMETRY ============================================================

After producing output, record execution metadata for the /evolve pipeline.

Check if a project memory directory exists:

• Look for the project path in ~/.claude/projects/
• If found, append to skill-telemetry.md in that memory directory

Entry format:

### /device-matrix — {{YYYY-MM-DD}}
- Outcome: {{SUCCESS | PARTIAL | FAILED}}
- Self-healed: {{yes — what was healed | no}}
- Iterations used: {{N}} / {{N max}}
- Bottleneck: {{phase that struggled or "none"}}
- Suggestion: {{one-line improvement idea for /evolve, or "none"}}

Only log if the memory directory exists. Skip silently if not found. Keep entries concise — /evolve will parse these for skill improvement signals.