testing-strategies — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited testing-strategies (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
3-tier testing strategy for Kailash applications. Tier 2/3 require real infrastructure — NO mocking (@patch, MagicMock, unittest.mock are BLOCKED) per rules/testing.md.
Use when asking about testing, test strategy, 3-tier testing, unit tests, integration tests, end-to-end tests, testing workflows, testing DataFlow, testing Nexus, real infrastructure, NO mocking, test organization, or testing best practices.
rules/probe-driven-verification.md.| Tier | Scope | Mocking | Speed | Infrastructure |
|---|---|---|---|---|
| 1 - Unit | Functions, classes | Allowed | <1s/test | None |
| 2 - Integration | Workflows, DB, APIs | BLOCKED — real infra only | 1-10s/test | Real DB, real runtime |
| 3 - E2E | Complete user flows | BLOCKED — real infra only | 10s+/test | Real HTTP, real everything |
Why: Mocking hides database constraints, API timeouts, race conditions, connection pool exhaustion, schema migration issues, and LLM token limits.
What to use instead: Test databases (Docker containers), test API endpoints, test LLM accounts (with caching), temp directories.
@pytest.fixture
def db():
"""Real database for testing."""
db = DataFlow("postgresql://test:test@localhost:5433/test_db")
db.create_tables()
yield db
db.drop_tables()
@pytest.fixture
def runtime():
return LocalRuntime()tests/
tier1_unit/ # Mocking allowed
tier2_integration/ # Real infrastructure
tier3_e2e/ # Full system
conftest.py # Shared fixtures| Component | Tier | Key Point |
|---|---|---|
| Workflows | 2 | Real runtime execution, verify results["node"]["result"] |
| DataFlow | 2 | Real DB, verify with read-back after write |
| Nexus API | 3 | Real HTTP requests to running server |
| Kaizen Agents | 2 | Real LLM calls with response caching |
Regression tests lock in bug fixes. They MUST exercise the actual code path -- call the function, assert the raise or return value. Source-grep tests are BLOCKED as the sole assertion because they pin the implementation, not the contract: when the fix moves to a shared helper (the right refactor), the grep breaks even though the protection is still in place.
# Behavioral (survives refactors)
@pytest.mark.regression
def test_null_byte_rejected():
parsed = urlparse("mysql://user:%00x@h/db")
with pytest.raises(ValueError, match="null byte"):
decode_userinfo_or_raise(parsed)
# Source-grep (BLOCKED as sole assertion)
def test_null_byte_exists_in_source():
assert "\\x00" in open("src/kailash/db/connection.py").read()See rules/testing.md "MUST: Behavioral Regression Tests Over Source-Grep" for the full rule and rationale.
Unit and integration tests per primitive cannot observe the handoff between primitives; each primitive's tests construct test fixtures with exactly the fields it needs, and the chain between A → B fails only when A's real output is missing a field B actually needs. For every pipeline the docs teach (README Quick Start, tutorial, specs/*.md canonical example), add a regression test that executes the docs-exact code against real infrastructure AND asserts a deterministic fingerprint over the output. Flipped fingerprints block release. See skills/16-validation-patterns/SKILL.md § "End-to-End Pipeline Regression Above Unit/Integration" for the full pattern + kailash-ml 1.0.0 W33b evidence, and rules/testing.md § "End-to-End Pipeline Regression Tests Above Unit + Integration" for the MUST clause.
Tests that exercise optional extras (e.g., [hpo], [redis], [vault]) MUST guard against the dependency being absent. Use pytest.importorskip at module or class scope so the test is _skipped_ (not _failed_) in CI environments that don't install the extra.
# At module level — skips entire file if optuna is missing
optuna = pytest.importorskip("optuna", reason="optuna required for HPO tests")
class TestSuccessiveHalving:
@pytest.mark.asyncio
async def test_pruning(self):
# optuna is guaranteed available here
...Why: Base CI installs core dependencies only. A test that imports an optional extra without a skip guard fails every CI matrix entry, blocking unrelated PRs. pytest.importorskip is the standard mechanism — it imports the module if available and calls pytest.skip if not.
Where to place the guard: Before the first use of the optional module — typically at module scope (before the test class) or inside a fixture. Placing it inside a test function body is too late if the class-level setup already depends on the import.
@patch/MagicMock/unittest.mockpytest tests/tier1_unit/ # Fast CI
pytest tests/tier2_integration/ # With real infra
pytest tests/tier3_e2e/ # Full system
pytest --cov=app --cov-report=html # Coveragetesting-specialist - Testing strategies and patternstdd-implementer - Test-driven developmentdataflow-specialist - DataFlow testing patterns~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.