ddia-systems — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited ddia-systems (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.
Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.
Goal: 10/10. Rate any data architecture 0-10 against the principles below: deliberate trade-off choices for data models, storage, replication, partitioning, transactions, and pipelines score high; accidental complexity and ignored failure modes score low. Report the current score and the improvements needed to reach 10/10.
Seven domains for reasoning about data-intensive systems:
Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.
Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| User profiles with nested data | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |
| Social network connections | Graph model for relationship traversal | Neo4j Cypher: MATCH (a)-[:FOLLOWS*2]->(b) for friend-of-friend |
| Financial ledger with joins | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |
See: references/data-models.md for relational/document/graph trade-offs and query language comparisons.
Core concept: Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.
Why it works: Understanding your database's storage engine lets you predict performance characteristics, choose appropriate indexes, and avoid pathological workloads.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| High write throughput | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes/sec |
| Mixed read/write OLTP | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |
| Analytical queries | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |
See: references/storage-engines.md for LSM vs B-tree internals, compaction, and column storage.
Core concept: Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.
Why it works: Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| Read-heavy web app | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer |
| Multi-region writes | Multi-leader replication | CockroachDB or Spanner with bounded staleness |
| Shopping cart availability | Leaderless with merge | DynamoDB with last-writer-wins or application-level cart merge |
See: references/replication.md for lag anomalies, conflict resolution, and CRDTs.
Core concept: Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.
Why it works: Without partitioning, one node bottlenecks storage and throughput. Effective partitioning spreads load evenly and avoids hotspots.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| Time-series data | Key-range partitioning by time + source | Partition by (sensor_id, date) to avoid current-day write hotspot |
| User data at scale | Hash partitioning on user ID | Cassandra consistent hashing on user_id for even distribution |
| Celebrity/hot-key problem | Key splitting with random suffix | Append random digit to hot key, fan out reads across 10 sub-partitions |
See: references/partitioning.md for rebalancing, request routing, and secondary index strategies.
Core concept: Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.
Why it works: Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| Account balance transfer | Serializable transaction | BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT; |
| Inventory reservation | SELECT FOR UPDATE to prevent write skew | SELECT stock FROM items WHERE id = X FOR UPDATE before decrementing |
| Cross-service operations | Saga instead of distributed transaction | Charge card, reserve inventory; on failure, run compensating refund |
See: references/transactions.md for isolation-level anomalies and serializability techniques.
Core concept: Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.
Why it works: Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| Daily analytics pipeline | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse |
| Real-time fraud detection | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows |
| Syncing search index | Change data capture | Debezium captures PostgreSQL WAL, Kafka feeds Elasticsearch |
| Audit trail / event replay | Event sourcing | Store OrderPlaced, OrderShipped events; rebuild state by replaying |
See: references/batch-stream.md for dataflow engines, CDC, and exactly-once semantics.
Core concept: Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.
Why it works: Hardware fails, software has bugs, humans make mistakes. Systems that assume perfect operation are brittle; systems that expect faults are resilient.
Key insights:
Code applications:
| Context | Pattern | Example |
|---|---|---|
| Service communication | Timeouts + retries with backoff | retry(max=3, backoff=exponential(base=1s, max=30s)) with jitter |
| Leader election | Consensus algorithm (Raft/Paxos) | etcd or ZooKeeper for distributed locks and leader election |
| Graceful degradation | Circuit breaker | Resilience4j: open circuit after 50% failures in 10-second window |
See: references/fault-tolerance.md for consensus, timeout tuning, and safety/liveness guarantees.
| Mistake | Why It Fails | Fix |
|---|---|---|
| Choosing a database by popularity | Engines have fundamentally different trade-offs | Match storage engine to actual read/write patterns |
| Ignoring replication lag | Stale reads, phantom reads, lost updates | Implement read-your-writes and monotonic-read guarantees |
| Distributed transactions everywhere | 2PC is slow, fragile; coordinator is a SPOF | Design single-partition operations; use sagas across services |
| Hash partitioning everything | Destroys range query ability | Key-range partitioning for time-series; composite keys for locality |
| Assuming serializable isolation | Defaults are weaker; write skew appears in production | Check the actual default; use explicit locking where needed |
| Conflating batch and stream | Wrong tool adds latency or wasted complexity | Match processing model to data boundedness and latency needs |
| Treating all faults as recoverable | Corruption and Byzantine faults need different handling | Classify faults; design a recovery strategy per class |
| Question | If No | Action |
|---|---|---|
| Can you explain why you chose this database over alternatives? | Choice was familiarity, not requirements | Evaluate data model fit, read/write ratio, consistency needs, scaling path |
| Do you know your database's default isolation level? | Latent concurrency bugs | Check docs; test for write skew and phantom reads |
| Is your replication strategy explicitly chosen? | Implicit consistency/durability assumptions | Document sync vs async, failover behavior, lag tolerance |
| Can your system handle a hot partition key? | One popular entity can down the cluster | Add key-splitting or load shedding for hot keys |
| Do you separate system of record from derived data? | Every change requires migrating everything | Introduce CDC or event sourcing to decouple |
| Are timeouts and retries tuned, not defaulted? | Cascading failures or needless delays | Measure p99; set timeouts above p99, below cascade threshold |
| Have you tested failover in production conditions? | Recovery plan is theoretical | Run chaos experiments: kill leaders, partition networks, fill disks |
For the complete treatment with detailed diagrams and research references:
Martin Kleppmann is a distributed-systems researcher at the University of Cambridge and a former engineer at LinkedIn and Rapportive, known for his work on CRDTs and local-first software. His book Designing Data-Intensive Applications (2017) is the definitive reference for engineers building data systems, praised for making distributed-systems concepts accessible and practical.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.