Tool comparison evaluation

Produce publication-grade architectural comparison documents that evaluate two solutions against weighted criteria derived from organizational requirements. The output follows a 12-section structure with quantified scoring, bidirectional gap analysis, Mermaid architecture diagrams, cost projections, risk matrices, and reversal conditions for every recommendation.

Workflow

Phase 1: Research both solutions

Gather enough context to write authoritative profiles. Research is front-loaded. Weak research produces weak evaluations.

Research checklist:

• [ ] Identify Solution A and Solution B
• [ ] For each solution, collect: type, maturity, vendor/maintainer, mechanism, output format, integrations, generation time
• [ ] Find official documentation, product pages, blog posts, press coverage
• [ ] Identify pricing model and tiers
• [ ] Find academic research relevant to the comparison domain
• [ ] Identify the organization's current usage and existing investments
• [ ] Document limitations of the research (no hands-on testing, beta status, etc.)

Source priority:

1. Official documentation and technical specs
2. Academic research papers (arXiv, conference proceedings)
3. Press coverage and analyst reports
4. Community discussions and case studies

Phase 2: Define evaluation framework

Design criteria and weights before scoring. The framework must reflect the organization's priorities, not generic equal-weight distribution.

Framework checklist:

• [ ] Identify 6-10 evaluation criteria from organizational requirements
• [ ] Assign percentage weights that sum to exactly 100%
• [ ] Write a one-sentence rationale for each weight
• [ ] Define the 1-5 scoring scale with semantic anchors
• [ ] Verify the highest-weighted criteria reflect the organization's primary constraints

See evaluation methodology for scoring frameworks, weight calibration, and sensitivity analysis techniques.

Phase 3: Write the document

Follow the 12-section structure. Each section has a specific purpose. Do not merge, skip, or reorder sections.

Use the comparison template as a starting point. Copy it and fill in the placeholders. See document structure for section-by-section guidance with examples.

#### Metadata and framing (Sections 1-2)

• [ ] Document metadata (status, date, classification, authors)
• [ ] Section 1: Executive summary (scope + finding + recommendation)
• [ ] Section 2: Scope and methodology (in/out table, assumptions, limitations)

#### Solution profiles (Section 3)

• [ ] Section 3: Solution profiles (symmetric fields for both solutions)

#### Scoring and analysis (Sections 4-5)

• [ ] Section 4: Evaluation framework (criteria, weights, matrix, sensitivity check)
• [ ] Section 5: Architectural fit analysis (Mermaid diagrams, integration topology)

See evaluation methodology for scoring frameworks and sensitivity analysis.

#### Cost, risk, and gaps (Sections 6-8)

• [ ] Section 6: Total cost of ownership (assumptions, formulas, multi-year projection)
• [ ] Section 7: Risk assessment (risk matrix covering both solutions)
• [ ] Section 8: Gap analysis (bidirectional: A-has/B-lacks AND B-has/A-lacks)

#### Synthesis and recommendation (Sections 9-10, Appendices)

• [ ] Section 9: Improvement candidates (features from rejected solution worth adopting)
• [ ] Section 10: Recommendation (primary recommendation, reversal conditions, next steps, re-eval date)
• [ ] Appendix A: Sources (grouped by solution + research, with inline citation context)
• [ ] Appendix B: Evaluation criteria weight justification

Phase 4: Validate

Use the Quality Checklist below. Do not skip the arithmetic verification.

See writing guidelines for style, voice, formatting, and diagram conventions.

Document architecture

The 12-section structure follows a deliberate argument arc: situate the reader (1-2), present the candidates (3), evaluate systematically (4), analyze architecture (5), quantify cost (6), assess risk (7), identify gaps (8), extract improvements (9), recommend (10), support with evidence (A-B).

Executive summary structure

The executive summary contains exactly three elements in this order:

1. Evaluation scope - What is being compared and why. Correct false equivalences upfront. If one solution covers two stages of the other's pipeline, say so here. This prevents unfair downstream comparisons.
2. Finding - Neutral factual summary of what the evaluation discovered. No recommendation yet. State the structural relationship between the solutions.
3. Recommendation - Decisive action with quantified evidence. "Retain Solution B. Do not adopt Solution A." followed by the key numbers and structural reasons.

Core principles

Symmetry

Every data point for Solution A must exist for Solution B. Asymmetric profiles produce biased comparisons. If data is unavailable for one solution, note it as unavailable rather than omitting the field entirely.

Quantification

Numbers replace adjectives. "42x cost differential" not "significantly more expensive". "4.45/5.00 vs 2.05/5.00" not "scored much higher". Show calculations so readers can verify the arithmetic independently.

Intellectual honesty

• State limitations of the analysis upfront (Section 2.4)
• Include a sensitivity check that stress-tests the conclusion
• Provide reversal conditions: what circumstances would change this recommendation
• Set a re-evaluation date

Bidirectional gap analysis

Section 8 answers both questions:

• What does A have that B lacks?
• What does B have that A lacks?

Each gap entry needs an "Addressable?" column explaining whether the gap can be closed internally. One-sided analysis is advocacy, not evaluation. The asymmetry between addressable and structural gaps often tells the real story.

Improvement extraction

The rejected solution exists for a reason. Section 9 extracts specific features worth adopting into the recommended solution. Each improvement needs:

• Problem statement (what's missing)
• Proposed approach (how to add it)
• Effort estimate (small/medium/large with one-sentence reasoning)

This transforms a "don't adopt X" conclusion into a "retain Y and strengthen it with ideas from X" action plan.

Diagrams

Use Mermaid for all architecture diagrams. Include at minimum:

1. Integration topology for each solution - how each connects to the broader system. Side-by-side comparison reveals structural differences.
2. Governance/knowledge propagation - how project context, rules, and constraints flow through the system.

Color coding (mandatory):

• Red (fill:#f4cccc,stroke:#cc0000): Problems, risks, manual steps, gaps
• Green (fill:#d9ead3,stroke:#38761d): Advantages, automation, knowledge sources

Label connections when the relationship is not self-evident:

A -- "manual copy/paste" --> B

Use subgraphs to group related elements:

subgraph knowledge["Project Knowledge"]
    direction LR
    F["AGENTS.md or CLAUDE.md"]
    G["Rules"]
end

Keep each diagram focused on one concept. Two clear diagrams communicate more than one crowded diagram.

Anti-patterns

Quality checklist

Before finalizing, verify every item:

Structure

• [ ] All 12 sections present and in correct order
• [ ] Solution profiles are symmetric (same fields for both)
• [ ] Gap analysis is bidirectional (A has / B lacks AND B has / A lacks)

Arithmetic

• [ ] Weights sum to 100%
• [ ] Weighted score calculations are arithmetically correct
• [ ] Cost projections show formulas, not just totals

Analytical rigor

• [ ] Sensitivity check included (remove highest-weight criterion, verify winner holds)
• [ ] Every score has inline evidence (no naked numbers)
• [ ] Risk matrix covers both solutions
• [ ] Every recommendation has reversal conditions
• [ ] Every recommendation has a re-evaluation date

Visual and source integrity

• [ ] Mermaid diagrams render correctly
• [ ] Sources cited inline where used and collected in Appendix A

Writing standards

• [ ] No AI-marker words (see writing guidelines)
• [ ] No hedging phrases
• [ ] Third person throughout
• [ ] Active voice throughout
• [ ] Sentence case headings

Language

Write all evaluation documents in English. Communicate with the user in their language during the research and drafting process.