alive-analysis Skill

Data analysis workflow kit based on the ALIVE loop. Provides structured analysis methodology for data analysts and non-analyst roles.

Overview

alive-analysis helps structure data analysis work using the ALIVE loop: Ask → Look → Investigate → Voice → Evolve

It serves two personas:

• Data analysts: Deep, systematic analysis with full ALIVE flow
• Non-analyst roles (PM, engineers, marketers): Quick analysis with guided framework

ALIVE Loop Reference

Stage 1: ASK (❓)

Core question: What do we want to know — and WHY?

Purpose:

• Define the problem clearly and confirm the requester's REAL goal (not just what they said)
• Frame the question: Is this about causation ("Why did X happen?") or correlation ("Are X and Y related?")?
• Set success criteria and scope boundaries
• Build a hypothesis tree before touching any data
• Set up multi-lens perspective: macro (market/industry) → meso (company/product) → micro (user/session)

#### Hypothesis Tree Before diving into data, structure thinking:

Main question: "Why did D30 retention drop?"
├── Internal factors
│   ├── Product changes (releases, feature removals)
│   ├── Channel mix changes (acquisition source shift)
│   ├── Cross-service impact (did another service change affect this?)
│   └── Pricing / promotion changes
├── External factors
│   ├── Seasonality / holidays
│   ├── Competitor actions
│   ├── Market / economic shifts
│   └── Platform changes (iOS/Android policy, algorithm updates)
└── Data artifacts
    ├── Tracking changes (instrumentation broke?)
    ├── Definition changes (metric recalculated?)
    └── Population changes (new user mix shifted?)

#### Causal vs Correlational Framing Ask explicitly:

• "Are we trying to prove X caused Y? Or just that they move together?"
• "If we find a correlation, what would we need to prove causation?"
• This determines the methodology: correlation → observational analysis; causation → quasi-experimental or controlled experiment

#### Structured Data Request (5 Elements) When the analysis requires data, structure every data request with these 5 elements. This saves massive back-and-forth time — most data request delays come from unclear questions, not technical difficulty.

[Period]  +  [Subject]  +  [Condition]  +  [Metric]  +  [Output Format]
 WHEN         WHO/WHAT      FILTER         MEASURE       HOW TO SHOW

For non-analysts: The AI should actively help structure vague questions into these 5 elements. For analysts: Use this as a self-check before writing SQL or requesting data from others.

This framework applies at every data touchpoint:

• ASK: When defining what data is needed to answer the question
• LOOK: When formulating MCP queries or data requests to colleagues
• INVESTIGATE: When running ad-hoc queries to test hypotheses

#### Key Questions to Ask the User

• What triggered this question? (event, dashboard alert, stakeholder request)
• What decision will this analysis inform?
• What's the cost of being wrong?
• Are there related analyses or prior findings to build on?
• What data can you access? (MCP, exported files, BI dashboards)

#### Analyst Mindset Good analysis comes from asking better questions, not just finding right answers.

• Data + intuition are complementary, not competing: Data without domain context is noise. Intuition without data is guessing. The best analysis combines both.
• Three maturity levels of data use:
• Level 1 "Data-driven": Let the numbers decide (risky — numbers without context mislead)
• Level 2 "Data-informed": Use data as ONE input alongside domain expertise and judgment
• Level 3 "Data-inspired": Use data to discover questions you didn't know to ask
• "Why?" is more valuable than "What?": Dashboards tell you WHAT happened. Analysis tells you WHY and WHAT TO DO about it.
• Resist the urge to answer immediately: A well-framed question saves more time than a fast answer to the wrong question.

#### Actionable Question Design Before starting any analysis, apply the "execution test" — if the analysis result won't change what we do, the question needs redesigning.

Transform abstract questions into actionable ones:

• BAD: "Why is our conversion rate low?" (too broad, no clear action path)
• GOOD: "Among our 4 ad channels, which targeting conditions should we adjust to improve first-purchase conversion?" (specific lever, measurable outcome, within team's control)

Three checks before starting:

1. Control check: Can our team directly change the variable in question? If not, reframe to what we CAN control.
2. Measurement check: Can we clearly measure success? Define the metric and threshold upfront.
3. Resource check: If we find the answer, do we have the resources to act on it within a reasonable timeframe?

The "data-driven paradox" trap: More data and better tools don't automatically lead to better decisions. The gap is usually between insight and execution — always design the analysis with the end action in mind.

#### Metric Framework — Know Which Metrics Matter and Why Every analysis touches metrics. Understanding the metric hierarchy prevents optimizing the wrong thing.

4-Tier Metric Classification:

🌟 North Star Metric (NSM)
    The ONE metric capturing core product value for customers.
    ├── NOT revenue itself (lagging) — but a leading indicator of revenue
    ├── NOT directly manipulable — "if you can move it directly, it's not a good NSM"
    ├── GOOD: "Weekly active buyers", "Monthly items received on time"
    └── BAD: "DAU" (vanity), "Page views" (no value signal), "MRR" (lagging)

📊 Leading / Input Metrics
    3-5 drivers that teams can directly influence to move the NSM.
    ├── More responsive and under immediate team control
    ├── Example: NSM = "Weekly active buyers"
    │   → Inputs: signup rate, first-purchase conversion, repeat purchase rate
    └── Each team owns 1-2 input metrics (their OMTM — One Metric That Matters)

🛡️ Guardrail Metrics (Counter-Metrics)
    Metrics that must NOT get worse while optimizing the NSM.
    ├── "For every metric, have a paired metric that addresses adverse consequences"
    ├── 2-3 max — too many creates false positive noise
    ├── Example: Optimizing Stories engagement? → Guardrail: main feed engagement
    └── If a guardrail triggers during an experiment → escalate before proceeding

🔬 Diagnostic Metrics
    Metrics for root cause analysis — not optimization targets.
    ├── Used in INVESTIGATE to decompose problems
    ├── Example: "Why did conversion drop?" → check: page load time, error rate, funnel drop-off by step
    └── These explain the WHY behind NSM/Input movements

When the AI encounters metrics during analysis:

• ASK: "Which tier does this metric belong to? Are we optimizing an Input metric, or diagnosing a problem?"
• LOOK: "Is there a counter-metric we should also check?"
• INVESTIGATE: "Are we accidentally improving one metric at the cost of a guardrail?"
• VOICE: "Frame recommendations in terms of the NSM hierarchy — what moves, what's protected"
• EVOLVE: "Should the metric framework be updated based on these findings?"

Common mistakes to prevent:

• Starting analysis without a clear question
• Scope creep — trying to answer everything at once
• Not confirming the requester's actual goal (vs stated goal)
• Confusing "interesting" with "actionable"
• Ignoring the hypothesis tree — jumping to the first plausible explanation
• Optimizing a metric without checking its counter-metric

Stage 2: LOOK (👀)

Core question: What does the data ACTUALLY show — and what's missing?

Purpose:

• Review data quality (missing values, outliers, date ranges)
• Segment before averaging — never trust aggregate numbers alone
• Identify confounding variables that could mislead the analysis
• Check for external factors (seasonality, holidays, competitor actions)
• Map cross-service dependencies — changes in service A can affect metrics in service B
• Validate data access methods and establish a query/file pipeline

#### Segmentation Strategy Always break the data down BEFORE forming conclusions:

• By time: daily/weekly trends, before/after specific events
• By cohort: new vs returning users, acquisition channel, geography
• By platform: iOS vs Android, web vs app
• By segment: free vs paid, high-value vs low-value

Ask: "Does the pattern hold across ALL segments, or is it driven by one?"

#### Confounding Variables Before attributing any pattern, check:

• Did something else change at the same time? (release, campaign, holiday)
• Is there a selection bias? (are we comparing different populations?)
• Is there a survivorship bias? (are we only seeing users who stayed?)
• Could this be a Simpson's paradox? (aggregate trend vs segment trends)

#### External Data Consideration

• Check the calendar: holidays, industry events, competitor launches
• Check macro context: economic indicators, platform policy changes
• Check company context: other team's releases, cross-service changes
• Reference config.md guardrail metrics — are any of them moving too?

#### Data Specification Gotchas Before trusting ANY data, verify these common traps:

Geographic data:

• Verify coordinate system: WGS84 (lat/lng) vs projected (UTM-K, etc.) — mixing them silently corrupts distance/area calculations
• Administrative district mismatch: 행정동 ≠ 법정동 — same name, different boundaries, M:N relationship between the two
• Population data is usually 행정동-based, but business addresses are often 법정동 — combining them without matching creates wrong numbers
• When in doubt, use higher-level geography (시군구) or raw coordinates with spatial joins

Metric definitions:

• Same metric name, different calculation across teams (e.g., "GMV" = gross? net? after coupon?)
• Check if definitions changed over time (metric redefined mid-period → trend break that looks like a real change)
• Verify with data owners: "Is this the same definition used in the dashboard?"

Time and date:

• Timezone: UTC vs local time — a 9-hour shift (KST) can move an entire day of data
• Business day vs calendar day (weekends, holidays treated differently)
• Event timestamps: server time vs client time vs display time

Units and filters:

• Currency: before/after tax, with/without discounts, local vs USD
• Hidden filters: "active users" might exclude recently churned, test accounts, internal accounts
• Population drift: the definition of "user" may change as tracking evolves

When you find a gotcha: Document it in the analysis as a Data Quality Note — future analyses will thank you.

#### Metric Interpretation Guide Key concepts: CV (variability check), deviation vs error, STEDII metric validation, risk-adjusted metrics, trend momentum, cohort analysis pitfalls.

For details, see core/references/analytical-methods.md § Metric Interpretation Guide

#### Data Access During Conversation

• MCP connected: AI can run queries directly — ask before executing
• User provides files: Read CSV/Excel/JSON files provided during conversation
• BI tool screenshots: User shares dashboard images — AI interprets visually
• No direct access: AI generates SQL/Python for user to run, then discusses results

Common mistakes to prevent:

• Looking at averages without segmentation
• Ignoring seasonality and external factors
• Re-verifying already confirmed data
• Skipping data quality checks
• Assuming correlation found in LOOK implies causation
• Mixing data with different specifications without verifying alignment

Stage 3: INVESTIGATE (🔍)

Core question: Why is it REALLY happening — can we prove it?

Purpose:

• Eliminate hypotheses systematically (not just confirm the first one)
• Apply multi-lens analysis: macro → meso → micro
• Test for causation vs correlation rigorously
• Check cross-service impacts and interaction effects
• Perform sensitivity analysis — how robust are the findings?
• Handle data mid-conversation (MCP queries, file uploads, ad-hoc analysis)

#### Hypothesis Elimination Process Work through the hypothesis tree from ASK:

1. List all hypotheses (from the tree)
2. For each: What evidence would support it? What would disprove it?
3. Test the easiest-to-disprove hypotheses first (efficient elimination)
4. Track: ✅ Supported / ❌ Eliminated / ⚠️ Inconclusive
5. For surviving hypotheses: estimate relative contribution (e.g., "Channel mix explains ~70%, product change ~20%, unknown ~10%")

#### Multi-Lens Analysis Framework

Macro (market/industry level)
├── Market trends — is this industry-wide?
├── Competitor analysis — did competitors change?
├── Economic factors — recession, regulation, etc.
└── Platform/ecosystem — iOS policy, algorithm changes?

Meso (company/product level)
├── Cross-service impact — did another team's change affect us?
├── Channel mix — acquisition source distribution shift?
├── Product changes — releases, A/B tests, feature flags?
└── Business operations — pricing, campaigns, partnerships?

Micro (user/session level)
├── User behavior patterns — funnel analysis, session depth
├── Cohort-specific trends — new vs returning, by segment
├── Edge cases — specific user groups, extreme behaviors
└── Qualitative signals — CS tickets, reviews, survey data

#### Causation Testing (when causal claims are needed)

Step 1: Draw the causal picture (even informally) Before running any numbers, sketch the relationships:

• What's the treatment (T)? What's the outcome (Y)?
• What other variables might affect BOTH T and Y? (these are confounders)
• Are there mediators in between? (T → M → Y)
• Watch out for colliders (T → C ← Y) — conditioning on these creates false associations

Three patterns to recognize:

Chain:    T → M → Y     Conditioning on M blocks the path (breaks the causal flow)
Fork:     T ← X → Y     Conditioning on X blocks confounding (this is what you WANT)
Collider: T → C ← Y     Conditioning on C OPENS a false path (this is a TRAP)

Step 2: Block confounding paths

• Identify all "backdoor paths" — paths from T to Y that go through common causes
• Block them by conditioning on the confounders (compare within similar groups)
• Example: "Comparing companies that hired consultants vs didn't" — must condition on prior revenue (companies with higher past revenue are more likely to both hire consultants AND have higher future revenue)
• The goal: make it as if treatment was randomly assigned within each subgroup

Step 3: Handle unmeasurable confounders Sometimes you can't directly measure the confounder (e.g., "manager quality"):

• Use proxy variables: manager tenure, education level, team turnover rate
• Proxies aren't perfect but reduce bias significantly
• Be explicit about what you CAN'T control for — this goes into limitations

Step 4: Verify comparable groups Before comparing treatment vs control:

• Were the groups comparable BEFORE the treatment? (check pre-period metrics)
• If not comparable → the difference is NOT purely due to treatment (bias exists)
• Within subgroups of similar characteristics, does the treatment effect hold?

Step 5: Basic causal checklist

• Time ordering: Did the cause precede the effect?
• Mechanism: Is there a plausible pathway from cause to effect?
• Dose-response: Does more of the cause produce more of the effect?
• Counterfactual: What happened to the control group / unaffected segment?
• Consistency: Does the pattern hold across different segments and time periods?
• If true experiment isn't possible → see "Quasi-Experimental Methods" section below

Common traps:

• Selection bias: Only surveying people who responded → biased sample (e.g., satisfaction survey only captures motivated respondents)
• Collider bias: Conditioning on an effect of both variables opens a false path (e.g., conditioning on "got promoted" when studying stats skill vs office politics)
• Survivorship bias: Only looking at users who stayed → missing the ones who left

#### Statistical Rigor (when making claims from numbers)

Don't just report numbers — report how reliable they are:

Sample size awareness:

• Small samples → high chance of noise masquerading as signal
• Ask: "If I repeated this analysis with different data, would I get the same result?"
• Rule of thumb: be skeptical of conclusions from fewer than ~30 data points per group

Confidence and significance:

• Confidence interval: "The true value is likely between X and Y" (not just a point estimate)
• Statistical significance: "Could this difference be random noise?"
• For stakeholders — translate into plain language:
• NOT: "p = 0.02, CI [1.2, 3.5]"
• YES: "We're quite confident this is a real effect, not noise. The improvement is likely between 1.2% and 3.5%."

Power and sample size:

• Before running an experiment: calculate how many samples you NEED
• Low statistical power = high chance of missing a real effect
• More samples → narrower confidence interval → more reliable conclusion

When comparing groups:

• Always check: are the groups comparable before the treatment/event?
• If confidence intervals of two groups overlap heavily → the difference may not be real
• Report effect sizes, not just p-values — "statistically significant" ≠ "practically important"

#### Cross-Service Impact Analysis In organizations with multiple products/services:

• Map service dependencies: "If service A changes onboarding, does service B's retention change?"
• Check shared resources: same user base, shared auth, shared data pipeline
• Look for cannibalisation: did a new feature pull users from an existing one?
• Check infrastructure: shared API, CDN, DB performance impacts

#### Analytical Methods Toolkit Method selection quick reference:

For details, see core/references/analytical-methods.md § Analytical Methods Toolkit

#### When A/B Testing Isn't Possible — Quasi-Experimental Methods

For details, see core/references/analytical-methods.md § Quasi-Experimental Methods

#### Model Interpretability When building prediction models, always pair prediction with explanation (feature importance, SHAP). Never deploy a "black box" — stakeholders need to know which levers to pull.

For details, see core/references/analytical-methods.md § Model Interpretability

#### Time Series Pattern Reading When analyzing metrics over time (sales forecasting, trend analysis), focus on patterns and uncertainty, not just point predictions.

What to look for:

• Trend: Is there a long-term upward or downward direction?
• Seasonality: Are there repeating patterns (weekly, monthly, annual cycles)?
• Irregularity: Are there sudden spikes/drops that don't fit the pattern? (external events, data issues)

Uncertainty communication is crucial:

• Never present a single predicted number — always present a range
• "We expect next month's revenue to be ₩500M-₩600M, with ₩550M as the most likely" (not just "₩550M")
• Wider range = more uncertainty = signal to investigate what's driving the instability
• Short-term forecasts are more reliable than long-term — be explicit about this

When uncertainty suddenly increases:

• Ask: "What changed? New competitor? Seasonal shift? Tracking change?"
• Increased uncertainty itself is an insight — "the market is becoming less predictable"
• This is valuable for planning: wider uncertainty → larger safety margins needed

#### Sensitivity Analysis Before finalizing findings, test robustness:

• "If we change the date range by ±1 week, does the conclusion hold?"
• "If we exclude the top/bottom 5% of users, does the pattern persist?"
• "If we use a different metric definition, do we get the same result?"
• "What's the minimum effect size that would be actionable?"

#### Simulation Analysis (when evaluating policies or strategies) When the analysis involves "What would happen if we do X?", use simulation methodology. Simulation is often more effective than ML prediction because the calculation is transparent and stakeholders can follow every step.

When to use simulation:

• Policy/strategy evaluation ("If we lower delivery fees, what happens to revenue?")
• Pricing changes, promotion design, business model adjustments
• Any decision where you need to show the P&L impact of different options

4-Step Simulation Framework:

Step 1: Identify variable accounts

• From all financial/business items, select ONLY what changes with the policy
• Example: delivery fee policy → variable: delivery revenue, promo cost, subscription conversion
• Everything else (headcount, server cost, fixed marketing) stays constant

Step 2: Define variable relationships

• Map how the policy change cascades to other variables
• Use historical data or analogous cases as evidence for assumptions
• Example: "Lowering delivery fee → order volume +X% → non-subscriber conversion +Y%"
• X, Y should be grounded in past promotion data or competitor benchmarks

Step 3: Scenario experiments

• Run multiple scenarios by adjusting input values
• Sensitivity analysis: Which variable has the biggest impact on the result?
• Breakeven analysis: At what point does P&L flip from positive to negative?
• Present as: conservative / neutral / aggressive scenarios
• For 3+ uncertain variables: consider Monte Carlo simulation (assign probability distributions, run 1000+ iterations, present confidence intervals)

Step 4: Continuous refinement

• Simulation is NOT "build once, done forever" — it's a living model
• Compare simulation predictions vs actual results after policy execution
• Discover new variables you missed (e.g., "average order value also changed")
• Version-manage the simulation: track how assumptions evolved

Monte Carlo Simulation (when 3+ variables are uncertain): When a simulation has multiple uncertain inputs, point estimates become misleading. Use Monte Carlo to capture the full range of outcomes:

• Spreadsheet approach: Use Data Table (Excel/Sheets) for 1-2 variable sensitivity, Goal Seek for breakeven analysis. For full Monte Carlo, use add-ins or manual iteration with RAND().
• Python approach: Use numpy.random to generate distributions for each uncertain variable, run 1,000-10,000 iterations, and present results as confidence intervals.

  Example: revenue_sim = price * np.random.normal(mean_volume, std_volume, 10000)
  → "90% probability revenue falls between ₩X and ₩Y"

• Present results as probability distributions, not single numbers: "There's a 70% chance this policy is profitable, with expected value ₩X."

Handle Bar concept — the bridge from Simulation to decision-making: The analyst's role is not to "give the answer" but to "build the decision tool."

• Create adjustable inputs (sliders, parameters) that stakeholders can tweak themselves
• "What if we set the minimum order at 20,000 instead of 15,000?"
• This empowers the business team and speeds up iteration
• Handle Bars are the natural output of Simulation analysis — every Simulation should produce at least one adjustable decision lever
• In a 🔮 Simulation analysis, the Handle Bar IS the deliverable (not a static report)

When simulation is less reliable:

• Sudden competitor response or regulatory changes
• High market volatility
• Unprecedented policy with no analogous data
• → In these cases, present wider confidence intervals and emphasize worst-case scenarios

Common mistakes to prevent:

• Confirmation bias — only looking for supporting evidence
• Stopping at the first plausible explanation without testing alternatives
• Claiming causation from correlation alone
• Ignoring external and cross-service factors
• Not recording queries/code for reproduction
• Over-complicating analysis when a simple comparison suffices
• Running a simulation without grounding assumptions in data

Stage 4: VOICE (📢)

Core question: So what — and now what?

Purpose:

• Apply the "So What → Now What" framework for every finding
• Assign confidence levels to each conclusion
• Frame recommendations with trade-off analysis
• Tailor messages to different audiences using config.md stakeholder list
• Make limitations a first-class part of the story, not a footnote

#### "So What → Now What" Framework For every finding, answer both questions:

Finding: "D30 retention dropped 8pp, driven by TikTok-acquired users"
├── So What?  → "TikTok users have 3x lower retention than organic.
│                Our channel mix shifted from 20% to 45% TikTok."
└── Now What? → "Option A: Reduce TikTok budget, reallocate to higher-LTV channels
                 Option B: Keep TikTok but improve onboarding for these users
                 Option C: Accept lower D30 if TikTok CAC justifies it on LTV basis"

#### Confidence Levels Tag each finding explicitly:

• 🟢 High confidence: Multiple data sources confirm, robust to sensitivity checks, clear mechanism
• 🟡 Medium confidence: Supported by data but with caveats (small sample, single source, confounders possible)
• 🔴 Low confidence: Suggestive only — needs further analysis, could be noise or artifact

Include reasoning: "High confidence because the pattern holds across 3 months, 2 platforms, and survives exclusion of outliers."

#### Trade-off Analysis For each recommendation, make the trade-offs explicit:

• What do we gain if we act on this?
• What do we risk or lose?
• What's the cost of inaction?
• Reference guardrail metrics from config.md: "This recommendation would improve conversion but check impact on refund rate."

Counter-metric check (mandatory for every recommendation): For every success metric, identify a counter-metric that reveals if we're just plugging one hole with another. Reference config.md guardrails if defined; if none exists, propose one. See Counter-Metric Monitoring in the Monitoring Guide for the full framework and examples.

#### Presenting Simulation Results When the analysis involved simulation (policy/strategy evaluation):

• Present 3 scenarios: conservative / neutral / aggressive — never just one number
• Show the breakeven point: "This policy becomes unprofitable if conversion drops below X%"
• Highlight the most sensitive variable: "The result is most affected by subscription conversion rate"
• If Monte Carlo was used: "90% probability the result falls between X and Y"
• For stakeholders, avoid jargon ("confidence interval") → use plain language ("best case / expected / worst case")
• Offer handle bars: "If you want to test different assumptions, here are the adjustable inputs"

#### Communicating Numbers Effectively When presenting quantitative findings:

• Always provide context for numbers: "Conversion dropped 2pp" → "Conversion dropped from 12% to 10% (2pp), which means ~500 fewer purchases per week at current traffic"
• Use relative AND absolute: "30% increase" sounds big but might be 3 → 4 users. Always show both.
• Report variability, not just averages: "Average order value is ₩50,000" → "Average ₩50,000, but ranges from ₩10,000 to ₩200,000 (CV=0.8). The 'average customer' doesn't exist."
• Distinguish signal from noise: If the confidence interval includes zero or the opposite direction, say so: "The effect could be anywhere from -2% to +8%, so we can't be sure it's positive."
• Use benchmarks: "Is 12% conversion good?" → Compare to industry, historical, or team-set targets from config.md

#### Audience-Specific Communication

• Executives / C-level: Lead with business impact in one sentence. Numbers, not methodology. "So what does this mean for revenue?"
• Product / Engineering: Include technical detail. "Which feature, which release, which segment." Actionable next steps.
• Cross-functional (marketing, ops): Connect to their KPIs. "Here's how this affects your campaign ROI."
• Reference stakeholders from config.md to auto-suggest audience sections.

#### From Insight to Execution — The Translation Step Analysis without an execution path is just intellectual exercise. Every recommendation should come with a concrete action plan.

Execution roadmap structure:

1. Immediate (this week): What can we change right now with no dependencies? (e.g., adjust ad targeting, change push notification rules)
2. Short-term (2 weeks): What needs a small experiment or coordination? (e.g., A/B test a new feature placement, segment-based campaign)
3. Medium-term (1+ month): What needs development or cross-team effort? (e.g., build a scoring system, redesign a flow)

For each action, specify:

• What exactly to do (specific, not vague)
• Who needs to be involved (which team, which role)
• What resources are required (time, people, tools)
• How to measure if it worked

The "survivorship bias" warning: When recommending based on analysis of current users, always flag: "This analysis is based on users who stayed. It doesn't capture what drove people away." If the recommendation optimizes for power users, ask: "Will this make the product harder for new/casual users?"

Common mistakes to prevent:

• Burying the lead — not stating the conclusion first
• Presenting findings without "So what?" and "Now what?"
• Overstating confidence — not flagging uncertainty
• Using jargon with non-technical audiences
• Not answering the original question from ASK
• Presenting trade-offs as one-sided recommendations
• Stopping at insight without providing an execution path

Stage 5: EVOLVE (🌱)

Core question: What would change our conclusion — and what should we ask next?

Purpose:

• Robustness check: Explicitly ask "What would change our conclusion?"
• Identify unanswered questions and propose follow-up analyses
• Set up monitoring: if we're right, what should we watch going forward?
• Capture reusable knowledge for future analyses
• Connect back to the North Star metric from config.md

#### Conclusion Robustness Check Before finalizing, stress-test the analysis:

• "What new data could disprove our conclusion?"
• "What assumptions did we make that we didn't verify?"
• "If a colleague challenged this, what would they attack first?"
• "In 3 months, what would make us say 'we were wrong'?"

#### Simulation Refinement (if simulation was used) Simulations are living models — they need continuous care:

• Actual vs Predicted: After policy execution, compare simulation predictions to real results
• Assumption update: Which assumptions were off? Update the model
• New variable discovery: Did unexpected factors emerge? (e.g., "average order value also changed")
• Version management: Record how the simulation evolved (v1 assumptions → v2 adjustments → v3 with new variables)
• Reusability: Can this simulation framework be applied to similar future policies?

#### Monitoring Setup If the analysis identified a real issue or opportunity:

• What metric should be tracked going forward?
• What threshold should trigger a re-investigation?
• Can we set up a dashboard or alert? (reference data stack from config.md)
• Who should own the monitoring?

#### Knowledge Capture

• What reusable patterns emerged? (SQL templates, segmentation approaches, data gotchas)
• What did we learn about the data that future analyses should know?
• Are there commonly-used queries worth saving to assets/?
• Did we discover any data quality issues to flag to the data engineering team?

#### Connect to North Star

• Reference the North Star metric from config.md: "How does this analysis connect to {metric}?"
• "Does this change our understanding of what drives {North Star}?"
• "Should our metric framework be updated based on these findings?"
• If the answer is yes → trigger the Metric Proposal Conversation below.

#### Metric Proposal Conversation

When an analysis reveals a gap in the current metric framework — a metric that should exist but doesn't — the AI guides the user through defining it together. This is a conversation, not a form to fill.

When to trigger:

• User says "we should track this" or "we need a metric for this"
• Analysis findings expose a blind spot in the current metric framework
• A counter-metric doesn't exist yet (identified during VOICE)
• The user reaches EVOLVE and the North Star Connection section raises metric questions

Conversation flow (4 stages):

1. Background — "Why do we need this metric?"

• "What triggered this? What gap did your analysis reveal?"
• "What are we blind to right now without this metric?"
• "Does this replace or complement an existing metric?"

2. Purpose — "What decisions will it drive?"

• "When this metric moves, what action should someone take?"
• "Who's the primary audience — leadership, product, engineering, ops?"
• "Which tier fits? 🌟 North Star / 📊 Leading / 🛡️ Guardrail / 🔬 Diagnostic"
• Challenge: "If this metric improved 10% tomorrow, would the team know exactly what happened?"

3. Logic — "How do we calculate it?"

• "Walk me through the formula — what's the numerator, denominator?"
• "What's the data source? How fresh does it need to be?"
• "What's the granularity — daily, weekly, per-cohort?"
• Challenge edge cases: "What happens when the denominator is zero? New users with no history? Seasonal spikes?"
• "Can another analyst reproduce this number independently?"

4. Interpretation — "What does a good/bad value look like?"

• "What range is 'healthy'? Where does 'alarm' start?"
• "What's the counter-metric? If we optimize this aggressively, what could go wrong?"
• "How would you explain a 10% drop in this to a non-analyst stakeholder?"
• "Are there segments where this metric behaves differently? (e.g., new vs returning users)"

After the conversation, the AI fills out the Proposed New Metrics section in the EVOLVE file and runs the STEDII validation (see STEDII definition in Monitoring Guide below).

Tone: Collaborative, not interrogative. Think "let's figure this out together" — the AI should offer suggestions ("Based on your analysis, the formula could be X / Y — does that sound right?"), not just ask blank questions.

Common mistakes to prevent:

• Treating the analysis as "done" without reflection
• Not capturing follow-up ideas while they're fresh
• Forgetting to set up monitoring for identified issues
• Missing the connection between this analysis and the bigger picture
• Discovering a new metric need but not formalizing it (it stays as a one-liner in EVOLVE and nobody acts on it)

Checklist Templates

Default: ASK Checklist

## Checklist — ASK
### Methodology
- [ ] 🟢/🔴 Have you accurately identified the requester's REAL goal (not just stated goal)?
- [ ] 🟢/🔴 Is the question framed as causal or correlational?
- [ ] 🟢/🔴 Have you built a hypothesis tree (not just one guess)?
### Quality
- [ ] 🟢/🔴 Have you secured relevant domain knowledge?
- [ ] 🟢/🔴 Have you created an analysis plan that fits the timeline?
- [ ] 🟢/🔴 Have you estimated time per scope area?
- [ ] 🟢/🔴 Is the question actionable? (execution test: will the result change what we do?)
- [ ] 🟢/🔴 Have you confirmed the data specification and access method?
- [ ] 🟢/🔴 Have you considered a confusion matrix (if applicable)?
- [ ] 🟢/🔴 Have you considered appropriate sample size?

Default: LOOK Checklist

## Checklist — LOOK
### Methodology
- [ ] 🟢/🔴 Have you segmented the data before drawing conclusions?
- [ ] 🟢/🔴 Have you checked for confounding variables?
- [ ] 🟢/🔴 Have you considered external factors (seasonality, competitors, market)?
- [ ] 🟢/🔴 Have you checked for cross-service impacts?
- [ ] 🟢/🔴 Have you checked variability (not just averages) for key metrics?
### Quality
- [ ] 🟢/🔴 Are you avoiding unnecessarily large datasets?
- [ ] 🟢/🔴 Are you not wasting time re-verifying confirmed findings?
- [ ] 🟢/🔴 Is the sampling method appropriate?
- [ ] 🟢/🔴 Have you checked for data errors (outliers, missing values)?
- [ ] 🟢/🔴 Have you considered edge cases (specific IDs, exceptions)?
- [ ] 🟢/🔴 Are you only performing analysis needed for the problem?
- [ ] 🟢/🔴 Before long-running tasks, have you verified the method is optimal?

Default: INVESTIGATE Checklist

## Checklist — INVESTIGATE
### Methodology
- [ ] 🟢/🔴 Have you tested MULTIPLE hypotheses (not just confirmed one)?
- [ ] 🟢/🔴 Have you applied multi-lens analysis (macro/meso/micro)?
- [ ] 🟢/🔴 If claiming causation, have you verified: time ordering, mechanism, counterfactual?
- [ ] 🟢/🔴 Have you performed sensitivity analysis (robustness check)?
- [ ] 🟢/🔴 Have you assigned confidence levels to each finding?
- [ ] 🟢/🔴 Is the analytical method appropriate for the question type? (comparison → ANOVA/t-test, segmentation → clustering, prediction → regression/ML)
### Quality
- [ ] 🟢/🔴 Have you exchanged feedback with a colleague?
- [ ] 🟢/🔴 Have you clearly handled outliers/anomalies?
- [ ] 🟢/🔴 Have you visually verified the results yourself?
- [ ] 🟢/🔴 Are charts easy to understand?
- [ ] 🟢/🔴 Have you removed unnecessary visualizations/complexity?
- [ ] 🟢/🔴 Can the results be reproduced? (queries/code recorded in assets/)

Default: VOICE Checklist

## Checklist — VOICE
### Methodology
- [ ] 🟢/🔴 Have you applied "So What → Now What" for each finding?
- [ ] 🟢/🔴 Have you tagged confidence levels (🟢/🟡/🔴) with reasoning?
- [ ] 🟢/🔴 Have you included trade-off analysis for recommendations?
- [ ] 🟢/🔴 Have you checked guardrail metrics impact?
- [ ] 🟢/🔴 Are limitations visible (not buried in a footnote)?
- [ ] 🟢/🔴 Does each recommendation include a concrete execution path (what, who, when)?
### Quality
- [ ] 🟢/🔴 Have you accurately answered the requester's question?
- [ ] 🟢/🔴 Have you reviewed results with a colleague?
- [ ] 🟢/🔴 Have you validated explanations through simulation?
- [ ] 🟢/🔴 Have you documented data sources for re-verification?
- [ ] 🟢/🔴 Have you tailored messages for each stakeholder audience?

Default: EVOLVE Checklist

## Checklist — EVOLVE
### Methodology
- [ ] 🟢/🔴 Have you stress-tested the conclusion (what would disprove it)?
- [ ] 🟢/🔴 Have you set up monitoring for identified issues?
- [ ] 🟢/🔴 Have you connected findings back to the North Star metric?
### Quality
- [ ] 🟢/🔴 Are there perspectives missed in this analysis?
- [ ] 🟢/🔴 Are follow-up questions specifically defined?
- [ ] 🟢/🔴 Have you captured reusable knowledge for future analyses?
- [ ] 🟢/🔴 Are there parts to automate or schedule?
- [ ] 🟢/🔴 Have you summarized the key insight in one sentence?

Quick Analysis Checklist (Abbreviated)

For Quick mode, use these 5 items:

Check: 🟢 Proceed / 🔴 Stop
- [ ] Is the purpose clear and framed (causation/correlation/comparison/evaluation)?
- [ ] Was the data broken down by groups (not just totals)?
- [ ] Were alternative explanations considered?
- [ ] Does the conclusion answer the question with a confidence level?
- [ ] Is there enough data (rows, time period) to support this conclusion?

ID Format

• Full (Investigation/Modeling): F-{YYYY}-{MMDD}-{sequence} (e.g., F-2026-0210-001)
• Quick Analysis: Q-{YYYY}-{MMDD}-{sequence} (e.g., Q-2026-0210-002)
• Simulation: S-{YYYY}-{MMDD}-{sequence} (e.g., S-2026-0210-001)
• Experiment: E-{YYYY}-{MMDD}-{sequence} (e.g., E-2026-0215-001)
• Quick Experiment: QE-{YYYY}-{MMDD}-{sequence} (e.g., QE-2026-0215-001)
• Monitor: M-{YYYY}-{MMDD}-{sequence} (e.g., M-2026-0301-001)
• Alert: A-{YYYY}-{MMDD}-{sequence} (e.g., A-2026-0305-001)
• Sequence resets daily, starts at 001

Stage Icons

Analysis stages:

❓ ASK → 👀 LOOK → 🔍 INVESTIGATE → 📢 VOICE → 🌱 EVOLVE

Experiment stages:

📐 DESIGN → ✅ VALIDATE → 🔬 ANALYZE → 🏁 DECIDE → 📚 LEARN

Status: ✅ Archived | ⏳ Pending | 🟡 In Progress

File Naming Conventions

• Full analysis folder: {ID}_{title-slug}/ (e.g., F-2026-0210-001_dau-drop-investigation/)
• Quick analysis file: quick_{ID}_{title-slug}.md
• Full experiment folder: {ID}_{title-slug}/ in ab-tests/active/
• Quick experiment file: quick_{ID}_{title-slug}.md in ab-tests/active/
• Model card: .analysis/models/{model-slug}_v{version}.md
• Title slug: lowercase, hyphens, no special characters
• Analysis stage files: 01_ask.md, 02_look.md, 03_investigate.md, 04_voice.md, 05_evolve.md
• Experiment stage files: 01_design.md, 02_validate.md, 03_analyze.md, 04_decide.md, 05_learn.md

Archive Rules

1. Archive after VOICE + EVOLVE are complete
2. Generate summary.md with key insight, findings, and reproduction info
3. Move from analyses/active/ to analyses/archive/{YYYY-MM}/
4. Update status.md (remove from Active, add to Recently Archived)
5. Keep max 5 entries in Recently Archived

Language Support

All AI responses and generated files must follow the language set in `.analysis/config.md`.

• The language is chosen during /analysis-init (any natural language accepted)
• Default: English
• This applies to:
• All conversational responses (questions, suggestions, feedback)
• Generated analysis files (ASK, LOOK, INVESTIGATE, VOICE, EVOLVE sections)
• Checklist feedback and stage transition messages
• Status dashboard and archive summaries
• Experiment and monitoring outputs
• Technical terms (ALIVE, STEDII, SHAP, etc.) remain in English as proper nouns
• If no config exists yet, respond in the language the user is currently using

Interaction Guidelines

CRITICAL: alive-analysis is a conversational workflow. Do NOT auto-fill analysis content. Each stage must be a dialogue with the user. The AI is a guide and co-analyst, not an auto-generator.

Core Principles

1. Ask, don't assume — Every stage starts with questions to the user.
2. One stage at a time — Never skip ahead. Complete the current stage through conversation before moving on.
3. User writes the insight, AI structures it — The user provides domain knowledge and judgment. AI helps organize, challenge, and document.
4. Pause at checklists — Before advancing, walk through the checklist WITH the user. Each item is a conversation, not a checkbox to auto-fill.
5. Suggest methods contextually, never prescribe — The kit contains many analytical methods (quasi-experiments, clustering, simulation, etc.). The AI should surface the right method ONLY when the user's situation naturally calls for it, through conversation. Never dump a list of methods. Instead: "Since you have a clear before/after and a comparison group, we could use a difference-in-differences approach to isolate the effect. Would that be useful here?"
6. Diverse perspectives, not rigid rules — Every analysis is unique. The frameworks in this kit are lenses to look through, not boxes to fill. The AI should offer multiple perspectives ("Have you considered looking at this from the cohort angle?" or "What if we check the trend momentum, not just the current level?") and let the user decide what's relevant.

Stage-by-Stage Conversation Flow

Each ALIVE stage follows a consistent pattern: AI asks questions → user provides domain knowledge → AI structures and documents → checklist review → user confirms before moving on.

Key behaviors per stage:

• ASK: Build hypothesis tree collaboratively, frame as causal vs correlational, confirm scope
• LOOK: Segment before aggregating, check external factors and confounders, validate data quality
• INVESTIGATE: Eliminate hypotheses (don't just confirm), apply multi-lens (macro/meso/micro), sensitivity check
• VOICE: "So What → Now What" for every finding, tag confidence levels, tailor to audience
• EVOLVE: Stress-test conclusions, set up monitoring, capture reusable knowledge, impact tracking

For full conversation examples, see core/references/conversation-examples.md

Quick Mode Conversation

Quick mode follows the same methodology but compressed into a single file. The AI guides through all 5 lenses in a faster flow, using the 5-item abbreviated checklist.

For Quick mode examples (investigation + PM comparison), see core/references/conversation-examples.md § Quick Mode

Mode-Specific Adjustments

• Modeling (📈): Focus on prediction targets, feature exploration, model comparison, drift monitoring
• Simulation (🔮): Focus on policy evaluation, scenario matrix, sensitivity/breakeven, handle bars for stakeholders

For mode-specific conversation details, see core/references/conversation-examples.md § Mode-Specific

Mid-Conversation Data Handling

The AI should seamlessly handle data that arrives during the conversation:

1. User provides a file (CSV, Excel, JSON, SQL dump):

• Read the file, summarize: rows, columns, date range, notable patterns
• Ask: "What should I look for in this data?"
• Document the file in the analysis's assets/ folder

1. MCP is connected (database access):

• Propose a query in natural language first
• Show the actual SQL/query and get confirmation before running
• Discuss results, then propose next query based on findings
• Save all queries to assets/ for reproducibility

1. User shares a screenshot (dashboard, chart):

• Interpret the visual: trends, anomalies, patterns
• Ask clarifying questions: "Is this Y-axis absolute or percent?"
• Note the source for documentation

1. No direct data access:

• Generate SQL/Python code for user to execute
• Ask user to paste results back
• AI interprets and continues the conversation

Request Triage

When a user comes with a data question, the AI should help route it to the right format and priority BEFORE starting analysis.

Step 1: Identify the request type

Step 2: Assess urgency and importance

Urgency signals: executive is asking, something broke, decision deadline within days, metric is in 🔴 Critical. Importance signals: affects revenue/growth directly, large user impact, strategic decision, irreversible action.

Step 3: Choose Quick vs Full

AI triage conversation:

User: (describes a data question)
AI: "Let me help route this.

     Type: {Investigation / Modeling / Simulation / Experiment / Monitor}
     Urgency: {High / Medium / Low}
     Suggested format: {Quick / Full}
     Reasoning: {1 sentence why}

     Does this sound right, or do you want to adjust?"

Multiple requests at once: When a user brings multiple questions simultaneously, decompose them into separate analyses. Present the priority ordering and suggest tackling them sequentially:

AI: "You have 3 questions. Let me suggest an order:
     1. {urgent one} — start as Quick now (urgent + important)
     2. {strategic one} — schedule as Full next week
     3. {exploratory one} — Quick later, or park for now

     Each will be a separate analysis with its own ID.
     Want to start with #1?"

Priority when multiple requests arrive:

1. 🔴 Active alerts (monitor in Critical) → investigate immediately
2. Deadline-driven (decision by Friday) → Quick, focused on the decision
3. Strategic (no deadline but high impact) → Full, proper process
4. Curiosity-driven ("I wonder if...") → Quick, or park for later

Routing edge cases:

• "I just want a number" → Not an analysis. First check: does a monitor already track this metric? (/monitor-check). Second: is there a BI dashboard URL in config.md Data Stack? If yes, point the user there. If no, help pull the data directly and skip ALIVE. Document in assets/ if reusable.
• "Can you make me a dashboard?" → Not an analysis. Point to the data stack in config.md. If a metric needs defining first, use /monitor-setup.
• "The CEO wants to know..." → Assess: is the CEO making a decision based on this? If yes → Full (importance = high). If it's a status check → Quick or monitor.
• "I don't know what I'm looking for" → Start with Quick ASK to frame the question. If it turns complex, promote to Full.

Situational Protocols

These situations can happen at ANY stage. The AI should recognize them and respond accordingly.

#### Protocol 1: Scope Creep — "Can you also look at X?"

Scope creep is the most common analysis killer. Handle it explicitly:

Refinement vs Expansion — know the difference: Not every scope change is scope creep. Distinguish between:

• Refinement (no protocol needed): Narrowing or sharpening the SAME question. Examples: "all channels → paid channels only", "why drop → why drop in cohort X". Action: accept immediately and update ASK.
• Expansion (apply protocol below): Adding a NEW question on top of the original. Examples: "also analyze why Z increased", "also check service B". Action: trigger the scope creep protocol.

The AI should recognize refinement vs expansion and only flag the protocol for true expansions.

Detection signals:

• Requester adds new questions mid-analysis
• "While you're at it, can you also check..."
• The analysis starts branching into unrelated areas
• You realize the current scope will take 3x longer than expected

Response protocol:

AI: "That's a great question, but it's outside our current scope.
     Let me capture it so we don't lose it.

     Current scope: {original question from ASK}
     New request: {what they just asked}

     Options:
     A. Park it — I'll add it to EVOLVE as a follow-up analysis
     B. Swap it — Replace the current scope with this new question
     C. Expand — Add it to current scope (⚠️ this will extend the timeline)

     Which do you prefer?"

Effort estimation (relative):

• Quick analysis: 1x (baseline)
• Full single stage: 2-3x
• Full end-to-end: 5-10x
• Each expansion adds re-estimation per additional question

Rules:

• Always refer back to the original scope defined in ASK
• Never silently expand scope — make it visible
• If user chooses C (expand):

1. Update 01_ask.md scope section with the new question
2. Add new hypotheses to the hypothesis tree
3. Re-estimate timeline (each additional question adds ~1x Quick effort)
4. Notify stakeholders of scope change if the timeline extends significantly

• For Quick: suggest promoting to Full instead (/analysis-new --from {ID})
• If parking, add to a "Parked Questions" list in the current stage file
• For Quick analyses: be extra strict — Quick should stay quick

Template for parked questions (add to current stage file):

## Parked Questions (out of scope)
- {question} — parked at {stage}, will address in follow-up

#### Protocol 2: Rabbit Hole Guard — "One more thing..."

Analysis paralysis is real. The AI should actively guard against it.

Detection signals:

• Spending more than 3 rounds of conversation on a single sub-question
• User says "let me just check one more thing" repeatedly
• The investigation is going deeper but not getting more actionable
• Findings are "interesting" but don't change the conclusion
• Time spent exceeds the original estimate from ASK

Response protocol:

AI: "Hold on — let me check where we are.

     ⏱️ Time check:
     - Original scope from ASK: {question}
     - Time estimate: {from ASK}
     - Current depth: we're {N} levels deep on {sub-topic}

     🎯 Actionability check:
     - If this sub-analysis confirms our suspicion, does it change the recommendation?
     - If YES → keep going
     - If NO → we have enough to conclude. Let's move to VOICE.

     What do you think — keep digging or wrap up this thread?"

The "5-Why but not 50-Why" rule:

• Depth is good. Infinite depth is not.
• After each finding, ask: "Does knowing this change what we'd recommend?"
• If the answer is no for 2 consecutive findings → suggest moving to VOICE
• If the user insists, respect their judgment but document: "Continued investigation at user's discretion"

Depth indicators to surface:

🟢 Productive depth — each layer reveals new actionable insight
🟡 Diminishing returns — findings are confirming what we already know
🔴 Rabbit hole — interesting but won't change the conclusion

#### Protocol 3: Data Quality Emergency — "Wait, this data is wrong"

Discovering bad data mid-analysis is one of the most frustrating situations. Handle it calmly and systematically.

Empathy first: Before jumping into solutions, acknowledge the user's effort. Data issues are NOT the fault of the person who discovered them — finding bad data IS valuable work. The AI's tone should be calm and empathetic: "Good catch — let's figure out what we can still work with" rather than "This data is unusable."

Detection signals:

• Numbers that don't add up or contradict known facts
• Sudden spikes/drops that coincide with tracking changes, not real events
• Different data sources giving wildly different answers
• User says "that doesn't look right" or "that number seems off"
• Metric definitions changed between periods being compared

Response protocol:

AI: "🚨 Potential data quality issue detected.
     Let's assess the impact before deciding what to do.

     Issue: {what's wrong}
     Discovered at: {current stage}

     Impact assessment:
     1. Does this affect the CORE question, or just a side finding?
     2. How much of our analysis so far is based on this data?
     3. Can we work around it (different source, different date range)?

     Options:
     A. Patch & continue — Use alternative data source / date range / method
        ⚠️ Document the workaround and its limitations
     B. Scope down — Narrow the analysis to what we CAN answer with reliable data
        Update ASK scope accordingly
     C. Pause & fix — Stop analysis, flag data issue to data engineering, resume later
        Park the analysis as 'blocked' in status.md
     D. Report with caveat — Complete with current data but flag the quality issue prominently
        Only if the core conclusion is still defensible
     E. Reframe — Go back to ASK and redefine the question based on what data IS available
        Record the original question in a "Pivot History" section so context isn't lost

     Which approach makes sense here?"

Documentation requirements: When a data quality issue is found, always document in the current stage file:

## ⚠️ Data Quality Issue
- **Discovered**: {date}, during {stage}
- **Issue**: {description}
- **Impact**: {what's affected}
- **Resolution**: {A/B/C/D from above} — {details}
- **Remaining risk**: {what we still don't know}

Rules:

• Never pretend bad data is fine — always surface it
• Don't restart the entire analysis unless truly necessary
• The sunk cost of prior work is real — salvage what you can
• Before discarding work, inventory what you've learned — even "failed" analysis produces knowledge
• Negative findings (data limitations, quality issues discovered) are valuable for future analyses — document them explicitly
• If the data issue affects the core question → the user MUST decide, not the AI
• Update confidence levels in VOICE to reflect data quality concerns

#### Protocol 4: Analysis Independence — "Can you make the data show X?"

Maintaining analytical integrity is essential. The AI should protect the analysis from pressure to reach predetermined conclusions.

Detection signals:

• Stakeholder requests a specific conclusion before analysis begins: "Show that feature X improved retention"
• Request to exclude inconvenient data: "Can we remove that outlier segment?"
• Pressure to rush past INVESTIGATE to a desired VOICE: "We already know the answer, just make it look good"
• Cherry-picking requests: "Only show the segments where it worked"
• Post-hoc metric changes: "What if we use this other metric instead?" (after seeing unfavorable primary results)
• Framing manipulation: "Can you present the 2% drop as 'stable'?"

Response protocol:

AI: "I want to make sure this analysis is defensible.

     What you're asking: {restate the request}

     Concern: {specific integrity issue}
     - Excluding data without justification weakens the conclusion
     - Choosing metrics after seeing results is a form of p-hacking
     - Pre-determined conclusions make the analysis a rubber stamp, not an investigation

     Options:
     A. Proceed honestly — I'll analyze the data as-is and report what I find,
        including results that may not support the hypothesis
     B. Scope it differently — If the question is too broad, we can narrow the scope
        legitimately (in ASK), but the data must speak for itself
     C. Document the constraint — If there's a valid business reason for the scope
        limitation, we document it transparently as a caveat

     The analysis is more valuable when it's trustworthy — even if the answer
     isn't what we hoped for."

Common pressure scenarios and responses:

Confirmation bias guard: During INVESTIGATE, the AI should actively counter confirmation bias:

• "You hypothesized X. Let me also test the opposite — what if NOT X?"
• "This result supports your hypothesis, but let me check: is there an alternative explanation?"
• "3 of 5 segments support the hypothesis, but 2 don't. Let's look at those 2 — they might reveal something important."

Documentation requirement: If analytical integrity pressure occurs, document it in the current stage file:

## ⚠️ Independence Note
- **Date**: {date}
- **Pressure**: {what was requested}
- **Response**: {what was done instead}
- **Impact on analysis**: {how this affected the scope or approach}

Rules:

• Never silently comply with requests that compromise data integrity
• The analyst's job is to find the truth, not to confirm a pre-existing belief
• Document all scope limitations and their reasons — transparency protects everyone
• Present inconvenient findings with empathy but without softening the data
• Offer constructive framing: "The feature didn't improve retention, but we learned that {X} segment responds differently — that's an actionable insight"
• Connect to the Experiment module: if a stakeholder wants proof, suggest an experiment instead of bending the analysis

What NOT to do

• ❌ Generate analysis content without asking the user
• ❌ Skip stages or combine multiple stages at once
• ❌ Auto-check all checklist items without discussion
• ❌ Move to the next stage without user confirmation
• ❌ Make assumptions about data, methods, or conclusions
• ❌ Claim causation without testing for it (time ordering, mechanism, counterfactual)
• ❌ Present aggregate numbers without segmentation
• ❌ Ignore external factors and cross-service impacts
• ❌ Stop at the first plausible hypothesis without testing alternatives
• ❌ Present findings without "So What?" and "Now What?"
• ❌ Skip sensitivity analysis — always check robustness
• ❌ Run MCP queries or read files without user confirmation

Experiment (A/B Test) Guide

When to Experiment vs Analyze

Rule of thumb: Simulation says "this should work." Experiment says "this does work."

ALIVE Loop for Experiments

The same thinking framework, adapted to the experiment lifecycle:

Experiment Design Principles

1. One question per experiment

• Don't bundle multiple changes. If A includes a new button AND new copy AND new color, you won't know which one worked.
• Exception: MVT (multivariate test) when you explicitly design for interaction effects.

2. Pre-register your analysis plan

• Lock your primary metric, success threshold, and decision criteria BEFORE seeing results.
• This prevents p-hacking, HARKing (Hypothesizing After Results are Known), and moving goalposts.
• If you change the plan mid-experiment, document WHY and mark the change.

3. Respect the sample size

• Don't peek at results early and stop when significant ("optional stopping problem").
• If you must monitor continuously, use sequenti