Data Analysis Methodology

Prioritize correctness over speed — a wrong insight is worse than no insight.

Step 1: Data Ingestion

Uploaded files are located at /home/user/uploads/. Always list that directory first to discover available files:

import os
for f in os.listdir('/home/user/uploads/'):
    print(f)

Tool selection: Use code_run as the primary execution tool — it is universally supported across all sandbox providers. Use code_interpret only when you need rich output capture (e.g., inline dataframes, rendered plots); note that code_interpret may not be available in all environments.

Examine the raw file (first 20-30 lines) to understand format, delimiter, encoding, headers, and obvious quality issues before loading.

Load by file type — always specify dtypes for known columns and parse_dates for date columns:

Step 2: Mandatory EDA

Run this diagnostic block on every dataset before any analysis:

print(f"Shape: {df.shape}")
print(f"\nDtypes:\n{df.dtypes}")
print(f"\nMissing values:\n{df.isnull().sum()[df.isnull().sum() > 0]}")
print(f"\nDuplicate rows: {df.duplicated().sum()}")
print(f"\nNumeric summary:\n{df.describe()}")

for col in df.select_dtypes(include='object').columns:
    n_unique = df[col].nunique()
    print(f"\n{col}: {n_unique} unique values")
    if n_unique <= 20:
        print(df[col].value_counts())

Do not skip this step. Report findings before proceeding to analysis.

After completing EDA, send a progress update via user_message summarizing the dataset shape, quality issues found, and your planned analysis approach.

Step 3: Data Cleaning Decision Framework

• Missing values: Report percentage. Drop rows only if <5% missing. Otherwise, impute (median for skewed numeric, mode for categorical) and flag it.
• Duplicates: Drop exact duplicates. For near-duplicates, ask the user.
• Outliers: Detect with IQR method (1.5×IQR). Report them but do NOT remove unless the user asks — outliers can be real data.
• Type coercion: Convert string numbers to numeric, parse dates, encode categoricals.

Step 4: Analysis Technique Selection

Match technique to the question type:

Always report effect sizes and confidence intervals, not just p-values.

After selecting your analysis technique, send a progress update via user_message describing which technique you chose and why, before running the full analysis.

Step 5: Visualization Selection

Choose chart types by data shape:

Sandbox-specific rules:

• Use matplotlib.use('Agg') — there is no display
• Always call plt.close() after saving to free memory
• Save plots to /home/user/output/ at 150 dpi — create the directory first: os.makedirs('/home/user/output/', exist_ok=True)
• When using code_run, pass saved file paths via the output_files parameter so artifacts are tracked. When using code_interpret, rich outputs (plots, dataframes) are auto-captured — but for file-based outputs (saved PNGs), prefer code_run with output_files
• Use colorblind-friendly palettes (tab10, Set2)

Step 6: Report Structure

**Dataset Overview**: [rows × columns, date range if applicable]

**Key Findings**:
1. [Quantified finding — "Revenue increased 23% QoQ"]
2. [Finding with statistical backing — "Correlation r=0.82, p<0.001"]

**Data Quality Notes**: [Cleaning applied, missing data, caveats]

**Methodology**: [Tests/techniques used and why]

Never say "X causes Y" from observational data alone.