Debugging an Isabl Analysis

You are helping debug a failed or problematic Isabl analysis.

Checklist

Work through these steps systematically:

1. [ ] Get analysis details (status, application, targets)
2. [ ] Check the command script (head_job.sh)
3. [ ] Review stdout log (head_job.log)
4. [ ] Review stderr log (head_job.err)
5. [ ] Check input data exists (BAM files, dependencies)
6. [ ] Verify settings (application_settings values)
7. [ ] Identify root cause
8. [ ] Suggest fix

Step 1: Get Analysis Details

import isabl_cli as ii

# Get the analysis
analysis = ii.Analysis(ANALYSIS_PK)

print(f"Status: {analysis.status}")
print(f"Application: {analysis.application.name} v{analysis.application.version}")
print(f"Storage URL: {analysis.storage_url}")
print(f"Targets: {[t.system_id for t in analysis.targets]}")
print(f"References: {[r.system_id for r in analysis.references]}")

Step 2: Check Command Script

The generated command is in head_job.sh:

# View the command that was executed
cat {analysis.storage_url}/head_job.sh

Look for:

• Incorrect paths
• Missing environment variables
• Malformed command syntax

Step 3: Review Logs

# Standard output
cat {analysis.storage_url}/head_job.log

# Standard error (usually more informative)
cat {analysis.storage_url}/head_job.err

Common error patterns:

Step 4: Check Input Data

# Verify target experiments have data
for target in analysis.targets:
    print(f"{target.system_id}:")
    print(f"  Raw data: {target.raw_data}")
    print(f"  BAM files: {target.bam_files}")

# Check files exist
ls -la {path_to_input_file}

Step 5: Check Dependencies

If the app depends on other analyses:

# Find dependency analyses
from isabl_cli import utils

# Check if dependency result exists
result, dep_analysis = utils.get_result(
    experiment=analysis.targets[0],
    application_key=DEPENDENCY_APP_PK,
    result_key="expected_result"
)
print(f"Dependency analysis: {dep_analysis}")
print(f"Dependency result: {result}")

Step 6: Verify Settings

# Get application settings
app = ii.get_instance("applications", analysis.application.pk)
print(f"Settings: {app.settings}")

# Check specific setting
from my_apps import MyApplication
instance = MyApplication()
print(f"Tool path: {instance.settings.tool_path}")

Step 7: Common Fixes

Fix 1: Re-run with --force

# Wipe failed analysis and restart
isabl myapp-1-0 --analyses pk={ANALYSIS_PK} --force --commit

Fix 2: Re-run with --restart

# Resume from checkpoint (if supported)
isabl myapp-1-0 --analyses pk={ANALYSIS_PK} --restart --commit

Fix 3: Run locally for debugging

# Execute locally instead of batch system
isabl myapp-1-0 --targets EXPERIMENT_ID --local --commit

Fix 4: Manually patch status

# If analysis completed but status is wrong
ii.patch_instance("analyses", analysis.pk, status="SUCCEEDED")

Fix 5: Check batch system logs

# For SLURM
sacct -j JOB_ID --format=JobID,State,ExitCode,MaxRSS,Elapsed

# For LSF
bjobs -l JOB_ID
bhist -l JOB_ID

Signals and Automations

If the analysis SUCCEEDED but something else failed:

# Check if signals are configured
from isabl_cli.settings import system_settings
print(f"ON_STATUS_CHANGE: {system_settings.ON_STATUS_CHANGE}")

# Manually trigger signals
from isabl_cli import signals
signals.run_on_status_change(analysis)

Reporting the Issue

When reporting, include:

1. Analysis PK and status
2. Application name and version
3. Relevant portion of head_job.err
4. Input experiment IDs
5. Steps to reproduce