fabric-dataflows-perf-remediation — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited fabric-dataflows-perf-remediation (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Systematic diagnostic workflows for identifying and resolving performance issues in Microsoft Fabric Dataflow Gen2 workloads covering data movement, transformation execution, staging configuration, and destination write optimization.
| Symptom | Likely Cause | Jump To |
|---|---|---|
| Refresh takes 2x+ longer than Gen1 | Delta Parquet output overhead or staging misconfiguration | Consideration 5 in Performance Guide |
| Fast Copy not activating | Unsupported connector or non-foldable transforms | Fast Copy Diagnostics |
| "Insufficient permissions for staging artifacts" | Creator token expired (90+ days) | Common Errors |
| Slow design-time previews | Large dataset loaded in editor | Design-Time Optimization |
| Staging-to-Lakehouse write is slow | Extra data hop through staging Warehouse | Staging Strategy |
| Gateway dataflows are slow | All transforms running on gateway host | Gateway Optimization |
| Incremental refresh slower than full | Too many small buckets creating overhead | Incremental Refresh Tuning |
| HTTP 430 / capacity throttled | Concurrent Spark jobs exhausting CUs | Capacity Management |
Key metrics to capture: total duration, per-table duration, bytes read/written, rows read/written, engine type (Mashup vs CopyActivity vs SQL DW).
Dataflow Gen2 has three performance-critical components:
[Data Source] --> [Dataflow Engine] --> [Data Destination]
| | |
Connectors Mashup / Fast Copy / Lakehouse /
+ Gateway SQL DW Compute Warehouse /
SQL DatabaseBased on the bottleneck identified, follow the relevant section below.
Fast Copy provides up to 9x faster ingestion for supported scenarios.
ADLS Gen2, Azure Blob Storage, Azure SQL DB, Lakehouse, PostgreSQL, On-premises SQL Server, Warehouse, Oracle, Snowflake, SQL Database in Fabric.
| Indicator | Meaning | Action |
|---|---|---|
| Green | Step supported by Fast Copy | No action needed |
| Yellow | Some steps may support Fast Copy | Split query at boundary |
| Red | Step NOT supported by Fast Copy | Move to referenced query |
See Performance Optimization Guide for detailed walkthrough.
Is destination a Warehouse?
├── YES → Staging REQUIRED (enabled by default)
│ Write goes directly via SQL DW compute
│ This is the optimal path for Warehouse destinations
└── NO → Is destination a Lakehouse?
├── YES with staging enabled → Data moves:
│ Source → Staging LH → Staging WH → Lakehouse
│ ⚠ CONSIDER: Disable staging to avoid extra hop
│ OR: Switch destination to Warehouse instead
└── YES with staging disabled → Data writes directly
Source → Lakehouse (via Mashup engine)
✓ Fewer hops, but no SQL DW compute for transformsKey Decision: If your transforms fold to the source, disable staging for Lakehouse destinations. If transforms are complex (joins, aggregations), either enable staging or switch to Warehouse destination.
When working with large datasets in the dataflow editor:
DesignDateFilter parameter to limit preview data during authoring, then adjust to full range before publishWhen using on-premises data gateway:
If incremental refresh is slower than full refresh:
| Engine | When Used | Billing Basis |
|---|---|---|
| Standard Compute (Mashup) | Staging disabled or non-foldable queries | Query evaluation time |
| High Scale Compute (SQL DW) | Staging enabled | Lakehouse + Warehouse duration |
| Fast Copy | Supported connectors with Fast Copy enabled | Copy job duration |
All operations are background operations smoothed over 24 hours.
If QueryDuration < 600s:
CU_seconds = QueryDuration × 12
Else:
CU_seconds = (QueryDuration - 600) × 1.5 + 600 × 12| Error | Cause | Resolution |
|---|---|---|
| "Insufficient permissions for staging artifacts" | Creator inactive 90+ days or left org | Have creator log in to Fabric; if gone, open support ticket |
| "Expression.Error: import matches no exports" | Unsupported connector in Premium workspace | Check connector compatibility list |
| "Gateway version not supported" | On-premises gateway out of support | Update to latest gateway version |
| "Staging lakehouse couldn't be found" | CI/CD branch workspace missing staging | Create a new Dataflow Gen2 in workspace to trigger staging creation |
| Validation failure on save | Query schema can't be determined in 10 min | Simplify query or check source connectivity |
The Modern Evaluator engine can provide significant performance improvements:
Enable via dataflow settings. Monitor results after enabling — some connectors may not yet be fully optimized.
Run the Dataflow Health Check Script to programmatically audit dataflow configurations and recent refresh performance across a workspace.
Run the Dataflow Refresh Monitor Script to poll and track active refresh status in real-time.
See REST API Reference for complete API documentation covering CRUD operations, scheduling, refresh triggering, and monitoring.
See Performance Optimization Guide for deep-dive scenarios including Fast Copy benchmarks, staging architecture patterns, and query folding analysis techniques.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.