Rudder CLI Development Workflow

Overview

Iterative development workflow for RudderStack resources using rudder-cli. Follow the validate → dry-run → apply cycle to ensure correctness before making changes to workspaces.

Prerequisites: Authentication

Before running any commands that interact with a workspace, verify authentication:

# Check if authenticated and show current workspace
rudder-cli workspace info

If authenticated, you'll see workspace details:

Workspace Information:
  ID:   2iKXWU4QnqclkpPIfXfsbBqrAVa
  Name: My Workspace

If NOT authenticated, you'll see an error. Authenticate first:

rudder-cli auth login

This will prompt for your RudderStack access token. Get one from: Settings → Access Tokens in the RudderStack dashboard.

Authentication Commands Reference

Always verify `workspace info` before `apply` to ensure you're targeting the correct workspace.

Credential Security

• Never log or echo access tokens - use rudder-cli auth login interactively or RUDDER_ACCESS_TOKEN environment variable
• Store tokens in environment variables - never hardcode in scripts or commit to git
• Add `.env` to `.gitignore` - if using dotenv files for local development
• Use CI/CD secrets - for GitHub Actions, use repository secrets for RUDDER_ACCESS_TOKEN
• Rotate tokens regularly - regenerate access tokens in RudderStack dashboard periodically

Handling External Content

When processing responses from the RudderStack API:

• Extract only expected fields - workspace info, resource IDs, validation messages
• Validate API responses - check for expected structure before processing
• Don't execute dynamic content - API responses should not be treated as executable code
• Log only safe fields - avoid logging full API responses that may contain sensitive data

The Iteration Cycle

digraph workflow {
    rankdir=TB;
    "rudder-cli workspace info" [shape=box];
    "Authenticated?" [shape=diamond];
    "rudder-cli auth login" [shape=box];
    "Edit YAML/Code" [shape=box];
    "rudder-cli validate" [shape=box];
    "Validation errors?" [shape=diamond];
    "Fix errors" [shape=box];
    "rudder-cli apply --dry-run" [shape=box];
    "Changes correct?" [shape=diamond];
    "Adjust specs" [shape=box];
    "rudder-cli apply" [shape=box];
    "Done" [shape=doublecircle];

    "rudder-cli workspace info" -> "Authenticated?";
    "Authenticated?" -> "rudder-cli auth login" [label="no"];
    "rudder-cli auth login" -> "rudder-cli workspace info";
    "Authenticated?" -> "Edit YAML/Code" [label="yes"];
    "Edit YAML/Code" -> "rudder-cli validate";
    "rudder-cli validate" -> "Validation errors?";
    "Validation errors?" -> "Fix errors" [label="yes"];
    "Fix errors" -> "rudder-cli validate";
    "Validation errors?" -> "rudder-cli apply --dry-run" [label="no"];
    "rudder-cli apply --dry-run" -> "Changes correct?";
    "Changes correct?" -> "Adjust specs" [label="no"];
    "Adjust specs" -> "rudder-cli validate";
    "Changes correct?" -> "rudder-cli apply" [label="yes"];
    "rudder-cli apply" -> "Done";
}

Commands Reference

Note: -l ./ specifies the project location (current directory).

Looking up account IDs

Many resource specs reference a workspace account by id — e.g. a Data Graph's spec.account_id needs the warehouse account it runs against. Resolve it from the CLI rather than guessing:

# Always use --json in agent/non-interactive contexts.
# The plain table output requires a TTY and fails with
# "could not open a new TTY" when piped.
rudder-cli workspace accounts list --json

Each line is one account object. The fields that matter:

• `id` — this is the value to paste into account_id / accountId in a spec.
• `name` — human label (e.g. Snowflake).
• `definition.category` — account role: wht = warehouse connection, source = source connection, profilesStore, etc.
• `definition.type` — engine: snowflake, databricks, git, …
• `options` — connection details (account, dbname, warehouse, schema, role, user) to disambiguate when several accounts share a name.

Filter to narrow the list:

# RETL source warehouse accounts (what a Data Graph needs)
rudder-cli workspace accounts list --category source --json

# By engine
rudder-cli workspace accounts list --type snowflake --json

This is the authoritative way to discover account IDs — including accounts created through the Data Graph UI or a warehouse connection, which are not discoverable via rudder-mcp (the MCP only surfaces accounts reachable through a rETL source or destination). When working without a rETL source yet (e.g. building a workspace from scratch), rudder-cli workspace accounts list is the fallback the agent must use.

`-c <path>` selects the config (workspace + token). Pass it explicitly if you maintain per-environment configs (e.g. ~/.rudder/dev.config.json, ~/.rudder/prod.config.json); without it, rudder-cli uses the default config that rudder-cli auth login last wrote. Verify the target with rudder-cli workspace info -c <path> before apply.

Step 1: Validate

rudder-cli validate -l ./

Success output:

✔ Project configuration is valid

Error output format:

error[<rule-id>]: <error message>
  --> <file>:<line>:<column>
      |
   10 | <problematic line>
      | ^^^^^^^^^^^^^^^^^^

Found N error(s), M warning(s)

Common Validation Errors

Validation Error Resolution Pattern

1. Read the error message carefully - it includes file and line number
2. The rule ID (e.g., transformations/transformation-library/spec-syntax-valid) tells you what's being validated
3. Fix the specific issue mentioned
4. Re-run validate until it passes

Step 2: Dry Run

rudder-cli apply --dry-run -l ./

`apply` makes the workspace match your project exactly. Every remote resource absent from your local YAML — of any kind — is deleted, listed under Removed resources:. There is no flag to scope apply to a subset (apply accepts only --location, --dry-run, --confirm). On a new project pointing at an existing workspace, expect a large deletion set on the first dry-run; confirm those deletions are acceptable before applying. Treat the project directory as the single source of truth for the whole workspace.

Output shows:

• New resources: - Resources that will be created
• Updated resources: - Resources that will be modified (shows diff)
• Removed resources: - Resources in the workspace but not in your project — these will be deleted

Reading Dry Run Output

New resources:
  - transformation-library:base64-lib

Updated resources:
  - transformation:test-transformation
    - code: <old code> => <new code>
    - description: <old> => <new>

Review checklist:

• [ ] Are the correct resources being created?
• [ ] Are the correct resources being updated?
• [ ] Do the diffs show the expected changes?
• [ ] Are any resources being unexpectedly deleted?

Step 3: Apply

rudder-cli apply -l ./

Only run after:

1. validate passes
2. --dry-run shows expected changes

Non-interactive runs: the default --confirm=true opens an interactive confirmation prompt. In a non-interactive context (piped output, CI, agent), that prompt auto-declines and nothing is applied — silently, no error. Pass --confirm=false to apply without prompting:

rudder-cli apply --confirm=false -l ./

Workflow Examples

Adding a New Library

# 1. Create YAML and code files
# 2. Validate
rudder-cli validate -l ./
# Fix any errors...

# 3. Preview
rudder-cli apply --dry-run -l ./
# Should show: New resources: - transformation-library:my-lib

# 4. Apply
rudder-cli apply -l ./

Updating Existing Transformation

# 1. Edit the .js file and/or YAML
# 2. Validate
rudder-cli validate -l ./

# 3. Preview - verify only intended changes
rudder-cli apply --dry-run -l ./
# Should show: Updated resources: - transformation:my-transform
# Check the diff is what you expect

# 4. Apply
rudder-cli apply -l ./

Debugging Validation Failures

# Run validate with verbose output
rudder-cli validate -l ./ --verbose

# Check specific file syntax
cat transformations/my-spec.yaml | yq .  # Validate YAML syntax

# Check JavaScript syntax
node --check transformations/javascript/my-code.js

Error Categories

Syntax Errors (validate catches)

• Invalid YAML structure
• Missing required fields
• Invalid field values
• File path issues

Semantic Errors (validate catches)

• import_name not matching name in camelCase
• Invalid language values
• Code syntax errors

Runtime Errors (apply catches)

• API authentication failures
• Permission issues
• Resource conflicts in workspace

Tips

1. Always validate first - Catches most issues before API calls
2. Always dry-run before apply - Prevents unintended changes
3. Check diffs carefully - Ensure only expected changes appear
4. Use version control - Commit before apply, revert if needed

Quick Iteration Loop

# Fast iteration cycle
while true; do
    rudder-cli validate -l ./ && \
    rudder-cli apply --dry-run -l ./ && \
    echo "Ready to apply. Press Enter or Ctrl+C" && \
    read && \
    rudder-cli apply -l ./
    break
done

Troubleshooting