BoxLang Testing with TestBox

BoxLang is the AI-native software productivity platform for building, modernizing and running applications, with developers and AI agents working together.

Overview

TestBox is the standard testing framework for BoxLang. It supports two styles:

• BDD (Behavior-Driven Development): describe(), it(), feature(), story(), given()/when()/then()
• xUnit: class-based, test*() methods, setup()/tearDown()

Both styles use the same assertions/expectations library and MockBox for mocking.

Test Bundles

A test bundle is a BoxLang class (.bx file) that contains your tests. Name files with Test or Spec suffix by convention: UserServiceSpec.bx, OrderServiceTest.bx.

// tests/specs/UserServiceSpec.bx
class extends="testbox.system.BaseSpec" {

    function run() {
        describe( "UserService", () => {
            it( "can greet a user", () => {
                expect( "hello" ).toBe( "hello" )
            } )
        } )
    }

}

Extending testbox.system.BaseSpec is optional but recommended: it enables IDE introspection, direct web runner execution, and faster test loading.

Injected Variables (always available in bundles)

BDD Style

Basic Structure

class extends="testbox.system.BaseSpec" {

    function run() {

        describe( "Calculator", () => {

            it( "can add two numbers", () => {
                var calc = new Calculator()
                expect( calc.add( 1, 2 ) ).toBe( 3 )
            } )

            it( "throws on divide by zero", () => {
                expect( () => new Calculator().divide( 10, 0 ) ).toThrow()
            } )

        } )

    }

}

Suite Functions (all equivalent blocks)

feature( "User authentication", () => {
    scenario( "valid credentials", () => {
        given( "a registered user", () => {
            when( "they log in with correct password", () => {
                then( "they should receive a JWT token", () => {
                    var token = authService.login( "[email protected]", "secret" )
                    expect( token ).notToBeEmpty()
                } )
            } )
        } )
    } )
} )

Spec Functions

// it() is the primary spec block
it( "does something", () => {
    expect( true ).toBeTrue()
} )

// then() is an alias for it(): title arg is "then" instead of "title"
then( "something should happen", () => {
    expect( result ).notToBeNull()
} )

Life-Cycle Methods (BDD)

Life-cycle methods control setup and teardown around specs.

Global (whole bundle)

class extends="testbox.system.BaseSpec" {

    // Runs ONCE before all specs in the bundle
    function beforeAll() {
        variables.db = new Database()
        variables.db.connect()
    }

    // Runs ONCE after all specs in the bundle
    function afterAll() {
        variables.db.disconnect()
        directoryDelete( "/tests/tmp", true )
    }

    function run() { ... }

}

Suite-Level (BDD: beforeEach / afterEach / aroundEach)

describe( "UserService", () => {

    beforeEach( ( currentSpec, data ) => {
        variables.service = new UserService()
    } )

    afterEach( ( currentSpec, data ) => {
        variables.service = javaCast( "null", "" )
    } )

    aroundEach( ( spec, suite, data ) => {
        // wraps the spec; you MUST call spec.body() yourself
        transaction action="begin" {
            arguments.spec.body()
            transaction action="rollback"
        }
    } )

    it( "can find a user by id", () => {
        var user = variables.service.findById( 1 )
        expect( user ).toHaveKey( "name" )
    } )

} )

beforeEach walks down the nesting tree (outer → inner). afterEach walks up the tree (inner → outer).

Life-Cycle Annotations (for parent classes)

Annotate any function to hook into the life-cycle without overriding named methods. Useful for base test classes:

class extends="testbox.system.BaseSpec" {

    /**
     * @beforeAll
     */
    function initAppContext() {
        // Called once before all specs; discovered up the inheritance chain
    }

    /**
     * @aroundEach
     */
    function wrapInTransaction( spec, suite ) {
        transaction action="begin" {
            try {
                arguments.spec.body()
            } finally {
                transaction action="rollback"
            }
        }
    }

}

Life-Cycle Data Binding

Pass data to life-cycle methods so dynamic values are captured correctly in loops:

var fixtures = [ "admin", "guest", "editor" ]

for ( var role in fixtures ) {
    describe( "Role: #role#", () => {

        beforeEach(
            data = { roleName = role },
            body = ( currentSpec, data ) => {
                variables.user = userFactory.create( data.roleName )
            }
        )

        it(
            title = "has correct permissions",
            data  = { role = role },
            body  = ( data ) => {
                expect( variables.user.getRole() ).toBe( data.role )
            }
        )

    } )
}

xUnit Style

@DisplayName "OrderService Tests"
class extends="testbox.system.BaseSpec" {

    // Runs ONCE before all tests
    function beforeTests() {
        application.dbPool = createDBPool()
    }

    // Runs ONCE after all tests
    function afterTests() {
        structClear( application )
    }

    // Runs before EACH test method
    function setup() {
        variables.service = new OrderService()
    }

    // Runs after EACH test method
    function tearDown() {
        variables.service = javaCast( "null", "" )
    }

    // Any function with "test" prefix is a test
    function testCalculateTotalWithDiscount() {
        var total = variables.service.calculateTotal( items=[], discount=0.1 )
        $assert.isEqual( total, 0 )
    }

    // Or use @test annotation for any naming
    @DisplayName "It should reject empty orders"
    @test
    function itRejectsEmptyOrders() {
        expect( () => variables.service.place( [] ) ).toThrow( type="EmptyOrderException" )
    }

    // Skip a specific test
    function testSkipped() skip {
        $assert.fail( "should not run" )
    }

}

Expectations (Fluent API)

Use expect( actual ) to start a fluent assertion chain. Prefix any matcher with not for the negative version.

Common Matchers

// Equality
expect( result ).toBe( expected )
expect( result ).notToBe( expected )
expect( "Hello" ).toBeWithCase( "Hello" )

// Boolean
expect( isActive ).toBeTrue()
expect( isDeleted ).toBeFalse()

// Null
expect( value ).toBeNull()
expect( value ).notToBeNull()

// Type checks
expect( items ).toBeArray()
expect( config ).toBeStruct()
expect( id ).toBeNumeric()
expect( dob ).toBeDate()
expect( obj ).toBeComponent()
expect( obj ).toBeInstanceOf( "com.example.MyClass" )
expect( obj ).toBeTypeOf( "uuid" )         // uses isValid()

// Size / emptiness
expect( [] ).toBeEmpty()
expect( items ).notToBeEmpty()
expect( items ).toHaveLength( 3 )
expect( "Hello" ).toHaveLength( 5 )

// Struct keys
expect( user ).toHaveKey( "name" )
expect( config ).toHaveDeepKey( "database.host" )

// Strings / arrays
expect( message ).toMatch( "^Error" )         // regex, case-insensitive
expect( message ).toMatchWithCase( "ERROR" )  // regex, case-sensitive
expect( items ).toInclude( "admin" )          // case-insensitive
expect( items ).toIncludeWithCase( "Admin" )  // case-sensitive

// Numbers / dates
expect( value ).toBeGT( 0 )
expect( value ).toBeGTE( 1 )
expect( value ).toBeLT( 100 )
expect( value ).toBeLTE( 99 )
expect( value ).toBeBetween( 1, 100 )
expect( score ).toBeCloseTo( 3.14, 0.01 )

// Exceptions (ALWAYS use a closure)
expect( () => service.doThing() ).toThrow()
expect( () => service.doThing() ).toThrow( type="MyException" )
expect( () => service.doThing() ).toThrow( regex="not found" )
expect( () => service.doThing() ).toThrow( type="MyException", regex="not found" )

Chaining

// Multiple matchers on the same value
expect( result )
    .notToBeNull()
    .toBeStruct()
    .toHaveKey( "id" )
    .toHaveKey( "name" )

Collection Expectations (expectAll)

Test every element of an array or struct at once:

expectAll( users ).toHaveKey( "name" )
expectAll( scores ).toBeGT( 0 )
expectAll( [ true, true, true ] ).toBeTrue()
expectAll( { a: 2, b: 4, c: 6 } ).toSatisfy( (x) -> x mod 2 == 0 )

Exception Testing Patterns

// Preferred: closure-based
it( "throws on invalid input", () => {
    expect( () => {
        service.process( javaCast( "null", "" ) )
    } ).toThrow( type="InvalidArgumentException" )
} )

// xUnit annotation style
function testThrowsOnNull() expectedException="InvalidArgumentException" {
    service.process( javaCast( "null", "" ) )
}

Assertions Library ($assert)

The traditional assertions library available via $assert in every bundle:

$assert.isTrue( condition, [message] )
$assert.isFalse( condition, [message] )
$assert.isEqual( expected, actual, [message] )
$assert.isNotEqual( expected, actual, [message] )
$assert.isNull( value, [message] )
$assert.isNotNull( value, [message] )
$assert.isEmpty( value, [message] )
$assert.isNotEmpty( value, [message] )
$assert.includes( collection, needle )
$assert.notIncludes( collection, needle )
$assert.includesWithCase( collection, needle )
$assert.throws( closure, [type], [regex], [message] )
$assert.fail( [message] )

// Dynamic assertion methods (BoxLang only)
assertIsTrue( condition )
assertIsEqual( expected, actual )
assertIsNull( value )
// Any $assert method can be called as assert{MethodName}()

Skipping and Focusing

Skip Specs/Suites

Prefix any suite/spec function with x OR use the skip argument:

// Prefix approach: skip entire suite
xdescribe( "Slow integration tests", () => { ... } )

// Prefix approach: skip single spec
xit( "is temporarily disabled", () => { ... } )

// skip argument: boolean or closure
it( title="runs only on production", skip=!isProduction(), body=() => { ... } )

describe( title="Lucee only", skip=() => !structKeyExists( server, "lucee" ), body=() => {
    it( "uses Lucee ORM", () => { ... } )
} )

// skip() method inside spec: programmatic
it( "can run on multiple engines", () => {
    if ( !server.keyExists( "boxlang" ) ) {
        skip( "BoxLang-only feature" )
    }
    // ... rest of spec
} )

Focus Specs/Suites

Prefix with f or use focused=true: ONLY focused specs/suites run:

// Focus a single spec
fit( "is the only spec that runs", () => {
    expect( true ).toBeTrue()
} )

// Focus a suite (all children run)
fdescribe( "Critical path", () => {
    it( "step one", () => { ... } )
    it( "step two", () => { ... } )
} )

Warning: Never commit focused tests: CI will only run those specs.

Mocking with MockBox

MockBox is built into TestBox. Helper methods are available in every bundle.

Creating Mocks

// Create a mock of a class (real methods unless overridden)
var mockService = createMock( "services.UserService" )

// Create an empty mock (ALL methods removed: you mock everything)
var mockRepo = createEmptyMock( "repositories.UserRepository" )

// Decorate an existing instance with mocking capabilities
var realService = new services.UserService()
prepareMock( realService )

// Chainable setup
var mockUser = createMock( "model.User" )
    .$( "isActive", true )
    .$( "getRole", "admin" )
    .$( "getName", "Luis Majano" )

Mocking Methods: $()

// Return a single value
mockRepo.$( "findById", { id: 1, name: "Luis" } )

// Return nothing (void)
mockRepo.$( "deleteById" )

// Throw an exception
mockRepo.$( "findById",
    throwException = true,
    throwType       = "NotFoundException",
    throwMessage    = "Record not found"
)

// Use a callback for dynamic responses
mockRepo.$( method="findById", callback=( id ) => {
    if ( id == 1 ) return { id: 1, name: "Luis" }
    return javaCast( "null", "" )
} )

Sequenced Return Values: $results()

Returns values in sequence, cycling when exhausted:

// First call → true, second call → false, third → true (repeats)
mockSession.$( "isLoggedIn" ).$results( true, false )

expect( mockSession.isLoggedIn() ).toBeTrue()   // 1st call
expect( mockSession.isLoggedIn() ).toBeFalse()  // 2nd call
expect( mockSession.isLoggedIn() ).toBeTrue()   // 3rd call (cycles)

Argument-Specific Mocking: $args()

Return different values based on what arguments were passed:

mockCache.$( "get" )
    .$args( "user-1" ).$results( { id: 1, name: "Luis" } )
    .$args( "user-2" ).$results( { id: 2, name: "Alexia" } )
    .$args( "missing" ).$results( javaCast( "null", "" ) )

Property Mocking: $property()

// Mock a property on the object
mockService.$property( "timeout", "variables", 5000 )
mockService.$property( "cache", "instance", mockCache )

Query Simulation: $querySim()

var mockQuery = mockDAO.$querySim(
    "id, name, email
     1  | Luis Majano   | [email protected]
     2  | Alexia Majano | [email protected]"
)
mockDAO.$( "getUsers", mockQuery )

Verification Methods

Check how and how many times mocked methods were called:

// Get call count
mockRepo.$count( "findById" )   // calls to findById specifically
mockRepo.$count()               // total calls to all mocked methods

// Assert call counts
expect( mockRepo.$times( 1, "findById" ) ).toBeTrue()  // called exactly once
expect( mockRepo.$once( "save" ) ).toBeTrue()          // alias for $times(1)
expect( mockRepo.$never( "delete" ) ).toBeTrue()       // never called
expect( mockRepo.$atLeast( 2, "findById" ) ).toBeTrue()
expect( mockRepo.$atMost( 3, "findById" ) ).toBeTrue()

// Inspect call log (arguments per call)
var log = mockRepo.$callLog()
expect( log.findById[ 1 ].id ).toBe( 42 )  // 1st call, named arg "id"

// Reset all mock state
mockRepo.$reset()

Full Mocking Example

class extends="testbox.system.BaseSpec" {

    function beforeAll() {
        // Create service under test with mocked dependencies
        variables.mockUserDAO   = createEmptyMock( "repositories.UserDAO" )
        variables.mockEmailSvc  = createEmptyMock( "services.EmailService" )
        variables.service       = createMock( "services.UserService" )
        variables.service.init( variables.mockUserDAO, variables.mockEmailSvc )
    }

    function run() {

        describe( "UserService.register()", () => {

            beforeEach( () => {
                variables.mockUserDAO.$reset()
                variables.mockEmailSvc.$reset()
            } )

            it( "creates user and sends welcome email", () => {
                var newUser = { id: 99, email: "[email protected]", name: "Test" }
                variables.mockUserDAO.$( "create", newUser )
                variables.mockEmailSvc.$( "sendWelcome" )

                var result = variables.service.register( "[email protected]", "Test" )

                expect( result ).toBe( newUser )
                expect( variables.mockUserDAO.$once( "create" ) ).toBeTrue()
                expect( variables.mockEmailSvc.$once( "sendWelcome" ) ).toBeTrue()
            } )

            it( "throws when email already exists", () => {
                variables.mockUserDAO.$( "existsByEmail", true )

                expect( () => variables.service.register( "[email protected]", "Test" ) )
                    .toThrow( type="DuplicateEmailException" )
            } )

        } )

    }

}

Mock Data Generation (mockData())

Available in all bundles via the cbMockData module included with TestBox:

// Generate 5 fake user records
var users = mockData(
    $num         = 5,
    id           = "autoincrement",
    name         = "name",
    email        = "email",
    age          = "age",
    isActive     = "boolean",
    createdDate  = "datetime",
    bio          = "lorem:2"      // 2 paragraphs of lorem ipsum
)

// Random count
var items = mockData( $num = "rnd:5:20", title = "name" )

// Return as struct instead of array
var single = mockData(
    $returnType = "struct",
    $num        = 1,
    id          = "uuid",
    name        = "name",
    email       = "email"
)

Available Mock Types

Async Testing

Run all specs in a suite concurrently using asyncAll=true:

describe( title="Parallel specs", asyncAll=true, body=() => {

    beforeEach( () => {
        variables.service = new MyService()
    } )

    it( "spec one", () => {
        expect( variables.service.ping() ).toBeTrue()
    } )

    it( "spec two", () => {
        expect( variables.service.version() ).notToBeEmpty()
    } )

} )

Warning: In async mode, all variables must be properly var-scoped or thread-safe. Do NOT use shared variables scope when running async specs.

Attachments, Retries and Browser Tests

Attach Files to a Spec: attach()

Every spec can attach files (screenshots, logs, dumps) to the running spec. They are kept for passed and failed specs and reporters list them (JSON report, links in the Simple report, [[ATTACHMENT|path]] lines in JUnit <system-out>, text/console/stream output under failed specs).

it( "exports the report", () => {
    var file = service.export()
    attach( file, "file", "export.csv" )   // attach( path, type = "file", name = file name of path )
    attach( logPath, "log" )
    expect( fileExists( file ) ).toBeTrue()
} )

Call it from a spec body or a beforeEach() / afterEach() / aroundEach() closure; outside a running spec it throws TestBox.InvalidContext.

Retries

A failing or erroring spec can rerun up to N more times. Each attempt reruns beforeEach(), the body and afterEach() (xUnit: setup(), the test, teardown()), and only the final attempt is recorded. Output shows "(passed after N attempts)".

// 1. Spec argument (highest precedence), also on fit() and xit()
it( title = "talks to a flaky service", retries = 2, body = () => { ... } )

// 2. Bundle annotation (BoxLang annotations go above the declaration)
@retries( 1 )
class extends="testbox.system.BaseSpec" { ... }

// xUnit: method annotation overrides the bundle annotation
@retries( 3 )
function testFlaky() { ... }

// CFML components keep inline attributes: component extends="testbox.system.BaseSpec" retries="1" { ... }

# 3. Global default (lowest precedence)
./testbox/run --retries=2

Skipped specs are never retried. Use retries for genuinely flaky I/O (browsers, networks), not to hide bugs.

Rerun Only Failures

results.getFailedTargets() returns what failed in a run, on every engine: { bundles, specs, bundleErrors } (bundle paths, spec ids, and bundles that failed outside of a spec). Run only those with the existing filters:

var failed = results.getFailedTargets()
new testbox.system.TestBox( bundles = failed.bundles ).runRaw( testBundles = failed.bundles, testSpecs = failed.specs )

• HTML reports (Simple, Min, Dot, Doc): a Run Failed (N) button next to Run All, shown only when something failed. Its link (?testBundles=...&testSpecs=...) is built from the report itself, so web runners keep no state; past 2000 characters it reruns the failed bundles whole.
• BoxLang CLI --failed: see below.

#### --failed

Every BoxLang runner run writes {reportpath}/.testbox-failed.json with the bundles and specs that failed or errored. ./testbox/run --failed reruns only those; when the file is missing or empty it prints a message and runs nothing. A bundle that failed outside of a spec (beforeAll(), afterAll()) is broken, not flaky: it goes to bundleErrors in the file and is listed, never rerun. The other bundles still rerun only their failed specs.

Playwright Failures

Playwright.AssertionFailed (thrown by bx-playwright assertions such as page.assertSee()) counts as a spec failure, like TestBox.AssertionFailed, keeping its message and detail. Other Playwright.* exceptions count as errors.

Browser Tests

For browser tests annotate the spec with @browser (or @browserProfile( "ci" ) / @baseURL( "http://..." )) above class extends="testbox.system.BaseSpec" (in ColdBox apps, coldbox.system.testing.BaseTestCase) and use browse(), the browser matchers (expect( page ).toSee( "Welcome" )) and the --web-server runner option. Load the bx-playwright-testing skill for the full API.

Running Tests

BoxLang CLI Runner

# Run all tests (default: tests/specs)
./testbox/run

# Run a specific bundle (dot notation)
./testbox/run tests.specs.UserServiceSpec

# Run all specs in a directory
./testbox/run --directory=tests.specs.unit

# Specify a bundle pattern
./testbox/run --bundles-pattern=*Spec*.bx

# Stream results in real-time (great for CI)
./testbox/run --stream

# Stop on first failure
./testbox/run --eager-failure

# Verbose output (every spec printed)
./testbox/run --verbose

# Retry failing specs, or rerun only the last failures
./testbox/run --retries=2
./testbox/run --failed

# Start the app before the tests (browser tests)
./testbox/run --web-server="boxlang-miniserver --port 8080" --web-server-url=http://localhost:8080

CLI Runner Options

CommandBox Runner

box testbox run
box testbox run bundles=tests.specs.UserServiceSpec
box testbox run --verbose

Reporters

Pick one with reporter= in the runner URL, --reporter= on the BoxLang CLI runner, or reporter = { type : "...", options : {} } in code.

HTML reporters (Simple, Min, Dot, Doc)

• A verdict banner first (counts, proportion bar, status filters), light, dark and system themes, F jumps to the next failure, / searches.
• Run links on every bundle, suite and spec, Run All, and Run Failed (N) when something failed (built from the report, no state, see Rerun Only Failures).
• ?editor=vscode (default) and other editors turn failures and stack frames into open-in-editor links.
• Ask AI on every failure: copy or preview a prompt, open it in ChatGPT or Claude, or copy it as Agent JSON. Nothing leaves the page until a provider is clicked.

reporter = {
    type    : "testbox.system.reports.SimpleReporter",
    options : {
        aiAssist       : true,   // false (or ?aiAssist=false) removes Ask AI
        aiContextLines : 5,      // code lines around the failing line
        aiStackFrames  : 8,
        aiProviders    : [ { id : "acme", name : "Acme AI", url : "https://ai.acme.test/?p={prompt}" } ],
        aiPrompt       : "{intro} {spec} ({status}) {message} {code} {stack} Rerun: {rerun}",
        urlParams      : { editor : "idea" }   // request params for reports produced from code (no url scope)
    }
}

Agent reporter (reporter=agent)

One minified JSON line: {"ok":false,"totals":{"pass","fail","error","skipped","specs","ms"},"failures":[{"bundle","spec","status","message","at"}],"truncated":0}. Branch on ok; the exit code is not changed. Options: detail (summary, failures, all), maxFailures (20), maxMessageLength (300), includeStack (false), stackDepth (3), includeSkipped, includeDebug. Prefer it when an agent runs the tests: ./testbox/run --reporter=agent.

Test Harness Layout

tests/
├── specs/
│   ├── unit/
│   │   ├── UserServiceSpec.bx
│   │   └── OrderServiceSpec.bx
│   ├── integration/
│   │   └── UserFlowSpec.bx
│   └── resources/
│       └── fixtures.json
├── Application.bx        ← configures mappings for tests
└── runner.bxm            ← web runner template

Minimal Application.bx for Tests

class {
    this.name              = "MyAppTests"
    this.mappings[ "/testbox" ] = expandPath( "/testbox" )
    this.mappings[ "/app" ]     = expandPath( "../src" )
}

Common Patterns

Testing Classes with Dependencies (Constructor Injection)

function beforeAll() {
    variables.mockRepo    = createEmptyMock( "UserRepository" )
    variables.mockMailer  = createEmptyMock( "Mailer" )
    variables.service     = new UserService( variables.mockRepo, variables.mockMailer )
}

Testing Private Methods via prepareMock

it( "validates email format internally", () => {
    var svc = prepareMock( new UserService() )
    // expose private method for targeted testing
    expect( svc.validateEmail( "bad-email" ) ).toBeFalse()
    expect( svc.validateEmail( "[email protected]" ) ).toBeTrue()
} )

Data-Driven Specs

var validInputs = [ "hello", "world", "boxlang" ]

for ( var word in validInputs ) {
    it(
        title = "validates '#word#' as a valid slug",
        data  = { word = word },
        body  = ( data ) => {
            expect( SlugValidator.isValid( data.word ) ).toBeTrue()
        }
    )
}

Testing Query Results

it( "returns paginated user query", () => {
    var mockQ = mockDAO.$querySim(
        "id, name
         1  | Luis
         2  | Alexia"
    )
    mockDAO.$( "paginate", mockQ )

    var result = variables.service.getPage( 1, 10 )
    expect( result ).toBeQuery()
    expect( result.recordCount ).toBe( 2 )
} )

Resetting Mocks Between Tests

Always reset mocks in beforeEach to avoid state bleed:

beforeEach( () => {
    variables.mockRepo.$reset()
    variables.mockMailer.$reset()
} )

Anti-Patterns to Avoid