Labs Experiments

Labs is where you try out a new version of a decision tree before (or while) it affects real applicants. You pick the trees to test, choose the versions to compare, and LendAPI records how each version decides. Labs offers three kinds of test, and each answers a different question:

Key concepts

• Test: one Labs experiment. It has a name, a type, variants and a time window.
• Variant: a pair of versions for one decision tree (a Rule in the form): the version you have today and the version you are testing.
• Champion / Challenger: in an A/B test, Variant A (usually your current version) is the champion and Variant B is the challenger.
• Default: in shadow tests and backtests, the version used for comparison, normally the live version.
• Outcome distribution: how many applications each version sent to Approve, Decline, Review and Request Docs.

How Labs works

Choosing a test type

A typical rollout uses all three: backtest first, then shadow test, then an A/B test with a small challenger share, then set the new version live in Rules Studio.

How each test runs

stateDiagram-v2
    [*] --> NotStarted: Create test
    NotStarted --> NotStarted: Configure and Save
    NotStarted --> Running: Run
    Running --> Completed: Window ends / count or sample reached
    Running --> Completed: End Test
    Completed --> [*]

• Backtest: LendAPI draws a sample of applications decided by the selected trees between the start and end dates, re-runs each one with the backtest version, and stores the result next to the original decision.
• Shadow test: While the test window is open, every application that runs one of the selected trees is also run, in the background, with the shadow version. The shadow result is stored with the test and does not replace the application's decision.
• A/B test: Each new application that runs a selected tree is randomly assigned to Variant A or B according to the size distribution, and the assigned variant's version is run for it and recorded for that variant. An application keeps its variant for the rest of the test. With Count of Applications, the test completes once that many applications have been assigned.

🚧

Tests can call integrations and run pricing

Labs runs the tested tree versions against real applications the same way a live decision runs them. If a tested version has an integration partner that has not already been called for an application, LendAPI calls that integration, and your provider may bill for it. If a tested version has a pricing engine attached, that engine runs and its results are saved to the application. Check the integration partners and pricing engine on each version before you start a test.

❗️

Tests can't be edited once started

Once a test starts, it cannot be edited, and once ended it cannot be resumed. Results collected up to that point are kept.

Use Labs

Create and run a test

1. Go to Tools → Labs and click + Test.
2. In New Test, enter a Name, an optional Description, and choose the Type: A/B Test, Backtest or Shadow Test.
3. On the test page, open Variants and configure one row per decision tree:
   • Rule: the decision tree to test.
   • Variant A / Default: the version you have today. For shadow tests and backtests this is filled in with the tree's live version.
   • Variant B / Shadow / Backtest: the version you are testing.
4. Fill in the settings for the test type:
   • A/B test: set Variant A and Variant B under Size Distribution (they must add up to 100%; the default is 90/10). Under Run Time, choose Fixed Date Range (pick Start Date and End Date) or Count of Applications (enter the number of applications).
   • Shadow test: pick a Start Date and End Date. Dates cannot be in the past.
   • Backtest: set the Sample Size, then a Start Date and End Date for the historical window.
5. Click Save.
6. Click Run, then confirm Run Test.

While a test is running, the header shows when it ends. Click End Test to stop it early.

Read the results

Open a test from the Labs list. Completed tests show a View Results action.

• Outcome distribution: a chart (or list) comparing how many applications each version sent to Approve, Decline, Review and Request Docs. Look for shifts, for example more approvals or fewer manual reviews with the challenger.
• Per-application results: click View on an application to open Underwriting Results. For shadow tests and backtests it shows the CURRENT RESULT next to the SHADOW TEST RESULT or BACKTEST RESULT, including the Tree Result, Time of run, Decision Tree Name and Version, the Outcome, and any Next Decision Tree in the chain.

When reading results:

1. Check volume first. A difference across 20 applications is noise; wait for enough applications before drawing conclusions.
2. Review the disagreements. Open applications where the two versions disagree and confirm the new behavior is what you intended.
3. Look beyond approval rate. For A/B tests, compare downstream results such as conversion and performance, not just the decision mix.
4. Act in Rules Studio. Labs never changes your live version. When you're satisfied, set the tested version live in Decision Trees.

API access

Labs is managed in the dashboard. There are no public API endpoints for creating, running or reading Labs tests.

Access and best practices

Permissions

Access is controlled by the Labs module. View access lets a user see tests and results; Edit access is needed to create, save, run and end tests. See Roles and Permissions.

Best practices

• Test one change at a time. If the challenger changes several rules, you won't know which one moved the results.
• Start A/B tests small. Give the challenger a small share (the default 10%) and raise it once early results look healthy.
• Avoid overlapping tests on the same decision tree, so each application's results are easy to interpret.
• Pick representative windows. Backtest over a period with normal volume, not a holiday spike or a marketing campaign.
• Submit the challenger version first. A version exists only after it is submitted in Rules Studio, so finish and submit your changes before configuring the test.

Related guides