Experiments

This guide describes functionality that is currently available as a preview feature. If you’d like early access to this feature, please contact us.

Experiments let you compare how your model performs across different scenarios and datasets in a single run. Instead of manually running and comparing individual datasets, you define the variables up front, and the platform runs all combinations automatically. The result is a comparison report where each column represents a scenario, showing aggregated metrics across all the datasets you ran it on.

When to use Experiments

Experiments support three main use cases, and they’re not mutually exclusive:

Goal alignment

Compare different configuration profiles to see which constraint weight setup produces the best outcomes for your business goals. Run the same datasets against multiple profiles and compare the differences side by side.

Learn more: Balancing different optimization goals.

What-if scenarios

Modify input data to understand how your model performs under different conditions. For example, add 20% more workload, change shift counts, or remove a resource. Each variant becomes a dataset in the experiment, and each setup you want to test is a scenario.

One common starting point is an inefficiency you’ve identified in your current plan. Once you know what to change, an experiment lets you validate whether that change actually improves outcomes before applying it to production.

Benchmarking

Compare different model versions, solver configurations, or constraint implementations to validate changes before promoting them to production. Experiments give you reproducible, side-by-side results across the same inputs.

Experiments are complementary to the Comparison UI and Insights. Use Experiments when you want to systematically run and compare new combinations of data and configuration that don’t yet exist as saved datasets. Use the Comparison UI for ad hoc comparisons of datasets you’ve already run. Use Insights to track trends over time.

Core concepts

Before creating an experiment, it helps to understand the two building blocks:

Datasets: The input data you want to solve. An experiment references existing datasets from your model by their dataset ID. If you want to use new input data, first submit it (without solving) as a dataset via the model API, then reference it in the experiment.
Scenarios: A scenario defines how to solve those datasets: which model version and which configuration profile (including constraint weights, termination settings, and other parameters) to use. Each scenario can also include specific parameter overrides on top of a base profile.

An experiment runs every dataset against every scenario, creating M × N solves (M datasets × N scenarios). The results are then presented in a comparison report with one column per scenario.

Creating an experiment

Experiments are created with the experiments API endpoint of the Model API, the same API your model uses for submitting and querying datasets. The UI shows created experiments, but submission happens through the API. This makes it straightforward to trigger experiments from a local environment or a CI pipeline.

Authentication

The experiments API uses model API key authentication, the same as other model API operations. See Model API usage for details on generating API keys and configuring permissions.

The full API reference is available in the interactive API reference. Look for endpoints under /api/models/experiments in the "model" definition.

Submitting a create request

Send a POST request with the experiment name, datasets, and one or more scenarios.

curl -X POST "https://app.timefold.ai/api/models/experiments/{modelId}?operation=SOLVE" \
  -H "X-API-Key: <your-model-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Shift weight comparison",
    "scenarios": [
      {
        "name": "Baseline",
        "modelVersion": "1.0",
        "configurationProfileId": "<baseline-profile-id>",
        "datasets": [
          { "originDatasetId": "<dataset-id-1>" },
          { "originDatasetId": "<dataset-id-2>" }
        ]
      },
      {
        "name": "Higher overtime weight",
        "modelVersion": "1.0",
        "configurationProfileId": "<overtime-profile-id>",
        "datasets": [
          { "originDatasetId": "<dataset-id-1>" },
          { "originDatasetId": "<dataset-id-2>" }
        ]
      }
    ]
  }'

The response returns the created experiment, including its id and the status of each dataset and scenario combination.

Running an experiment

Add operation=SOLVE to start all solves immediately; without it, the experiment is created in CREATED status and you can trigger it later from the UI or via a separate API call.

Experiments run on a dedicated solve queue with a separate pool of resources. This means running an experiment never competes with or delays your production solves; you can safely run large experiments without affecting live workloads.

Figure 1. Experiment UI showing a running experiment

Reading the results

When the experiment completes, the results are displayed as a comparison table (one column per scenario) with metrics aggregated across all datasets and delta values shown relative to your baseline scenario. This works similarly to comparing runs, but scoped to the scenarios you defined in the experiment. From the results view you can also add scenarios, re-run the experiment, or replicate it with different settings.

Figure 2. Experiment results showing multiple scenarios compared side by side

Behavior and constraints

Datasets created by an experiment are regular datasets. They show up in your model’s dataset list and can be combined with existing datasets in other comparisons.
Experiment datasets are tagged with the experiment ID and scenario reference, making them easy to filter in and out of other views. See Datasets overview for more on using tags.
Experiments are scoped to a single tenant and a single model.