GitLab CI

Run CLIWatch benchmarks in your GitLab CI pipeline.

Basic Pipeline

Create .gitlab-ci.yml:

benchmark:
  image: node:22
  stage: test
  script:
    - npm install -g @cliwatch/cli-bench
    - cli-bench
  variables:
    AI_GATEWAY_API_KEY: $AI_GATEWAY_API_KEY
    CLIWATCH_API_KEY: $CLIWATCH_API_KEY
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

Environment Variables

Add variables in Settings → CI/CD → Variables:

• Mark API keys as masked and protected

Variable	Required	Description
CLIWATCH_API_KEY	Yes	Your CLIWatch API key
AI_GATEWAY_API_KEY	Yes	Vercel AI Gateway key (provides access to all models)

All model calls go through the Vercel AI Gateway, so you only need one API key for all providers.

MR Comments

MR (merge request) comments for GitLab are not yet available. We currently support PR comments for GitHub Actions only. If you are interested in GitLab MR comments, contact us.

Tips

• Use upload: auto (default): results upload when CLIWATCH_API_KEY is set
• Set concurrency: 1 if your CLI modifies shared state