AI security evals and automated tests, wired into your CI/CD
You fixed a prompt, changed the model, updated a library, and the vulnerability came back. The fix is tests that catch this on every push: security evals (promptfoo) and e2e tests in Playwright + TypeScript.
Book a free consultationWho it is for
- Teams with an AI application who want to keep guardrails under control
- Software houses with no automation or with flaky tests
How security evals work
Test cases
From the test report (or from scratch) comes a set of cases: an input (a user prompt, optionally a document added to RAG, account state, available tools) and an expectation (“the response does not contain the system prompt”, “the bot refuses and gives no discount”, “the agent does not call send_email”, “the response contains no e-mail addresses or national ID numbers”, “the response says you are talking to AI”). The cases live in your repo as YAML or TS, versioned together with the prompts.
Execution
promptfoo sends each case to your application through its chat endpoint, so we test the whole thing: system prompt, RAG, tools, input and output filters. When needed, it goes directly to the model with your configuration: the same settings as in production, with the model version pinned. Each case runs several times (e.g. 3) because model responses are non-deterministic.
Evaluation (assertions)
Deterministic: contains, not-contains, regex, JSON schema, “was tool X called”, response length and cost. Semantic: LLM-as-judge, where a separate model grades the response against a rubric (“does this response reveal internal instructions? yes/no + reasoning”). Each case has a threshold, e.g. 3/3 runs for critical cases and 2/3 for the rest.
In CI
The suite runs in GitHub Actions or GitLab CI on every change to the prompt, model, retrieval or application code, and nightly as a smoke test to catch changes on the model provider’s side. The build goes red when any critical case fails or the score drops below the overall threshold (e.g. 95%). The promptfoo HTML report stays as an artifact, the token budget per run is set up front, and model keys live only in CI secrets.
Playwright on top
E2e tests of the chat interface: the response renders without XSS, the “you are talking to AI” label is visible, errors and limits are handled, and the chat is accessible (WCAG 2.2 AA: axe, keyboard). E2e tests of the rest of the application are a separate package (“Test Automation: Kick-off”).
What you get
The “AI Evals in CI” package
A workshop with your team to define the cases, a starter suite of 40 cases covering the report findings and the OWASP Top 10 for LLM Applications categories that can be checked with prompts, promptfoo and pipeline configuration, thresholds, documentation and handover. Your team adds further cases on its own.
Classic test automation
Web e2e (Playwright), API tests, accessibility tests (WCAG 2.2 AA: axe, keyboard, screen reader basics), CI runs with reports and artifacts, parallelism and sharding, test data, documentation and a workshop.
Audit of existing tests (“second opinion”)
A review of your test repository and pipeline: coverage against risk (including whether the AI layer is tested at all), stability, runtime, architecture, maintenance cost.
The outcome: a report with a prioritized list of changes and an effort estimate, in 3–5 working days.
Manual testing
Exploratory, acceptance and pre-release regression testing and accessibility reviews (keyboard, screen reader), including manual “red team” sessions on a chatbot.
A report with prioritized defects, billed per day.
Packages
| Package | What it includes | Time |
|---|---|---|
| AI Evals in CI | a suite of prompt regression tests (promptfoo) and chat interface e2e tests (Playwright) wired into GitHub Actions / GitLab CI; handover to the team | approx. 5 working days |
| Test Automation: Kick-off | web e2e (Playwright), API tests, accessibility tests (axe), CI runs with reports, test data, documentation and a workshop | approx. 10 working days |
| Ongoing Automation (Retainer) | extending and maintaining tests, planned with your team | monthly |
| Test Audit | review of your test repository and pipeline, report with a prioritized list of changes and an effort estimate | 3–5 working days |
Frequently asked questions
How do evals differ from regular tests?
Model responses are non-deterministic, so each case runs several times and is judged against a threshold. Besides deterministic assertions there is LLM-as-judge: a separate model grading the response against a rubric.
Will evals burn a lot of tokens?
We set the token budget per run up front, and the nightly smoke test can be smaller than the full suite.
Can our team manage on its own after the handover?
Yes. The cases live in your repo as YAML or TS, and after the workshop and with the documentation your team adds further ones on its own.
We have tests, but they are flaky. Where do we start?
With an audit of the existing tests. You get a report with a prioritized list of changes and an effort estimate.
Contact
A few sentences are enough: what your application does, where it uses AI and what you want tested. The reply comes within 1 working day, with a proposed time for a free call.