AI Redteam Workflow

This guide shows how to manually control your red teaming workflow by generating a scope and plan, and then executing the test.

Ideal if you want to:

Step 1: Generate a Red Teaming Scope

The scope defines the target AI agent and the security objectives for testing.

dtx redteam scope "This is my dummy agent" redteam_scope.yml

Result:
Creates redteam_scope.yml, describing your dummy agent.

Tip:
You can edit redteam_scope.yml to customize objectives or testing scope!

The plan builds structured test cases based on the scope and dataset.
Here, we'll use AISafety dataset — safe to combine with evaluators.

dtx redteam plan redteam_scope.yml redteam_plan.yml --dataset aisafety

Result:
Creates redteam_plan.yml with attack scenarios and evaluation criteria based on the AISafety dataset.

🔍 Why AISafety?

Safe for evaluator pairing

Contains real-world safety and alignment scenarios

The run phase executes the plan using the ECHO dummy agent.
We'll evaluate outputs using the IBM Granite HAP 38M evaluator.

dtx redteam run --agent echo --plan redteam_plan.yml --eval ibm38

Result:

Optional: Add --max_prompts 10 to limit number of prompts.

Step	Command	Purpose
1️⃣ Scope	`dtx redteam scope "This is my dummy agent" redteam_scope.yml`	Define red teaming objectives
2️⃣ Plan	`dtx redteam plan redteam_scope.yml redteam_plan.yml --dataset aisafety`	Generate test scenarios
3️⃣ Run	`dtx redteam run --agent echo --plan redteam_plan.yml --eval ibm38`	Execute the tests

⚠️ If using garak dataset, skip --eval:

dtx redteam plan redteam_scope.yml redteam_plan.yml --dataset stingray
dtx redteam run --agent echo --plan redteam_plan.yml

For real models, switch echo to:

dtx redteam run --agent hf_model --url arnir0/Tiny-LLM --plan redteam_plan.yml --eval ibm38

Customize your plan! You can manually edit redteam_plan.yml for advanced scenarios.