Skip to main content

GeminiAgent DetoxIO Integration Guide

This guide shows how to evaluate Google Gemini models using DetoxIO — via CLI or the Python SDK.

Prerequisites​

  • dtx installed (CLI and/or SDK)
  • A Google AI Studio API key
  • Environment variables set:
export GEMINI_API_KEY=your_gemini_api_key
# Optional: defaults to generativelanguage.googleapis.com
# export GEMINI_API_BASE=https://generativelanguage.googleapis.com

Tip: Use any supported Gemini model ID, for example gemini-2.0-flash, gemini-1.5-pro, or gemini-2.5-pro.

Quick Sanity Check (curl)​

Verify your key and model ID with the Gemini REST API:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent" \
  -H 'Content-Type: application/json' \
  -H "X-goog-api-key: $GEMINI_API_KEY" \
  -X POST \
  -d '{
    "contents": [
      {
        "parts": [
          { "text": "Explain how AI works in a few words" }
        ]
      }
    ]
  }'

Run with CLI​

dtx redteam run \
  --agent gemini \
  --url gemini-2.5-pro \
  --dataset stingray \
  --max_prompts 30 \
  --html report_gemini.html

Key flags:

  • --agent gemini: use the Gemini provider
  • --url <model-id>: the Gemini model ID (e.g. gemini-2.5-pro)
  • --dataset: which dataset module to use (e.g. stingray)
  • --max_prompts: cap prompts generated for the run
  • --html: save an HTML report

Run with Python SDK​

from dtx.sdk.runner import DtxRunner, DtxRunnerConfigBuilder
from dtx_models.providers.base import ProviderType

cfg = (
    DtxRunnerConfigBuilder()
      .agent_from_provider(
          ProviderType.GEMINI,
          "gemini-2.5-pro",
          load_env_vars=False,
          env_vars={
              "api_key": "your_gemini_api_key", # required
            #   "endpoint": "https://generativelanguage.googleapis.com" # optional
          }
      )
      .max_prompts(5)
      .build()
)

report = DtxRunner(cfg).run()
print(report.model_dump_json(indent=2))

Passing api_base and api_key inline avoids relying on environment variables. You provide the model ID only (e.g., "gemini-2.5-pro"); DetoxIO handles provider prefixing internally.

Troubleshooting​

  • Internal errors (e.g., VertexAIException or INTERNAL) can occur transiently or if a model isn’t enabled for your key/region. Try a stable model (e.g., gemini-1.5-pro or gemini-2.0-flash) and retry.
  • Ensure you’re using a Google AI Studio API key for generativelanguage.googleapis.com. If you use Vertex AI endpoints, ensure the base URL and credentials match that deployment.
  • Base URL format: api_base should be just the host, e.g. https://generativelanguage.googleapis.com (do not include /v1 or :generateContent).
  • Confirm quota and billing are active in your Google account if requests are rejected.
  • If the CLI fails without a dataset, provide --dataset explicitly (e.g., --dataset stingray).