Skip to main content

Hugging Face Models

Use dtx as a Python library to evaluate Hugging Face models with just a few lines of code.

Text Generation Example​

from dtx.sdk.runner import DtxRunner, DtxRunnerConfigBuilder
from dtx_models.providers.base import ProviderType

cfg = (
    DtxRunnerConfigBuilder()
    .agent_from_provider(ProviderType.HF, "HuggingFaceTB/SmolLM2-135M-Instruct")
    .max_prompts(5)
    .build()
)

report = DtxRunner(cfg).run()
print(report.model_dump_json(indent=2))

Text Classification Example (Toxicity Evaluation)​

from dtx.sdk.runner import DtxRunner, DtxRunnerConfigBuilder
from dtx_models.providers.base import ProviderType
from dtx_models.evaluator import EvaluatorInScope, ModelBasedPromptEvaluation
from dtx_models.evaluator import EvaluationModelType, EvaluationModelName

evaluator = EvaluatorInScope(
    evaluation_method=ModelBasedPromptEvaluation(
        eval_model_type=EvaluationModelType.TOXICITY,
        eval_model_name=EvaluationModelName.ANY,
    )
)

cfg = (
    DtxRunnerConfigBuilder()
    .agent_from_provider(ProviderType.HF, "unitary/toxic-bert")
    .evaluator(evaluator)
    .max_prompts(5)
    .build()
)

report = DtxRunner(cfg).run()
print(report.model_dump_json(indent=2))

Example Models​

Text Generation

  • gpt2
  • arnir0/Tiny-LLM
  • HuggingFaceTB/SmolLM2-360M-Instruct

Text Classification

  • unitary/toxic-bert
  • meta-llama/Prompt-Guard-86M
  • ibm-granite/granite-guardian-hap-125m