Skip to main content

Character Count Match (char_ct_match)

Contents​

Metric description​

Character count match counts characters in the output with filters for whitespace, punctuation, digits, optional regex exclusions, uppercase-only mode, and optional min and max total counts.

How to interpret the score​

  • 100: the count satisfies configured bounds (or none set).
  • 0: outside bounds or invalid for this check.

API usage​

Prerequisites

After the environment variables are configured, the next step is to create a JSON payload for the custom-runs request. For a field-by-field description of the payload (top-level keys, evaluations, and each row in data), see Custom run request body.

Shortname: char_ct_match

Default threshold: 100

Structural metrics run without an LLM (deterministic checks). Your run may still include model_slug where the API expects it; scoring does not depend on it for this category.

Inputs (each object in data)

  • output (str, required): Text to measure.

metric_args

  • min_count (number optional): Minimum number of characters required.

  • max_count (number optional): Maximum number of characters allowed.

  • include_whitespace (boolean optional): Include whitespace in the count. Default: true.

  • include_punctuation (boolean optional): Include punctuation in the count. Default: true.

  • include_digits (boolean optional): Include digits in the count. Default: true.

  • custom_exclude_pattern (string optional): Regex pattern of characters to exclude from the count.

  • only_uppercase (boolean optional): Count only uppercase letters. Default: false.

Eval metadata​

Structural metrics do not populate eval_metadata; the field is omitted or ull on the result object.

Example

import json
import os

import requests
from dotenv import load_dotenv

load_dotenv(override=True)

_API_KEY = os.getenv("AEGIS_API_KEY")
_BASE_URL = os.getenv("AEGIS_API_BASE_URL")
_CUSTOM_RUN_URL = f"{_BASE_URL}/runs/custom"


def post_custom_run(payload: dict) -> requests.Response:
    """POST JSON payload to Aegis custom runs; returns the raw response."""
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {_API_KEY}",
    }
    return requests.post(
        _CUSTOM_RUN_URL,
        headers=headers,
        data=json.dumps(payload),
    )


if __name__ == "__main__":
    data = [
        {"output": "Hello"}
    ]

    payload = {
        "threshold": 100,
        "model_slug": "o4-mini",
        "is_blocking": True,
        "data_collection_id": None,
        "evaluations": [
            {
                "metrics": [
                    {
                        "metric": "char_ct_match",
                        "metric_args": {
                            "min_count": 1,
                            "max_count": 1000,
                            "include_whitespace": True,
                        },
                    },
                ],
                "threshold": 100,
                "model_slug": "o4-mini",
                "data": data,
            }
        ],
    }

    response = post_custom_run(payload)
    response.raise_for_status()
    print(json.dumps(response.json(), indent=2))