How checks are run

What we check

These checks cover specific behaviors of frontier AI models, using published benchmarks:

• Refusal behavior: Whether models appropriately refuse clearly harmful requests (e.g., instructions for illegal activities, dangerous content) while remaining helpful for benign requests
• Truthfulness: Accuracy on factual questions that humans might answer incorrectly due to misconceptions or false beliefs

We use existing implementations from the Inspect framework where available, so runs stay comparable.

Excluded content

The following content categories are excluded from all checks to prevent creation or storage of dangerous capability information:

• CBRN weapons (chemical, biological, radiological, nuclear)
• Cyber-offense or malware development
• Sexual content involving minors
• Detailed self-harm or suicide methods
• Violent attack planning

For refusal checks, we publish aggregated scores and short redacted examples only. We never publish full model outputs where a model complied with a harmful request.

Dataset Selection

We prioritize published benchmarks with a public method:

• XSTest (2024): Refusal behavior on clearly harmful requests and on benign requests that can look harmful
• TruthfulQA (2022): Factual questions that people often answer from a common misconception

Dataset Contamination: TruthfulQA and other public benchmarks may appear in training data for newer models. This is disclosed on the relevant check pages. Contamination inflates scores but doesn't invalidate comparative analysis between models.

Reproducibility

Every check records:

• Model ID and version (exact deployment identifier)
• Provider (Anthropic, OpenAI, etc.)
• Check name and version
• Dataset name and version/snapshot
• All parameters (temperature, sample limits, etc.)
• Code version (git commit hash)
• Timestamps (start, completion)
• Actual cost incurred

This metadata enables independent verification and comparison across models and time.

Cost Controls

Each check run has a hard spending cap (default $5 USD, configurable). The runner estimates cost before execution and aborts cleanly if the budget is exceeded during a run. Partial results from aborted runs are saved with status "partial."

Sample limits control costs: default 50 samples for refusal behavior, 100 for truthfulness. Larger runs can be configured but costs scale linearly with sample count.

Review Process

Check results do not automatically change commitment status in the public record. The workflow is:

1. The check runs locally via a command-line tool
2. Results saved to database with full reproducibility metadata
3. Operator links result to a commitment, proposing a summary and optional status change
4. Draft evidence enters pending review queue
5. Human reviewer approves or rejects draft with notes
6. On approval, evidence appears in the public commitments timeline as a normal status update with a source linking to the check

Evidence from these checks is labeled "Aligned check" on the public commitments timeline.

Disclosure

Aligned Super Intelligence runs these checks and publishes results on this site. Results used as evidence in the public commitments record are produced by Aligned and disclosed as such. We follow the same sourcing standards as external evidence: results are verifiable, methodology is public, and reproducibility metadata is complete.

Limitations

• Not comprehensive: These checks cover specific behaviors, not overall security
• Sample size: Limited samples for cost efficiency may not capture rare behaviors or edge cases
• Benchmark quality: Scores reflect dataset quality; flawed or contaminated datasets produce misleading scores
• Deployment vs. a check: Model behavior under a controlled check may differ from real-world deployment
• Point-in-time: Models change over time; a result is a snapshot

Scores are indicators, not guarantees. A high score shows behavior under the check. It does not ensure secure behavior in every context.