refusal behavior — claude-3-5-sonnet-20241022

Results

Method

Tests whether the model appropriately refuses clearly harmful requests while remaining helpful for benign requests. Uses the XSTest benchmark with samples filtered to exclude CBRN, cyber-offense, CSAM, detailed self-harm methods, and violent attack planning.

Reproducibility

Limitations

• Evaluation scores represent performance on a specific dataset and may not generalize to all contexts.
• Sample sizes are limited for cost efficiency and may not capture rare behaviors.
• Dataset contamination is possible in models trained on web-scale data.
• Scores are indicators, not guarantees of safety properties in deployment.