AI Content Moderation API Guide: Filter NSFW, Spam, and Toxic Content

Published 2026-07-26 · 13 min read

Every AI application that accepts user input faces the same risk: users will submit inappropriate content. NSFW images, spam, hate speech, prompt injection attacks, and personally identifiable information (PII) can all flow through your chatbot or AI tool unless you actively filter them. Content moderation APIs provide automated, real-time filtering that protects your users, your brand, and your API budget. This guide covers everything from choosing a moderation API to implementing multi-layer filtering in production.

Why Content Moderation Is Non-Negotiable

Without moderation, bad actors can:

• Abuse your API budget by sending thousands of spam messages through your chatbot
• Extract harmful content by using jailbreak prompts to bypass model safety guardrails
• Damage your brand by generating screenshots of your chatbot producing offensive content
• Violate regulations (GDPR, COPPA, platform policies) by submitting PII through your forms
• Skew your analytics by flooding your system with bot-generated noise

A single viral screenshot of your AI producing harmful content can destroy months of brand building. Moderation is cheap insurance.

Types of Content to Moderate

OpenAI Moderation API (Built Into DrAI)

The most accessible moderation API is OpenAI's moderation endpoint, which is free for all users and accessible through DrAI's gateway:

curl https://ai.dr-ai.top/v1/moderations \
  -H "Authorization: Bearer sk-your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "omni-moderation-latest",
    "input": "Some text to moderate"
  }'

The response categorizes the input across multiple dimensions:

{
  "id": "modr-xxx",
  "results": [{
    "flagged": false,
    "categories": {
      "harassment": false,
      "hate": false,
      "self_harm": false,
      "sexual": false,
      "violence": false,
      "harassment/threatening": false,
      "hate/threatening": false,
      "self_harm/instructions": false,
      "self_harm/intent": false,
      "sexual/minors": false,
      "violence/graphic": false
    },
    "category_scores": {
      "harassment": 0.0001,
      "hate": 0.0002,
      "sexual": 0.0001,
      "violence": 0.0001
    }
  }]
}

The category_scores give you granular confidence values (0-1) for each category, letting you set custom thresholds rather than relying on the binary flagged boolean.

Implementing Multi-Layer Moderation

A robust moderation system uses multiple layers, each catching different threats:

Layer 1: Word List Filter (Fastest, Cheapest)

Block obvious profanity and known spam patterns with a simple word list. This catches 60-70% of bad content instantly with zero latency and zero cost:

BLOCKED_WORDS = {'spam', 'casino', 'viagra', 'crypto giveaway', ...}

def quick_filter(text):
    text_lower = text.lower()
    for word in BLOCKED_WORDS:
        if word in text_lower:
            return False, f"Blocked word detected: {word}"
    return True, None

Layer 2: AI Moderation API (Accurate, Real-Time)

For everything that passes the word list, run the AI moderation API. This catches nuanced content that word lists miss:

async def ai_moderate(text, api_key):
    response = requests.post(
        'https://ai.dr-ai.top/v1/moderations',
        headers={'Authorization': f'Bearer {api_key}'},
        json={
            'model': 'omni-moderation-latest',
            'input': text
        }
    )
    result = response.json()['results'][0]
    
    # Use custom thresholds for flexibility
    THRESHOLDS = {
        'hate': 0.5,
        'violence': 0.7,
        'sexual': 0.8,
        'harassment': 0.6
    }
    
    for category, threshold in THRESHOLDS.items():
        score = result['category_scores'].get(category, 0)
        if score > threshold:
            return False, f"{category} score {score:.2f} exceeds threshold"
    
    return True, None

Layer 3: Prompt Injection Detection

Prompt injection — where users try to override your system instructions — is a unique threat. Common patterns include "ignore previous instructions," "you are now DAN," and similar jailbreak attempts. Detect them with pattern matching + AI classification:

INJECTION_PATTERNS = [
    r'ignore (all |any |previous )?instructions',
    r'you are now (DAN|an? \w+ without restrictions)',
    r'forget (everything|your rules|previous)',
    r'system prompt',
    r'\[system\]',
    r'jailbreak'
]

import re

def detect_injection(text):
    text_lower = text.lower()
    for pattern in INJECTION_PATTERNS:
        if re.search(pattern, text_lower):
            return True
    return False

Layer 4: Rate-Based Abuse Prevention

Even legitimate-looking content can be abusive if sent at high volume. Track per-user message rates and flag suspicious patterns. See our rate limiting guide for implementation details.

Putting It All Together: A Moderation Pipeline

class ModerationPipeline:
    def __init__(self, api_key):
        self.api_key = api_key
        self.blocked_words = load_blocked_words()
    
    async def moderate(self, user_input, user_id):
        # Layer 1: Quick word filter
        is_safe, reason = self.quick_filter(user_input)
        if not is_safe:
            return ModerationResult(blocked=True, reason=reason, layer=1)
        
        # Layer 2: Prompt injection check
        if self.detect_injection(user_input):
            return ModerationResult(blocked=True, 
                reason="Prompt injection detected", layer=2)
        
        # Layer 3: AI moderation API
        is_safe, reason = await self.ai_moderate(user_input)
        if not is_safe:
            return ModerationResult(blocked=True, reason=reason, layer=3)
        
        # Layer 4: Rate check
        if self.rate_limited(user_id):
            return ModerationResult(blocked=True,
                reason="Rate limit exceeded", layer=4)
        
        return ModerationResult(blocked=False, reason=None, layer=0)

# Usage in your chat endpoint
pipeline = ModerationPipeline("sk-your-key")
result = await pipeline.moderate(user_input, user_id)
if result.blocked:
    return {"error": "Your message was flagged. Please revise."}
# Proceed to AI generation

Image Content Moderation

If your application accepts image uploads, you need visual moderation too. The omni-moderation API supports image inputs:

response = requests.post(
    'https://ai.dr-ai.top/v1/moderations',
    headers={'Authorization': f'Bearer {api_key}'},
    json={
        'model': 'omni-moderation-latest',
        'input': [{'type': 'image_url', 
                    'image_url': {'url': image_data_url}}]
    }
)

Handling Moderation Results: UX Best Practices

When content is flagged, how you communicate this to users matters:

Be vague, not specific. Don't tell users exactly which word triggered the filter — that helps bad actors reverse-engineer your rules. Use generic messages: "Your message was flagged as potentially inappropriate."

Offer alternatives. Instead of just blocking, suggest: "Please rephrase your message and try again."

Log everything. For analytics and false positive debugging, log all moderation decisions (blocked/allowed, which layer, scores) with timestamps. Never log the actual content if it contains PII.

Provide appeal mechanisms. False positives happen. For trusted users, offer a "report false positive" button that routes to human review.

Cost Optimization for Moderation

The OpenAI moderation API is free through DrAI, but you still pay for the network round-trip. To minimize overhead:

Cache moderation results. If the same text is submitted multiple times (common in spam), cache the moderation result. This is especially effective for the word-list and injection layers.

Only moderate what changes. In multi-turn conversations, only moderate the latest user message, not the entire conversation history each time.

Use the word list first. The word list filter is instant and free. If 30% of bad content is caught there, you save 30% of moderation API calls.

Legal and Compliance Considerations

Content moderation intersects with several legal frameworks:

GDPR: Moderation logs may contain PII. Ensure they're encrypted, retained for a limited period, and deletable on user request.

Section 230 (US): In the US, platforms are generally not liable for user-generated content, but active moderation can affect this safe harbor in some jurisdictions. Consult legal counsel.

Platform policies: Apple App Store, Google Play, and cloud providers all have content policies. Ensure your moderation catches violations of these policies, or risk app removal.

Third-Party Moderation Services

Beyond the OpenAI moderation API, several specialized services offer deeper filtering:

Perspective API (Google/Jigsaw): Free, focuses on toxicity scoring. Returns probability scores for toxicity, severe toxicity, identity attack, insult, profanity, and threat. Great for comment sections and community forums.

AWS Rekognition: Image and video moderation. Detects explicit and suggestive content, violence, and visually disturbing imagery. Integrates well if you're already on AWS.

Sightengine: Multi-modal moderation (text, image, video) with real-time API. Particularly strong at detecting deepfakes and AI-generated images — increasingly important in 2026.

For most applications, the free OpenAI moderation API (available through DrAI) plus a word-list filter is sufficient. Add specialized services only if you have specific needs (high-volume image moderation, regulatory compliance, or advanced deepfake detection).

Measuring Moderation Effectiveness

Track these metrics to evaluate your moderation system:

Regularly sample-blocked content and manually review it to calibrate your thresholds. If false positives are too high, raise thresholds. If bad content is slipping through, lower them or add new patterns to your word list.

Conclusion

Content moderation is not optional for production AI applications. The layered approach — word filter, AI moderation API, prompt injection detection, and rate limiting — provides defense in depth that catches the vast majority of problematic content while minimizing false positives. With DrAI's built-in moderation API (free for all users), implementing this pipeline costs nothing but saves you from potential brand damage, API budget waste, and compliance violations.

For more on protecting your AI application, see our chatbot integration guide and rate limiting guide. Ready to build a safe, moderated AI application? Get started with DrAI.