Best Practices

1. Actionable Over Informational

Every alert should require human action. If an alert doesn't need immediate attention, it shouldn't wake someone up.

2. Customer Impact Focus

Alert on symptoms that affect users, not just on causes. Monitor what matters to your customers first.

• Primary: API error rate, response time, availability
• Secondary: CPU usage, memory, disk space
• Context: Include both in alert metadata

3. Clear Ownership

Every alert must have a clear owner. Use tags and routing rules to ensure alerts reach the right team immediately.

// Always include team ownership
await a11ops.alert({
  title: "Payment processing errors increasing",
  priority: "high",
  metadata: {
    team: "payments",          // Clear ownership
    service: "payment-api",
    runbook: "link/to/runbook" // How to respond
  }
});

Use Appropriate Thresholds

Set thresholds based on actual impact, not arbitrary numbers:

Implement Alert Deduplication

Prevent duplicate alerts from overwhelming your team:

• Use consistent alert names and keys for deduplication
• Group related alerts (e.g., all hosts with same issue)
• Set appropriate evaluation windows to prevent flapping
• Use “for” duration in alert rules (minimum 5 minutes)

Regular Alert Review

Schedule monthly reviews to improve alert quality:

1. Review all alerts from the past month
2. Identify alerts that were not actionable
3. Find alerts that fired too frequently
4. Adjust thresholds or remove unnecessary alerts
5. Add missing alerts based on incidents

Alert Title Template

Use a consistent format for alert titles:

Message Body Structure

Include essential information in a scannable format:

📊 WHAT: API response time p95 > 5 seconds
📍 WHERE: Production us-east-1, service: user-api
📈 METRICS: Current: 5.2s, Normal: 0.8s, Duration: 10 min
👥 IMPACT: ~2,000 users experiencing slow page loads
🔧 ACTION: 
   1. Check recent deployments
   2. Review database slow query log
   3. Scale service if needed
📖 RUNBOOK: https://wiki.company.com/runbooks/api-latency
📊 DASHBOARD: https://grafana.company.com/d/api-performance

Include Context in Metadata

await a11ops.alert({
  title: "Order Processing: Queue backlog growing",
  message: "Order queue depth exceeded threshold",
  priority: "high",
  metadata: {
    // Current state
    queue_depth: 5420,
    threshold: 1000,
    processing_rate: "50/min",
    
    // Historical context
    normal_depth: 100,
    peak_hour_depth: 500,
    
    // Business impact
    orders_delayed: 5320,
    estimated_delay: "90 minutes",
    revenue_at_risk: "$45,000",
    
    // Technical details
    region: "us-west-2",
    cluster: "orders-prod-1",
    version: "2.3.1",
    last_deploy: "2024-01-15T10:00:00Z"
  }
});