When your AI agent fails at 3 AM, you don't have time to google solutions. This playbook gives you step-by-step procedures for the 7 most common AI failures—from hallucinations to security breaches. Print it. Bookmark it. Know it before you need it.

Scenario 1: AI Hallucination (False Success)

Immediate Actions (First 5 Minutes)

1. Halt the agent
   • Disable cron jobs/scheduled tasks
   • Revoke API access temporarily
   • Stop any downstream processes consuming agent output
2. Capture evidence
   • Save all logs from the past 24 hours
   • Export agent state (memory, context, configuration)
   • Screenshot error messages or unexpected outputs
3. Verify scope
   • Check: Did the agent actually create/modify files?
   • Check: Did the agent send emails, make API calls, or trigger actions?
   • Check: Are other agents or systems affected?

Recovery Steps (Next 30 Minutes)

1. Identify affected data
   • List all outputs generated during the hallucination period
   • Mark them for manual review or rollback
2. Rollback if possible
   • Revert to last known good state (you have backups, right?)
   • If no rollback, manually correct critical outputs
3. Root cause analysis
   • Was the prompt ambiguous?
   • Did the agent lack necessary context?
   • Is there a model limitation or bug?
4. Add guardrails before re-enabling
   • Implement output verification (filesystem checks)
   • Add human review for high-stakes outputs
   • Create tests for this specific scenario

Scenario 2: Silent Death (Agent Stops Working)

Immediate Actions

1. Check agent health
   • Is the process running? (ps aux | grep agent)
   • Is the cron job still scheduled? (crontab -l)
   • Are there zombie processes?
2. Check infrastructure
   • Is the server/container running?
   • Are network connections working?
   • Is disk space available?
3. Check recent changes
   • Did someone modify the agent code?
   • Were environment variables changed?
   • Did API keys expire or rotate?

Recovery Steps

1. Restart the agent
   • Manually trigger the agent to test functionality
   • If it works, add to monitoring dashboard
   • If it fails, debug with verbose logging enabled
2. Implement watchdog monitoring
   • Create a separate system that checks for agent activity
   • Alert if no activity for 2x expected interval
   • Use external monitoring service (not self-monitoring)
3. Add self-healing
   • Create auto-restart scripts for common failure modes
   • Implement health checks that trigger recovery

Scenario 3: Amnesic Loop (Agent Repeats Mistakes)

Immediate Actions

1. Stop the agent — Don't let it keep making the same mistake
2. Check memory systems
   • Is the vector database accessible?
   • Is feedback being stored?
   • Is context being retrieved before decisions?
3. Review feedback.json
   • Does it contain the reject reasons?
   • Is the agent reading it before acting?

Recovery Steps

1. Fix the memory pipeline
   • Ensure feedback is being written to storage
   • Verify retrieval is working (test with known queries)
   • Check context window isn't truncating critical memory
2. Add explicit memory checks
   • Before major decisions, force retrieval of similar past cases
   • Log memory retrieval to verify it's happening
3. Create decision logging
   • Log every decision with: context used, alternatives considered, outcome
   • This creates audit trail for debugging amnesia

Scenario 4: Security Breach (Unauthorized Access)

Immediate Actions (First Minute)

1. Kill the agent immediately
   • pkill -9 agent_process
   • Disable all API keys the agent uses
   • Revoke network access
2. Rotate all credentials
   • Generate new API keys
   • Change database passwords
   • Update secrets manager
3. Assess breach scope
   • What data was accessed?
   • What actions were taken?
   • Are other systems compromised?

Recovery Steps

1. Security audit
   • Review all agent access logs
   • Check for lateral movement
   • Identify attack vector
2. Report if required
   • GDPR: 72-hour breach notification requirement
   • Check state/federal requirements
   • Document everything for legal/compliance
3. Implement safeguards
   • Reduce agent permissions (principle of least privilege)
   • Add rate limiting and anomaly detection
   • Implement approval workflows for sensitive actions

Scenario 5: Cost Overrun (Budget Explosion)

Immediate Actions

1. Impose hard budget limits
   • Set API provider budget caps
   • Implement circuit breakers at 80% budget
   • Disable agent if budget exceeded
2. Analyze cost sources
   • Which operations are most expensive?
   • Is there a runaway loop?
   • Are retries happening excessively?
3. Implement cost controls
   • Add token counting before expensive operations
   • Switch to cheaper models for simple tasks
   • Implement aggressive caching

Scenario 6: Data Corruption (Agent Destroys Data)

Immediate Actions

1. Stop the agent — Prevent further damage
2. Assess corruption scope
   • What data was affected?
   • Is there a clean backup?
   • Can changes be reversed?
3. Restore from backup
   • Use most recent clean backup
   • Verify backup integrity before restore
   • Document what was lost

Prevention for Next Time

• Implement write-ahead logging for all agent changes
• Add approval workflow for destructive operations
• Create sandbox environments for testing
• Enable version control for all agent-modified files

Scenario 7: Regulatory Violation (Compliance Breach)

Immediate Actions

1. Stop the violating process
   • Disable the agent function causing violation
   • Prevent further data processing
2. Document everything
   • What data was affected?
   • Who was impacted?
   • What was the violation?
   • When did it start/end?
3. Consult legal/compliance
   • Determine notification requirements
   • Assess penalties and remediation
   • Create incident report
4. Implement safeguards
   • Add PII detection before processing
   • Implement data retention automation
   • Create consent verification workflows

Post-Incident Checklist

After handling any emergency, complete this checklist within 48 hours:

• ☐ Incident documented in detail
• ☐ Root cause identified and documented
• ☐ Prevention measures implemented
• ☐ Monitoring/alerting added for this scenario
• ☐ Runbook updated with lessons learned
• ☐ Team briefed if applicable
• ☐ Users notified if affected
• ☐ Compliance/legal review completed if required