使用 PagerDuty 构建自定义 CaptchaAI 警报

验证码管道半夜挂掉，值班工程师往往是被用户投诉叫醒的——这时候数据其实已经断了好几个小时。把 CaptchaAI 接入 PagerDuty 的 Events API，能在余额跌破阈值、错误率异常或 worker 集体掉线时直接触发寻呼，附带足够上下文，不用现翻日志。

国内团队常用钉钉、飞书机器人做值班通知，胜在轻量；PagerDuty 的优势是成熟的升级链路和电话、短信兜底——同一次 trigger() 可以同时推给两边。

警报规则怎么定：从余额到错误率

不是所有异常都值得半夜叫醒人：只有余额见底或 worker 全离线才走 critical 寻呼，其余先建事件，白天处理。

• 紧急，立即寻呼：余额 < $2；或全部 worker 离线。
• 较高，建紧急事件（不寻呼）：错误率 > 20%，持续 5 分钟。
• 警告，建低优先级事件：余额 < $10；或队列深度 > 100，持续 10 分钟。
• 信息，追加到已有事件或写日志：识别耗时 p95 > 120s。

PagerDuty 配置清单（6 步接入）

1. 在 PagerDuty 里为 "CaptchaAI Pipeline" 新建 Service。
2. 给 Service 添加 Events API v2 集成。
3. 把 routing key 复制到环境变量 PAGERDUTY_ROUTING_KEY。
4. 制定升级策略（值班 → 组长 → 经理）。
5. 配置通知方式（推送、短信、电话）。
6. 加计划维护窗口，避免误报。

Python 示例：PagerDuty 事件 API v2

import os
import time
import hashlib
import requests
from datetime import datetime

API_KEY = os.environ["CAPTCHAAI_API_KEY"]
PAGERDUTY_ROUTING_KEY = os.environ["PAGERDUTY_ROUTING_KEY"]

session = requests.Session()


class CaptchaPagerDuty:
    EVENTS_URL = "https://events.pagerduty.com/v2/enqueue"

    def __init__(self, routing_key):
        self.routing_key = routing_key

    def trigger(self, summary, severity="error", source="captcha-pipeline",
                details=None, dedup_key=None):
        """Trigger a new PagerDuty incident."""
        payload = {
            "routing_key": self.routing_key,
            "event_action": "trigger",
            "payload": {
                "summary": summary,
                "severity": severity,  # critical, error, warning, info
                "source": source,
                "timestamp": datetime.utcnow().isoformat() + "Z",
                "custom_details": details or {}
            }
        }

        if dedup_key:
            payload["dedup_key"] = dedup_key

        resp = requests.post(self.EVENTS_URL, json=payload, timeout=10)
        resp.raise_for_status()
        return resp.json()

    def resolve(self, dedup_key):
        """Resolve an existing incident."""
        payload = {
            "routing_key": self.routing_key,
            "event_action": "resolve",
            "dedup_key": dedup_key
        }
        resp = requests.post(self.EVENTS_URL, json=payload, timeout=10)
        resp.raise_for_status()
        return resp.json()

    def acknowledge(self, dedup_key):
        """Acknowledge an existing incident."""
        payload = {
            "routing_key": self.routing_key,
            "event_action": "acknowledge",
            "dedup_key": dedup_key
        }
        resp = requests.post(self.EVENTS_URL, json=payload, timeout=10)
        resp.raise_for_status()
        return resp.json()


pagerduty = CaptchaPagerDuty(PAGERDUTY_ROUTING_KEY)


class CaptchaMonitor:
    def __init__(self):
        self.error_window = []  # (timestamp, is_error)
        self.window_size = 300  # 5 minutes in seconds

    def record_solve(self, success):
        now = time.time()
        self.error_window.append((now, not success))
        # Prune old entries
        self.error_window = [
            (t, e) for t, e in self.error_window
            if now - t < self.window_size
        ]

    @property
    def error_rate(self):
        if not self.error_window:
            return 0.0
        errors = sum(1 for _, e in self.error_window if e)
        return errors / len(self.error_window)

    def check_balance(self):
        resp = session.get("https://ocr.captchaai.com/res.php", params={
            "key": API_KEY, "action": "getbalance", "json": 1
        })
        data = resp.json()
        if data.get("status") != 1:
            return None
        return float(data["request"])

    def run_checks(self):
        """Run all monitoring checks and trigger alerts."""
        # Check balance
        balance = self.check_balance()
        if balance is not None:
            if balance < 2:
                pagerduty.trigger(
                    summary=f"CaptchaAI balance critically low: ${balance:.2f}",
                    severity="critical",
                    dedup_key="captcha-balance-critical",
                    details={"balance": balance, "threshold": 2}
                )
            elif balance < 10:
                pagerduty.trigger(
                    summary=f"CaptchaAI balance low: ${balance:.2f}",
                    severity="warning",
                    dedup_key="captcha-balance-warning",
                    details={"balance": balance, "threshold": 10}
                )
            else:
                # Resolve if balance recovered
                try:
                    pagerduty.resolve("captcha-balance-critical")
                    pagerduty.resolve("captcha-balance-warning")
                except Exception:
                    pass  # No incident to resolve

        # Check error rate
        rate = self.error_rate
        if rate > 0.20:
            total = len(self.error_window)
            errors = sum(1 for _, e in self.error_window if e)
            pagerduty.trigger(
                summary=f"CaptchaAI error rate {rate:.0%} "
                        f"({errors}/{total} in 5 min)",
                severity="error",
                dedup_key="captcha-error-rate-high",
                details={
                    "error_rate": round(rate, 3),
                    "total_tasks": total,
                    "failed_tasks": errors,
                    "window_seconds": self.window_size
                }
            )
        elif rate < 0.05 and len(self.error_window) > 10:
            try:
                pagerduty.resolve("captcha-error-rate-high")
            except Exception:
                pass


monitor = CaptchaMonitor()

# After each solve:
# monitor.record_solve(success=True)

# Run checks every 60 seconds:
# while True:
#     monitor.run_checks()
#     time.sleep(60)

国内机房访问 events.pagerduty.com 一般没问题，但跨境线路偶尔会有额外延迟，代码里 timeout=10 秒是保守值，出口不稳定可以调大，避免误判成故障。

JavaScript 示例：PagerDuty 集成

const axios = require("axios");

const API_KEY = process.env.CAPTCHAAI_API_KEY;
const PD_ROUTING_KEY = process.env.PAGERDUTY_ROUTING_KEY;
const PD_EVENTS_URL = "https://events.pagerduty.com/v2/enqueue";

class PagerDutyAlerter {
  constructor(routingKey) {
    this.routingKey = routingKey;
  }

  async trigger(summary, severity = "error", details = {}, dedupKey = null) {
    const payload = {
      routing_key: this.routingKey,
      event_action: "trigger",
      payload: {
        summary,
        severity,
        source: "captcha-pipeline",
        timestamp: new Date().toISOString(),
        custom_details: details,
      },
    };
    if (dedupKey) payload.dedup_key = dedupKey;

    const resp = await axios.post(PD_EVENTS_URL, payload, { timeout: 10000 });
    return resp.data;
  }

  async resolve(dedupKey) {
    await axios.post(PD_EVENTS_URL, {
      routing_key: this.routingKey,
      event_action: "resolve",
      dedup_key: dedupKey,
    }, { timeout: 10000 });
  }
}

const alerter = new PagerDutyAlerter(PD_ROUTING_KEY);

class CaptchaHealthMonitor {
  constructor(windowMs = 300000) {
    this.results = [];
    this.windowMs = windowMs;
  }

  record(success) {
    this.results.push({ time: Date.now(), success });
    const cutoff = Date.now() - this.windowMs;
    this.results = this.results.filter((r) => r.time > cutoff);
  }

  get errorRate() {
    if (this.results.length === 0) return 0;
    const errors = this.results.filter((r) => !r.success).length;
    return errors / this.results.length;
  }

  async checkAndAlert() {
    // Balance check
    try {
      const resp = await axios.get("https://ocr.captchaai.com/res.php", {
        params: { key: API_KEY, action: "getbalance", json: 1 },
      });
      if (resp.data.status === 1) {
        const balance = parseFloat(resp.data.request);
        if (balance < 2) {
          await alerter.trigger(
            `CaptchaAI balance critically low: $${balance.toFixed(2)}`,
            "critical",
            { balance },
            "captcha-balance-critical"
          );
        } else if (balance < 10) {
          await alerter.trigger(
            `CaptchaAI balance low: $${balance.toFixed(2)}`,
            "warning",
            { balance },
            "captcha-balance-warning"
          );
        } else {
          await alerter.resolve("captcha-balance-critical").catch(() => {});
          await alerter.resolve("captcha-balance-warning").catch(() => {});
        }
      }
    } catch (err) {
      console.error("Balance check failed:", err.message);
    }

    // Error rate check
    const rate = this.errorRate;
    if (rate > 0.2 && this.results.length > 10) {
      await alerter.trigger(
        `CaptchaAI error rate: ${(rate * 100).toFixed(1)}%`,
        "error",
        { errorRate: rate, totalTasks: this.results.length },
        "captcha-error-rate"
      );
    } else if (rate < 0.05 && this.results.length > 10) {
      await alerter.resolve("captcha-error-rate").catch(() => {});
    }
  }
}

const monitor = new CaptchaHealthMonitor();

// Run checks every 60 seconds
setInterval(() => monitor.checkAndAlert(), 60000);

module.exports = { monitor, alerter };

常见故障排查

• 警报没触发：routing key 配错了——确认与该 Service 的 Events API 集成一致。
• 重复事件：没传 dedup_key——给每种警报类型固定一个值，别每次都变。
• 警报刷屏：PagerDuty 靠 dedup_key 去重抑制重复，检查代码里是否漏传。
• 自动恢复没生效：确保 resolve 和 trigger 传的是完全相同的 dedup_key。

常见问题

怎么避免被警报炸到疲劳？

用 dedup_key 把同类警报归并成一个事件，warning 级别都设成低优先级、不寻呼人。只有余额 < $2 或 worker 全挂了才配 critical。

上线前怎么确认 PagerDuty 真的能收到事件？

本地手动跑一次 trigger()，去后台确认测试事件已出现，再用同一个 dedup_key 跑 resolve() 确认能关闭。建议先在 staging 走一遍完整流程再接生产。

余额告警和错误率告警，哪个优先级该更高？

余额耗尽是硬性阻断，之后请求全会失败，所以余额 < $2 直接走 critical；错误率升高通常只是部分任务失败，建高优先级事件即可，不必半夜叫人。

trigger、acknowledge 和 resolve 具体有什么区别？

trigger 创建新事件；acknowledge 停止后续通知，但事件仍打开；resolve 才彻底关闭事件。三者都靠同一个 dedup_key 关联，用错会导致重复事件或自动恢复失效。

延伸阅读

• 客户端验证码处理管道搭建指南
• 负责任的自动化实践指南
• Datadog 指标监控与告警配置

立即开始接入

验证码管道一出问题就想第一时间知道？申请一个 CaptchaAI API Key，接上本文的 PagerDuty 配置就能跑起来。

相关指南：

• Datadog 监控接入
• New Relic APM 集成
• 错误码参考