New Relic APM 监控 CaptchaAI 验证码识别管道

验证码识别管道出问题时，你是先自己发现，还是先被工单通知？接入 New Relic APM 后，CaptchaAI 的提交延迟、轮询耗时、错误占比都会变成可查询、可画图、可告警的数据。下面用 Python 和 Node.js 演示怎么埋点、搭仪表板、定告警阈值。

需要监控的三个环节

验证码识别可以拆成三个阶段——提交任务、等待结果、拿到 token 之后使用。每个阶段容易出问题的地方不一样，对应的指标也不一样：

[Submit Task] → [Wait for Solution] → [Apply Token]
     ↓                  ↓                   ↓
  Submit latency    Poll duration       Token usage
  API errors        Timeout rate        Success rate

提交阶段看 API 报错率，轮询阶段看耗时和超时占比，用到 token 之后看它在业务里的最终成功率——分开看才能快速判断问题出在哪一侧。

举例来说，电商站点大促（比如双 11）验证码提交量常常几分钟内翻几倍，轮询耗时曲线通常先于工单出现异常——这正是告警要抓住的窗口。

Python 埋点：给识别函数加自定义追踪

下面用 @newrelic.agent.background_task 包住识别流程，function_trace 标记提交、轮询两阶段，再用自定义属性和事件把验证码类型、耗时、成败记下来，方便按类型筛选：

import os
import time
import requests
import newrelic.agent

API_KEY = os.environ["CAPTCHAAI_API_KEY"]
session = requests.Session()


@newrelic.agent.background_task(name="captcha_solve", group="CaptchaAI")
def solve_captcha(sitekey, pageurl, captcha_type="recaptcha_v2"):
    """Solve a CAPTCHA with full New Relic instrumentation."""
    # Add custom attributes for filtering
    newrelic.agent.add_custom_attributes([
        ("captcha_type", captcha_type),
        ("target_url", pageurl),
    ])

    # Submit phase
    submit_result = _submit_task(sitekey, pageurl, captcha_type)
    if "error" in submit_result:
        newrelic.agent.record_custom_event("CaptchaSolveError", {
            "error": submit_result["error"],
            "phase": "submit",
            "captcha_type": captcha_type,
        })
        return submit_result

    # Poll phase
    captcha_id = submit_result["captcha_id"]
    poll_result = _poll_result(captcha_id, captcha_type)

    # Record solve event
    event_data = {
        "captcha_type": captcha_type,
        "captcha_id": captcha_id,
        "success": "solution" in poll_result,
    }
    if "solution" in poll_result:
        event_data["solve_time"] = poll_result.get("elapsed", 0)
        newrelic.agent.record_custom_event("CaptchaSolveSuccess", event_data)
    else:
        event_data["error"] = poll_result.get("error", "unknown")
        newrelic.agent.record_custom_event("CaptchaSolveError", event_data)

    return poll_result


@newrelic.agent.function_trace(name="captcha_submit")
def _submit_task(sitekey, pageurl, captcha_type):
    payload = {
        "key": API_KEY,
        "method": "userrecaptcha",
        "googlekey": sitekey,
        "pageurl": pageurl,
        "json": 1
    }
    resp = session.post("https://ocr.captchaai.com/in.php", data=payload)
    data = resp.json()

    newrelic.agent.add_custom_attributes([
        ("submit_status", data.get("status")),
    ])

    if data.get("status") != 1:
        return {"error": data.get("request")}
    return {"captcha_id": data["request"]}


@newrelic.agent.function_trace(name="captcha_poll")
def _poll_result(captcha_id, captcha_type):
    start = time.time()
    poll_count = 0

    for _ in range(60):
        time.sleep(5)
        poll_count += 1
        result = session.get("https://ocr.captchaai.com/res.php", params={
            "key": API_KEY, "action": "get", "id": captcha_id, "json": 1
        }).json()

        if result.get("status") == 1:
            elapsed = time.time() - start
            newrelic.agent.add_custom_attributes([
                ("poll_count", poll_count),
                ("solve_time_seconds", round(elapsed, 2)),
            ])
            return {"solution": result["request"], "elapsed": elapsed}

        if result.get("request") != "CAPCHA_NOT_READY":
            return {"error": result.get("request")}

    return {"error": "TIMEOUT"}


def report_balance():
    """Record balance as a custom event."""
    resp = session.get("https://ocr.captchaai.com/res.php", params={
        "key": API_KEY, "action": "getbalance", "json": 1
    })
    data = resp.json()
    if data.get("status") == 1:
        balance = float(data["request"])
        newrelic.agent.record_custom_event("CaptchaBalance", {
            "balance": balance,
            "low": balance < 10,
        })
        return balance
    return None

report_balance() 用 cron 或 APScheduler 每分钟跑一次即可，低余额告警直接查 CaptchaBalance 表。

New Relic Agent 配置文件

newrelic.ini 里，transaction_threshold 决定多慢的事务被完整记录，两个 custom_insights_events.* 字段决定事件上报开关和每分钟存储条数：

# newrelic.ini
[newrelic]
app_name = CaptchaAI Pipeline
license_key = YOUR_NEW_RELIC_LICENSE_KEY
monitor_mode = true
log_level = info
transaction_tracer.enabled = true
transaction_tracer.transaction_threshold = 5.0
custom_insights_events.enabled = true
custom_insights_events.max_samples_stored = 5000

5 秒的阈值对识别场景偏高——轮询本身就要 5 到 120 秒，建议按类型分别设置。

Node.js 埋点：New Relic 集成

思路和 Python 一致，用 startBackgroundTransaction 包住流程，区别是要手动调用 transaction.end()——漏掉一个 return 分支忘记结束，耗时数据就会失真：

const newrelic = require("newrelic");
const axios = require("axios");

const API_KEY = process.env.CAPTCHAAI_API_KEY;

async function solveCaptchaWithNewRelic(sitekey, pageurl, captchaType = "recaptcha_v2") {
  return newrelic.startBackgroundTransaction(
    "CaptchaSolve",
    "CaptchaAI",
    async () => {
      const transaction = newrelic.getTransaction();
      newrelic.addCustomAttributes({
        captchaType,
        targetUrl: pageurl,
      });

      const startTime = Date.now();

      try {
        // Submit
        const submitResp = await axios.post(
          "https://ocr.captchaai.com/in.php",
          null,
          {
            params: {
              key: API_KEY,
              method: "userrecaptcha",
              googlekey: sitekey,
              pageurl: pageurl,
              json: 1,
            },
          }
        );

        if (submitResp.data.status !== 1) {
          newrelic.recordCustomEvent("CaptchaSolveError", {
            error: submitResp.data.request,
            phase: "submit",
            captchaType,
          });
          transaction.end();
          return { error: submitResp.data.request };
        }

        const captchaId = submitResp.data.request;
        newrelic.addCustomAttributes({ captchaId });

        // Poll
        let pollCount = 0;
        for (let i = 0; i < 60; i++) {
          await new Promise((r) => setTimeout(r, 5000));
          pollCount++;

          const pollResp = await axios.get(
            "https://ocr.captchaai.com/res.php",
            {
              params: {
                key: API_KEY, action: "get", id: captchaId, json: 1,
              },
            }
          );

          if (pollResp.data.status === 1) {
            const elapsed = (Date.now() - startTime) / 1000;
            newrelic.recordCustomEvent("CaptchaSolveSuccess", {
              captchaType,
              solveTime: elapsed,
              pollCount,
            });
            newrelic.addCustomAttributes({
              solveTime: elapsed,
              pollCount,
            });
            transaction.end();
            return { solution: pollResp.data.request, elapsed };
          }

          if (pollResp.data.request !== "CAPCHA_NOT_READY") {
            newrelic.recordCustomEvent("CaptchaSolveError", {
              error: pollResp.data.request,
              phase: "poll",
              captchaType,
            });
            transaction.end();
            return { error: pollResp.data.request };
          }
        }

        newrelic.recordCustomEvent("CaptchaSolveError", {
          error: "TIMEOUT",
          phase: "poll",
          captchaType,
          pollCount,
        });
        transaction.end();
        return { error: "TIMEOUT" };
      } catch (err) {
        newrelic.noticeError(err);
        transaction.end();
        throw err;
      }
    }
  );
}

// Balance monitoring
async function monitorBalance() {
  try {
    const resp = await axios.get("https://ocr.captchaai.com/res.php", {
      params: { key: API_KEY, action: "getbalance", json: 1 },
    });
    if (resp.data.status === 1) {
      const balance = parseFloat(resp.data.request);
      newrelic.recordCustomEvent("CaptchaBalance", { balance });
    }
  } catch (err) {
    newrelic.noticeError(err);
  }
}

setInterval(monitorBalance, 60000);

module.exports = { solveCaptchaWithNewRelic };

monitorBalance() 每 60 秒查一次余额写入 CaptchaBalance，和 Python 版对应同一张表，两种语言的服务可以共用一套仪表板和告警。

用 NRQL 搭建监控仪表板

有了三张事件表，下面几条 NRQL 覆盖成功率、按类型分组的平均耗时、错误分布、P95 延迟、余额趋势和每分钟任务量：

-- Solve success rate (last hour)
SELECT percentage(count(*), WHERE success = true)
FROM CaptchaSolveSuccess, CaptchaSolveError
SINCE 1 hour ago

-- Average solve time by CAPTCHA type
SELECT average(solveTime)
FROM CaptchaSolveSuccess
FACET captchaType
SINCE 1 hour ago TIMESERIES

-- Error breakdown
SELECT count(*)
FROM CaptchaSolveError
FACET error
SINCE 1 hour ago

-- P95 solve latency
SELECT percentile(solveTime, 95)
FROM CaptchaSolveSuccess
SINCE 1 hour ago TIMESERIES

-- Balance over time
SELECT latest(balance)
FROM CaptchaBalance
SINCE 24 hours ago TIMESERIES 5 minutes

-- Tasks per minute
SELECT rate(count(*), 1 minute)
FROM CaptchaSolveSuccess, CaptchaSolveError
SINCE 1 hour ago TIMESERIES

把六条查询加进一个仪表板，一屏就能看到识别管道从提交到余额的全貌。

告警策略：从数据到通知

告警项	NRQL 条件	触发阈值
识别成功率过低	`SELECT percentage(count(*), WHERE success = true)`	持续 5 分钟 < 85%
识别耗时过高	`SELECT percentile(solveTime, 95) FROM CaptchaSolveSuccess`	持续 10 分钟 > 120 秒
余额不足	`SELECT latest(balance) FROM CaptchaBalance`	< $10
错误率突增	`SELECT count(*) FROM CaptchaSolveError`	5 分钟内 > 50 次

成功率低指向 CaptchaAI 侧或参数问题，耗时高多半是排队或难度上升；余额和错误率最容易被忽略，后果却最直接。

常见故障排查

接入后最常遇到这四类问题，基本都能靠检查配置文件解决：

问题	原因	处理方式
自定义事件未出现	`custom_insights_events.enabled` 为 false	在 `newrelic.ini` 中启用
事务追踪缺失	门槛太高	将 `transaction_threshold` 降低至 1.0s
属性被截断	值太长	将属性值控制在 255 个字符以内
部署后无数据	许可证密钥错误或代理无法启动	检查 `newrelic-admin validate-config newrelic.ini`