DevOps & Scaling

CaptchaAI 指标的 Grafana 仪表板模板

这套 Grafana 仪表板模板用来监控 CaptchaAI 验证码管道:解决率、延迟、余额、队列深度和错误全部集中在一屏内,数据源是 Prometheus,几分钟即可导入。账户余额半夜跑空、某类验证码突然大面积超时——没有仪表板,往往等到第二天翻日志才发现。

谁该用这套仪表板

爬虫工程师盯解决率,运维同学盯余额和告警,这套模板把两边关心的指标放进同一屏:

  • 识别率下滑要第一时间发现
  • 余额告警要提前收到,不是任务批量失败才知道
  • 每个 Worker 的任务量和延迟要看得到

仪表板布局怎么设计

四行面板,从总览到具体的 Worker 状态,按优先级排查问题:总览异常就往下翻,直到定位到具体的行。

┌───────────────────────────────────────────────┐
│ Row 1: Overview                               │
│ [Solve Rate %] [Balance $] [Queue Depth] [TPM]│
├───────────────────────────────────────────────┤
│ Row 2: Performance                            │
│ [Latency P50/P95/P99]  [Solve Rate Over Time] │
├───────────────────────────────────────────────┤
│ Row 3: Errors                                 │
│ [Error Rate %]  [Error Breakdown by Type]      │
├───────────────────────────────────────────────┤
│ Row 4: Workers                                │
│ [Active Workers]  [Tasks Per Worker]           │
└───────────────────────────────────────────────┘

Prometheus 指标接入

开始前确认三件事:Prometheus/Grafana 已跑起来、求解服务能暴露 /metrics 端口、已拿到 CaptchaAI API Key。

先在验证码求解服务里暴露这些指标,Prometheus 才能抓取到。国内云主机上安装依赖,可以带上镜像参数:pip install -i https://pypi.tuna.tsinghua.edu.cn/simple prometheus_client

Python——Prometheus 客户端

import os
import time
import requests
from prometheus_client import (
    Counter, Histogram, Gauge, start_http_server
)

API_KEY = os.environ["CAPTCHAAI_API_KEY"]

# Define metrics
captcha_solves = Counter(
    "captcha_solves_total",
    "Total CAPTCHA solve attempts",
    ["captcha_type", "status"]
)
captcha_latency = Histogram(
    "captcha_solve_duration_seconds",
    "CAPTCHA solve latency",
    ["captcha_type"],
    buckets=[5, 10, 15, 20, 30, 45, 60, 90, 120, 180, 300]
)
captcha_balance = Gauge(
    "captcha_balance_dollars",
    "CaptchaAI account balance"
)
captcha_queue_depth = Gauge(
    "captcha_queue_depth",
    "Pending tasks in queue"
)
captcha_workers_active = Gauge(
    "captcha_workers_active",
    "Number of active workers"
)

session = requests.Session()


def solve_with_metrics(sitekey, pageurl, captcha_type="recaptcha_v2"):
    start = time.time()

    resp = session.post("https://ocr.captchaai.com/in.php", data={
        "key": API_KEY,
        "method": "userrecaptcha",
        "googlekey": sitekey,
        "pageurl": pageurl,
        "json": 1
    })
    data = resp.json()
    if data.get("status") != 1:
        captcha_solves.labels(captcha_type, "error").inc()
        return {"error": data.get("request")}

    captcha_id = data["request"]
    for _ in range(60):
        time.sleep(5)
        result = session.get("https://ocr.captchaai.com/res.php", params={
            "key": API_KEY, "action": "get", "id": captcha_id, "json": 1
        }).json()

        if result.get("status") == 1:
            elapsed = time.time() - start
            captcha_solves.labels(captcha_type, "success").inc()
            captcha_latency.labels(captcha_type).observe(elapsed)
            return {"solution": result["request"]}

        if result.get("request") != "CAPCHA_NOT_READY":
            captcha_solves.labels(captcha_type, "error").inc()
            return {"error": result.get("request")}

    captcha_solves.labels(captcha_type, "timeout").inc()
    return {"error": "TIMEOUT"}


def update_balance():
    resp = session.get("https://ocr.captchaai.com/res.php", params={
        "key": API_KEY, "action": "getbalance", "json": 1
    })
    if resp.json().get("status") == 1:
        captcha_balance.set(float(resp.json()["request"]))


# Start metrics server on port 9090
start_http_server(9090)

JavaScript

const promClient = require("prom-client");
const axios = require("axios");

const API_KEY = process.env.CAPTCHAAI_API_KEY;
const register = new promClient.Registry();

const solvesTotal = new promClient.Counter({
  name: "captcha_solves_total",
  help: "Total CAPTCHA solve attempts",
  labelNames: ["captcha_type", "status"],
  registers: [register],
});

const solveLatency = new promClient.Histogram({
  name: "captcha_solve_duration_seconds",
  help: "CAPTCHA solve latency",
  labelNames: ["captcha_type"],
  buckets: [5, 10, 15, 20, 30, 45, 60, 90, 120, 180, 300],
  registers: [register],
});

const balance = new promClient.Gauge({
  name: "captcha_balance_dollars",
  help: "CaptchaAI account balance",
  registers: [register],
});

const queueDepth = new promClient.Gauge({
  name: "captcha_queue_depth",
  help: "Pending tasks in queue",
  registers: [register],
});

async function solveWithMetrics(sitekey, pageurl, captchaType = "recaptcha_v2") {
  const end = solveLatency.startTimer({ captcha_type: captchaType });

  try {
    const resp = await axios.post("https://ocr.captchaai.com/in.php", null, {
      params: {
        key: API_KEY, method: "userrecaptcha",
        googlekey: sitekey, pageurl, json: 1,
      },
    });

    if (resp.data.status !== 1) {
      solvesTotal.inc({ captcha_type: captchaType, status: "error" });
      return { error: resp.data.request };
    }

    const captchaId = resp.data.request;
    for (let i = 0; i < 60; i++) {
      await new Promise((r) => setTimeout(r, 5000));
      const poll = await axios.get("https://ocr.captchaai.com/res.php", {
        params: { key: API_KEY, action: "get", id: captchaId, json: 1 },
      });
      if (poll.data.status === 1) {
        end();
        solvesTotal.inc({ captcha_type: captchaType, status: "success" });
        return { solution: poll.data.request };
      }
      if (poll.data.request !== "CAPCHA_NOT_READY") {
        solvesTotal.inc({ captcha_type: captchaType, status: "error" });
        return { error: poll.data.request };
      }
    }
    solvesTotal.inc({ captcha_type: captchaType, status: "timeout" });
    return { error: "TIMEOUT" };
  } catch (err) {
    solvesTotal.inc({ captcha_type: captchaType, status: "error" });
    throw err;
  }
}

// Expose metrics endpoint
const express = require("express");
const app = express();
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});
app.listen(9090);

两份代码逻辑一致:提交任务、每 5 秒轮询,成功/失败/超时计入计数器,延迟计入直方图。

Grafana 面板查询(PromQL)

按四行顺序取查询,复制到对应面板,图例按需调整。

第一行:总览指标

解决率(Stat 面板)

sum(rate(captcha_solves_total{status="success"}[5m]))
/
sum(rate(captcha_solves_total[5m]))

* 100

余额(Gauge 面板)

captcha_balance_dollars

队列深度(Stat 面板)

captcha_queue_depth

每分钟任务数(Stat 面板)

sum(rate(captcha_solves_total[5m])) * 60

第二行:性能指标

延迟百分位数(Time series 面板)

# p50
histogram_quantile(0.50, rate(captcha_solve_duration_seconds_bucket[5m]))

# p95
histogram_quantile(0.95, rate(captcha_solve_duration_seconds_bucket[5m]))

# p99
histogram_quantile(0.99, rate(captcha_solve_duration_seconds_bucket[5m]))

解决率趋势(Time series 面板)

sum(rate(captcha_solves_total{status="success"}[5m])) by (captcha_type) * 60

第三行:错误分析

错误率(Time series 面板)

sum(rate(captcha_solves_total{status!="success"}[5m]))
/
sum(rate(captcha_solves_total[5m]))

* 100

错误细分(Pie chart 面板)

sum by (status) (increase(captcha_solves_total{status!="success"}[1h]))

第四行:Worker 状态

活跃 Worker 数(Time series 面板)

captcha_workers_active

提示:如果同时监控多种验证码类型,把面板里写死的标签换成 $captcha_type 模板变量,一份面板就能覆盖所有类型,不用来回复制。

Grafana 告警规则配置

以下三条是基础告警:余额过低、错误率过高、延迟过高,按业务量调整阈值。跨境电商团队的欧美订单高峰常落在国内凌晨,靠告警自动提醒比人工盯屏幕更现实。

# Grafana alert rules
groups:

  - name: captcha-alerts
    rules:

      - alert: LowBalance
        expr: captcha_balance_dollars < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CaptchaAI balance low: {{ $value }}"

      - alert: HighErrorRate
        expr: |
          sum(rate(captcha_solves_total{status!="success"}[5m]))
          / sum(rate(captcha_solves_total[5m]))
          > 0.1
        for: 5m
        labels:
          severity: critical

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            rate(captcha_solve_duration_seconds_bucket[5m])
          ) > 120
        for: 10m
        labels:
          severity: warning

告警渠道怎么接

告警规则只判断阈值,通知渠道要在 Alerting → Contact points 单独配置:企业微信/钉钉 Webhook、Slack/邮件,或 PagerDuty/Opsgenie,按值班习惯接一个即可。

常见问题

把仪表板跑起来之后,这几个问题问得最多:

Prometheus 抓取间隔设多少合适?

大多数场景 15 秒够用,调用量小可以放到 30 秒;需要接近实时发现异常时不建议低于 10 秒。

一个 Grafana 实例能同时监控多个 CaptchaAI 项目吗?

可以。给指标加区分标签(比如 project="prod"),面板里用模板变量按标签筛选,不用为每个项目单独部署。

没有自建 Prometheus,能直接接入 Grafana Cloud 吗?

可以。Grafana Cloud 支持 Prometheus remote write,把指标推送过去即可,PromQL 查询语句不用改。

告警老是误报,要怎么调?

先看 for: 5m 评估窗口是不是太短——它能过滤短暂抖动;再确认 [5m] 速率窗口和抓取间隔匹配,窗口太小会放大偶发波动。

常见故障排查

先自查:curl 一下 /metrics 端口、确认 Prometheus targets 是 UP 状态、测试 Grafana 数据源连接。常见的坑基本逃不出下面这四类:

问题 原因 处理方式
面板上显示“无数据” Prometheus 没有抓取到指标端点 检查 prometheus.yml 里的目标配置;确认 /metrics 能正常返回数据
延迟百分位数不对 rate() 窗口设置错误,或者存储桶(bucket)划分不够细 [5m] 速率窗口;把存储桶划分得更细一些
仪表板变量不生效 模板变量的查询语句写错了 改用 label_values(captcha_solves_total, captcha_type)
告警一直不触发 告警评估间隔设置得太长 把评估间隔调整为 1 分钟

下一步

想让 CaptchaAI 管道的运行状况一目了然?获取 CaptchaAI API Key,把本文的 Prometheus 采集代码接入你的服务,再导入这套 Grafana 模板。

相关文章

该文章已禁用评论。