3. 压测环境搭建：Docker Compose 部署 3 节点 etcd 集群、配置 TLS 与认证、Prometheus + Grafana 监控栈部署

压测环境是瓶颈定位工程的基石。本章将使用 Docker Compose 在单机模拟生产级 3 节点 etcd 集群，配置双向 TLS 与 RBAC 认证，并部署 Prometheus + Grafana 监控栈。所有组件均通过容器化编排，确保环境可复现、可销毁。

3.1 整体架构与组件关系

下图展示了压测沙盘的完整组件拓扑。3 个 etcd 节点组成 Raft 集群，对外暴露客户端端口（2379）和 peer 通信端口（2380）。Prometheus 通过 etcd 的 /metrics 端点采集指标，Grafana 作为可视化面板连接 Prometheus 数据源。压测工具（如 etcdctl、benchmark）从外部向集群注入读写负载。

压测沙盘组件拓扑 etcd-0 (Leader) etcd-1 (Follower) etcd-2 (Follower) Client :2379 Client :2379 Client :2379 Peer :2380 Peer :2380 Peer :2380 Prometheus Grafana 数据源 压测工具 (etcdctl / benchmark) 图例 etcd 节点 监控组件

3.2 Docker Compose 编排文件

使用 Docker Compose 定义 3 个 etcd 服务、Prometheus 和 Grafana。每个 etcd 节点通过环境变量配置集群成员、数据目录、TLS 证书路径和认证参数。

关键设计原则： 每个 etcd 容器使用 host 网络模式（或固定 IP）以简化 peer 通信；TLS 证书通过卷挂载共享；Prometheus 使用 file_sd_configs 动态发现 etcd 节点。 
version: '3.8'

services:
  etcd-0:
    image: quay.io/coreos/etcd:v3.5.15
    container_name: etcd-0
    network_mode: host
    volumes:
      - ./certs:/etc/etcd/certs:ro
      - ./data/etcd-0:/var/lib/etcd
    environment:
      ETCD_NAME: etcd-0
      ETCD_DATA_DIR: /var/lib/etcd
      ETCD_INITIAL_CLUSTER: etcd-0=https://192.168.1.10:2380,etcd-1=https://192.168.1.11:2380,etcd-2=https://192.168.1.12:2380
      ETCD_INITIAL_CLUSTER_STATE: new
      ETCD_INITIAL_CLUSTER_: etcd-cluster-1
      ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
      ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
      ETCD_ADVERTISE_CLIENT_URLS: https://192.168.1.10:2379
      ETCD_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_CERT_FILE: /etc/etcd/certs/server.pem
      ETCD_KEY_FILE: /etc/etcd/certs/server-key.pem
      ETCD_CLIENT_CERT_AUTH: "true"
      ETCD_PEER_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_PEER_CERT_FILE: /etc/etcd/certs/peer.pem
      ETCD_PEER_KEY_FILE: /etc/etcd/certs/peer-key.pem
      ETCD_PEER_CLIENT_CERT_AUTH: "true"
      ETCD_AUTH_: jwt,secret=my-secret-key
    restart: unless-stopped

  etcd-1:
    image: quay.io/coreos/etcd:v3.5.15
    container_name: etcd-1
    network_mode: host
    volumes:
      - ./certs:/etc/etcd/certs:ro
      - ./data/etcd-1:/var/lib/etcd
    environment:
      ETCD_NAME: etcd-1
      ETCD_DATA_DIR: /var/lib/etcd
      ETCD_INITIAL_CLUSTER: etcd-0=https://192.168.1.10:2380,etcd-1=https://192.168.1.11:2380,etcd-2=https://192.168.1.12:2380
      ETCD_INITIAL_CLUSTER_STATE: new
      ETCD_INITIAL_CLUSTER_: etcd-cluster-1
      ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
      ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
      ETCD_ADVERTISE_CLIENT_URLS: https://192.168.1.11:2379
      ETCD_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_CERT_FILE: /etc/etcd/certs/server.pem
      ETCD_KEY_FILE: /etc/etcd/certs/server-key.pem
      ETCD_CLIENT_CERT_AUTH: "true"
      ETCD_PEER_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_PEER_CERT_FILE: /etc/etcd/certs/peer.pem
      ETCD_PEER_KEY_FILE: /etc/etcd/certs/peer-key.pem
      ETCD_PEER_CLIENT_CERT_AUTH: "true"
      ETCD_AUTH_: jwt,secret=my-secret-key
    restart: unless-stopped

  etcd-2:
    image: quay.io/coreos/etcd:v3.5.15
    container_name: etcd-2
    network_mode: host
    volumes:
      - ./certs:/etc/etcd/certs:ro
      - ./data/etcd-2:/var/lib/etcd
    environment:
      ETCD_NAME: etcd-2
      ETCD_DATA_DIR: /var/lib/etcd
      ETCD_INITIAL_CLUSTER: etcd-0=https://192.168.1.10:2380,etcd-1=https://192.168.1.11:2380,etcd-2=https://192.168.1.12:2380
      ETCD_INITIAL_CLUSTER_STATE: new
      ETCD_INITIAL_CLUSTER_: etcd-cluster-1
      ETCD_LISTEN_PEER_URLS: https://0.0.0.0:2380
      ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:2379
      ETCD_ADVERTISE_CLIENT_URLS: https://192.168.1.12:2379
      ETCD_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_CERT_FILE: /etc/etcd/certs/server.pem
      ETCD_KEY_FILE: /etc/etcd/certs/server-key.pem
      ETCD_CLIENT_CERT_AUTH: "true"
      ETCD_PEER_TRUSTED_CA_FILE: /etc/etcd/certs/ca.pem
      ETCD_PEER_CERT_FILE: /etc/etcd/certs/peer.pem
      ETCD_PEER_KEY_FILE: /etc/etcd/certs/peer-key.pem
      ETCD_PEER_CLIENT_CERT_AUTH: "true"
      ETCD_AUTH_: jwt,secret=my-secret-key
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v2.53.0
    container_name: prometheus
    network_mode: host
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./data/prometheus:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.listen-address=0.0.0.0:9090'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.1.0
    container_name: grafana
    network_mode: host
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_INSTALL_PLUGINS: ""
    volumes:
      - ./data/grafana:/var/lib/grafana
    restart: unless-stopped

3.3 TLS 证书生成与配置

etcd 要求客户端与节点之间、节点与节点之间的通信都使用 TLS。使用 cfssl 工具生成自签名 CA 证书、服务端证书和 peer 证书。

3.3.1 生成 CA 证书

# 生成 CA 配置文件
cat <<EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "server": {
        "usages": ["signing", "key encipherment", "server auth"],
        "expiry": "87600h"
      },
      "peer": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "87600h"
      },
      "client": {
        "usages": ["signing", "key encipherment", "client auth"],
        "expiry": "87600h"
      }
    }
  }
}
EOF

# 生成 CA 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

3.3.2 生成服务端与 Peer 证书

服务端证书需要包含所有 etcd 节点的 IP 地址作为 SAN。Peer 证书额外需要 client auth 用途以支持双向认证。

# server-csr.json 示例（包含 3 个节点 IP）
{
  "CN": "etcd-server",
  "hosts": [
    "192.168.1.10",
    "192.168.1.11",
    "192.168.1.12",
    "127.0.0.1",
    "localhost"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {"C": "CN", "ST": "Beijing", "L": "Beijing", "O": "etcd", "OU": "Server"}
  ]
}

# 生成服务端证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
  -profile=server server-csr.json | cfssljson -bare server

# 生成 peer 证书（使用 peer profile）
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
  -profile=peer server-csr.json | cfssljson -bare peer
常见陷阱： 证书 SAN 中必须包含所有 etcd 节点的实际 IP 地址，否则节点间 TLS 握手会失败。如果使用 hostname，需要确保 DNS 解析正确。生产环境建议使用内部 DNS 名称。

3.4 启用 RBAC 认证

etcd 支持基于角色的访问控制（RBAC）。在集群启动后，通过 etcdctl 创建 root 用户和压测专用用户。

# 添加 root 用户
etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  user add root:SuperSecretPassw0rd

# 启用认证
etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  auth enable

# 创建压测用户并授予读写权限
etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  --user=root:SuperSecretPassw0rd \
  user add benchmark:BenchPass123

etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  --user=root:SuperSecretPassw0rd \
  role add benchmark-role

etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  --user=root:SuperSecretPassw0rd \
  role grant-permission benchmark-role readwrite --prefix=true /

etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=/etc/etcd/certs/ca.pem \
  --cert=/etc/etcd/certs/client.pem \
  --key=/etc/etcd/certs/client-key.pem \
  --user=root:SuperSecretPassw0rd \
  user grant-role benchmark benchmark-role

3.5 Prometheus 配置与指标采集

Prometheus 通过静态配置或文件服务发现采集 etcd 指标。etcd 暴露的 /metrics 端点包含 Raft 状态、磁盘 IO、网络延迟等关键指标。

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'etcd'
    scheme: https
    tls_config:
      ca_file: /etc/etcd/certs/ca.pem
      cert_file: /etc/etcd/certs/client.pem
      key_file: /etc/etcd/certs/client-key.pem
      insecure_skip_verify: false
    static_configs:
      - targets:
        - '192.168.1.10:2379'
        - '192.168.1.11:2379'
        - '192.168.1.12:2379'
        labels:
          group: 'etcd-cluster'

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
指标采集验证： 启动后访问 https://192.168.1.10:2379/metrics（使用 curl 并指定客户端证书），应看到大量以 etcd_ 开头的指标，如 etcd_server_leader_changes_seen_total、etcd_disk_wal_fsync_duration_seconds。

3.6 Grafana 数据源与仪表盘导入

Grafana 通过 Prometheus 数据源查询 etcd 指标。推荐导入 etcd 官方仪表盘（ID: 3070）或自定义压测专用仪表盘。

步骤 操作 说明
1 登录 Grafana 浏览器访问 http://192.168.1.10:3000，默认账号 admin/admin
2 添加 Prometheus 数据源 Configuration → Data Sources → Add data source → Prometheus，URL 填写 http://localhost:9090
3 导入仪表盘 + → Import → 输入 ID 3070（etcd 官方仪表盘）或上传自定义 JSON
4 验证指标 在 Explore 页面执行查询 etcd_server_has_leader，应返回 1

3.7 启动与验证流程

按照以下步骤启动整个压测环境并验证集群健康状态。

  1. 生成证书： 执行 cfssl 命令生成 ca.pem、server.pem、peer.pem 等文件，放入 ./certs/ 目录。
  2. 创建数据目录： mkdir -p ./data/etcd-{0,1,2} ./data/prometheus ./data/grafana
  3. 启动所有服务： docker-compose up -d
  4. 检查 etcd 集群成员：
etcdctl --endpoints=https://192.168.1.10:2379 \
  --cacert=./certs/ca.pem \
  --cert=./certs/client.pem \
  --key=./certs/client-key.pem \
  endpoint health --cluster

# 预期输出：
# https://192.168.1.10:2379 is healthy: successfully committed proposal: took = 2.345ms
# https://192.168.1.11:2379 is healthy: successfully committed proposal: took = 3.012ms
# https://192.168.1.12:2379 is healthy: successfully committed proposal: took = 2.891ms
  1. 验证 Prometheus 目标状态： 访问 http://192.168.1.10:9090/targets，确认 etcd 三个 target 均为 UP 状态。
  2. 验证 Grafana 仪表盘： 打开导入的仪表盘，确认 Raft leader、磁盘 fsync 延迟、gRPC 请求速率等面板有数据。

3.8 环境参数速查表

组件 端口 协议 认证方式 关键配置项
etcd-0 2379 / 2380 HTTPS 双向 TLS + RBAC ETCD_INITIAL_CLUSTER, ETCD_CERT_FILE
etcd-1 2379 / 2380 HTTPS 双向 TLS + RBAC ETCD_INITIAL_CLUSTER, ETCD_CERT_FILE
etcd-2 2379 / 2380 HTTPS 双向 TLS + RBAC ETCD_INITIAL_CLUSTER, ETCD_CERT_FILE
Prometheus 9090 HTTP 无（内网） scrape_configs, tls_config
Grafana 3000 HTTP admin/admin 数据源指向 Prometheus

3.9 常见问题与排障

排障清单：
  • 集群无法形成： 检查 peer 端口（2380）是否可达，证书 SAN 是否包含所有节点 IP，initial-cluster- 是否一致。
  • TLS 握手失败： 使用 openssl s_client -connect 192.168.1.10:2379 -cert client.pem -key client-key.pem -CAfile ca.pem 测试。
  • Prometheus 采集报错： 检查 tls_config 中的证书路径是否正确，etcd 的 /metrics 端点是否启用了 client cert auth。
  • Grafana 无数据： 确认 Prometheus 数据源配置正确，在 Explore 中手动执行 up{job="etcd"} 验证。
← 上一章 📖 返回目录 下一章 →