分布式部署 #

本章详细介绍 Weaviate 的分布式部署和高可用配置。

分布式架构 #

text 
Weaviate 集群架构：

┌─────────────────────────────────────────────────────────────┐
│                      客户端请求                              │
└─────────────────────────────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                      负载均衡器                              │
└─────────────────────────────────────────────────────────────┘
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
┌───────────────┐  ┌───────────────┐  ┌───────────────┐
│   Node 1      │  │   Node 2      │  │   Node 3      │
│  ┌─────────┐  │  │  ┌─────────┐  │  │  ┌─────────┐  │
│  │ Weaviate│  │  │  │ Weaviate│  │  │  │ Weaviate│  │
│  └─────────┘  │  │  └─────────┘  │  │  └─────────┘  │
│  ┌─────────┐  │  │  ┌─────────┐  │  │  ┌─────────┐  │
│  │  Data   │  │  │  │  Data   │  │  │  │  Data   │  │
│  │ Shard 1 │  │  │  │ Shard 2 │  │  │  │ Shard 3 │  │
│  └─────────┘  │  │  └─────────┘  │  │  └─────────┘  │
└───────────────┘  └───────────────┘  └───────────────┘
        │                  │                  │
        └──────────────────┼──────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                    共识层 (Raft)                             │
└─────────────────────────────────────────────────────────────┘

分片与复制 #

分片配置 #

python 
import weaviate.classes as wvc

articles = client.collections.create(
    name="Article",
    sharding_config=wvc.config.Configure.sharding(
        virtual_per_physical=128,
        desired_count=4,
        strategy=wvc.config.ShardingStrategy.HASH
    ),
    properties=[
        wvc.config.Property(name="title", data_type=wvc.config.DataType.TEXT),
        wvc.config.Property(name="content", data_type=wvc.config.DataType.TEXT)
    ]
)

分片参数说明 #

text 
分片参数：

desired_count:
├── 物理分片数量
├── 建议与节点数相同
└── 默认值: 1

virtual_per_physical:
├── 每个物理分片的虚拟分片数
├── 影响数据分布均匀性
└── 默认值: 128

strategy:
├── HASH: 基于哈希的分片
└── 默认: HASH

复制配置 #

python 
articles = client.collections.create(
    name="Article",
    replication_config=wvc.config.Configure.replication(
        factor=3,
        async_enabled=True
    ),
    properties=[
        wvc.config.Property(name="title", data_type=wvc.config.DataType.TEXT),
        wvc.config.Property(name="content", data_type=wvc.config.DataType.TEXT)
    ]
)

复制参数说明 #

text 
复制参数：

factor:
├── 副本数量
├── 建议值: 3 (生产环境)
└── 默认值: 1

async_enabled:
├── 启用异步复制
├── 提高写入性能
└── 默认值: False

Kubernetes 部署 #

Helm 部署 #

添加 Helm 仓库：

bash 
helm repo add weaviate https://weaviate.github.io/weaviate-helm
helm repo update

创建 values.yaml：

yaml 
image:
  tag: 1.25.0

replicas: 3

env:
  - name: QUERY_DEFAULTS_LIMIT
    value: "25"
  - name: AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED
    value: "true"
  - name: PERSISTENCE_DATA_PATH
    value: "/var/lib/weaviate"
  - name: DEFAULT_VECTORIZER_MODULE
    value: "text2vec-openai"
  - name: ENABLE_MODULES
    value: "text2vec-openai"
  - name: OPENAI_APIKEY
    valueFrom:
      secretKeyRef:
        name: weaviate-secrets
        key: openai-api-key

persistence:
  enabled: true
  size: 100Gi
  storageClass: "standard"

resources:
  requests:
    cpu: "1000m"
    memory: "4Gi"
  limits:
    cpu: "4000m"
    memory: "16Gi"

service:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: grpc
      port: 50051
      targetPort: 50051

部署：

bash 
kubectl create namespace weaviate

kubectl create secret generic weaviate-secrets \
  --from-literal=openai-api-key=your-api-key \
  -n weaviate

helm install weaviate weaviate/weaviate \
  -n weaviate \
  -f values.yaml

验证集群 #

bash 
kubectl get pods -n weaviate
kubectl get svc -n weaviate

kubectl port-forward svc/weaviate 8080:80 -n weaviate

curl http://localhost:8080/v1/nodes

节点状态查询 #

python 
import weaviate

client = weaviate.connect_to_local()

nodes = client.cluster.nodes()

for node in nodes:
    print(f"Node: {node.name}")
    print(f"  Status: {node.status}")
    print(f"  Version: {node.version}")
    print(f"  GitHash: {node.git_hash}")

Docker Compose 集群 #

多节点配置 #

yaml 
version: '3.8'
services:
  weaviate-node-1:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.0
    ports:
      - "8080:8080"
      - "50051:50051"
    environment:
      CLUSTER_HOSTNAME: 'node1'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
      CLUSTER_JOIN: 'weaviate-node-2:7100,weaviate-node-3:7100'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
    volumes:
      - weaviate_data_1:/var/lib/weaviate

  weaviate-node-2:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.0
    ports:
      - "8081:8080"
      - "50052:50051"
    environment:
      CLUSTER_HOSTNAME: 'node2'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
      CLUSTER_JOIN: 'weaviate-node-1:7100,weaviate-node-3:7100'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
    volumes:
      - weaviate_data_2:/var/lib/weaviate

  weaviate-node-3:
    image: cr.weaviate.io/semitechnologies/weaviate:1.25.0
    ports:
      - "8082:8080"
      - "50053:50051"
    environment:
      CLUSTER_HOSTNAME: 'node3'
      CLUSTER_GOSSIP_BIND_PORT: '7100'
      CLUSTER_DATA_BIND_PORT: '7101'
      CLUSTER_JOIN: 'weaviate-node-1:7100,weaviate-node-2:7100'
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
    volumes:
      - weaviate_data_3:/var/lib/weaviate

volumes:
  weaviate_data_1:
  weaviate_data_2:
  weaviate_data_3:

一致性级别 #

一致性配置 #

python 
import weaviate.classes as wvc

articles = client.collections.get("Article")

uuid = articles.data.insert(
    properties={"title": "测试文章", "content": "内容..."},
    consistency_level=wvc.config.ConsistencyLevel.ALL
)

一致性级别说明 #

text 
一致性级别：

ONE:
├── 只需一个节点确认
├── 最快，但可能不一致
├── 适合高吞吐量场景
└── 最终一致性

QUORUM (默认):
├── 需要多数节点确认
├── 平衡性能和一致性
├── 推荐大多数场景
└── 强一致性

ALL:
├── 需要所有节点确认
├── 最强一致性，最慢
├── 适合关键数据
└── 强一致性

读取一致性 #

python 
article = articles.query.fetch_object_by_id(
    uuid=uuid,
    consistency_level=wvc.config.ConsistencyLevel.QUORUM
)

备份与恢复 #

创建备份 #

python 
from datetime import datetime

backup_id = f"backup-{datetime.now().strftime('%Y%m%d-%H%M%S')}"

result = client.backup.create(
    backup_id=backup_id,
    backend="filesystem",
    include_classes=["Article", "Author"],
    wait_for_completion=True
)

print(f"Backup status: {result.status}")

恢复备份 #

python 
result = client.backup.restore(
    backup_id=backup_id,
    backend="filesystem",
    wait_for_completion=True
)

print(f"Restore status: {result.status}")

云存储备份 #

python 
result = client.backup.create(
    backup_id=backup_id,
    backend="s3",
    include_classes=["Article"],
    s3_config=wvc.backup.S3BackupConfig(
        bucket="your-bucket",
        path="backups/weaviate",
        endpoint="s3.amazonaws.com",
        region="us-east-1"
    ),
    wait_for_completion=True
)

高可用配置 #

节点故障处理 #

text 
节点故障处理流程：

1. 检测故障
   └── 心跳检测失败

2. 重新分片
   └── 将分片迁移到健康节点

3. 数据同步
   └── 从副本恢复数据

4. 服务恢复
   └── 集群恢复正常

最小节点数 #

text 
最小节点数建议：

复制因子 = 3:
├── 最小节点数: 3
├── 容忍故障节点: 1
└── 推荐节点数: 5

复制因子 = 5:
├── 最小节点数: 5
├── 容忍故障节点: 2
└── 推荐节点数: 7

监控与运维 #

集群健康检查 #

python 
nodes = client.cluster.nodes()

for node in nodes:
    print(f"Node: {node.name}")
    print(f"  Status: {node.status}")
    
    for shard in node.shards:
        print(f"  Shard: {shard.name}")
        print(f"    Class: {shard.class_name}")
        print(f"    Object Count: {shard.object_count}")

分片状态 #

python 
articles = client.collections.get("Article")

shards = articles.config.get_shards()

for shard in shards:
    print(f"Shard: {shard.name}")
    print(f"  Status: {shard.status}")
    print(f"  Object Count: {shard.object_count}")
    print(f"  Node: {shard.node_name}")

性能优化 #

分片策略 #

text 
分片策略建议：

数据量 < 1000 万:
├── 分片数: 与节点数相同
└── 复制因子: 3

数据量 1000 万 - 1 亿:
├── 分片数: 节点数 × 2
└── 复制因子: 3

数据量 > 1 亿:
├── 分片数: 节点数 × 4
└── 复制因子: 3

资源配置 #

yaml 
resources:
  requests:
    cpu: "2000m"
    memory: "8Gi"
  limits:
    cpu: "8000m"
    memory: "32Gi"

persistence:
  size: 500Gi

小结 #

本章介绍了 Weaviate 的分布式部署：

  • 集群架构
  • 分片与复制配置
  • Kubernetes 部署
  • Docker Compose 集群
  • 一致性级别
  • 备份与恢复
  • 高可用配置

下一步 #

继续学习 监控与运维，了解如何监控和维护 Weaviate 集群！

最后更新：2026-04-04