Elasticsearch分片管理 #

分片类型
├── 主分片(Primary Shard)
│   ├── 存储原始数据
│   ├── 处理写入请求
│   └── 创建时确定数量
└── 副本分片(Replica Shard)
    ├── 主分片的副本
    ├── 处理读请求
    └── 可动态调整数量

分片作用
├── 水平扩展
│   └── 数据分布到多个节点
├── 并行处理
│   └── 查询并行执行
├── 高可用
│   └── 副本提供冗余
└── 负载均衡
    └── 读写分散

分片数量规划
├── 数据量
│   └── 单分片10-50GB
├── 节点数量
│   └── 分片数≥节点数
├── 查询并发
│   └── 分片数匹配并发需求
└── 增长预期
    └── 预留扩展空间

计算公式
├── 主分片数 = 数据总量 / 单分片大小
├── 副本数 = 节点数 - 1 (最大)
└── 总分片数 = 主分片数 × (1 + 副本数)

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

PUT /products/_settings
{
  "number_of_replicas": 2
}

路由公式
shard_num = hash(_routing) % num_primary_shards

默认routing = _id

PUT /products/_doc/1?routing=apple
{
  "name": "iPhone 15",
  "brand": "Apple"
}

PUT /products
{
  "mappings": {
    "_routing": {
      "required": true
    }
  }
}

GET /products/_doc/1?routing=apple

GET /products/_search?routing=apple
{
  "query": {
    "match_all": {}
  }
}

路由优势
├── 数据局部性
│   └── 相关数据在同一分片
├── 查询效率
│   └── 只查询相关分片
└── 资源隔离
    └── 不同租户数据分离

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.allocation.cluster_concurrent_rebalance": 2,
    "cluster.routing.allocation.node_concurrent_recoveries": 2,
    "cluster.routing.allocation.node_initial_primaries_recoveries": 4
  }
}

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.include._tier_preference": "data_hot,data_warm,data_cold"
  }
}

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.require.box_type": "hot"
  }
}

PUT /products/_settings
{
  "index.routing.allocation.include._tier_preference": "data_hot"
}

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.rebalance.enable": "all",
    "cluster.routing.balance.shard": 0.45,
    "cluster.routing.balance.index": 0.55,
    "cluster.routing.balance.threshold": 1.0
  }
}

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.rebalance.enable": "none"
  }
}

PUT /products/_settings
{
  "index.routing.rebalance.enable": "none"
}

PUT /_cluster/settings
{
  "persistent": {
    "indices.recovery.max_bytes_per_sec": "100mb",
    "indices.recovery.max_concurrent_file_chunks": 2,
    "indices.recovery.max_concurrent_operations": 1
  }
}

GET /_cat/recovery?v

GET /products/_recovery

GET /_cat/shards?v&state=UNASSIGNED

GET /_cluster/allocation/explain
{
  "index": "products",
  "shard": 0,
  "primary": true
}

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "products",
        "shard": 0,
        "node": "node-1",
        "accept_data_loss": true
      }
    }
  ]
}

未分配原因
├── 节点离线
├── 磁盘空间不足
├── 副本数过多
├── 分配规则限制
└── 分片损坏

GET /_cat/shards?v

GET /_cat/shards/products?v

GET /_stats

GET /products/_stats

GET /_cat/segments?v

GET /products/_segments

分片规划建议
├── 单分片大小
│   └── 10-50GB
├── 分片数量
│   └── 每节点< 20个/GB堆
├── 副本数量
│   └── 生产至少1个
└── 预留扩展
    └── 考虑数据增长

路由建议
├── 使用业务字段路由
│   └── 如用户ID、租户ID
├── 必须路由
│   └── 设置required: true
├── 查询时指定路由
│   └── 提高查询效率
└── 避免热点
    └── 合理选择路由字段