Notes
  • Introduce
  • Go
    • Grammar
      • Basic
      • Goroutines & Channels
      • Test
    • System Library
      • Module
      • sync
      • context
      • net
    • Concurrency in Go
    • The Go Memory Model
    • Code Snippet
  • Rust
    • The Rust Programming Language
    • Rust by Example
  • JAVA
    • Preface
    • Grammar
      • Basic
      • Data Types
      • Operator
      • Exceptions
    • Class Libraries
      • Collection
      • Stream
      • IO
      • NIO
      • RMI
    • Concurrency
      • Preface
      • JMM
      • Synchronized & CAS
      • Deadlock
      • Thread
      • Lock & Condition
      • Utility Class
      • Thread-safe Collection
      • Atomic Class
      • Fork/Join
      • Concurrency Design Patterns
        • Immutable
        • Copy-on-Write
        • ThreadLocal
        • Multitheading If
        • Division
    • JVM
      • Class & Instance Initialization
      • Runtime Data Area
      • Garbage Collection
    • Web Container
      • Tomcat Architecture
      • Jetty Architecture
    • Spring
    • Tuning
      • Programming
  • Computer Science
    • Computer Organization
    • Algorithm
      • Complexity
      • Linear List
      • Sort
      • Binary Search
      • Skip List
      • Hash Table
      • Tree
      • Graph
      • String Matching
      • Bloom Filter
      • Greedy Algorithm
      • Divide and Conquer
      • Back Tracking
      • Dynamic Programming
    • Network Protocol
      • Pysical Layer
      • Data Link Layer
      • Network Layer
      • Transport Layer
      • Application layer
      • HTTP
      • HTTP/2 in Action
    • Operating System
      • Basic
      • System Initialization
      • Diagnostic Tools
      • CPU Diagnosis
      • Memory Diagnosis
      • Disk Diagnosis
      • Network Diagnosis
      • Monitor System
    • Design Patterns
      • UML
      • OOP
      • Principle
      • Refactoring & Specification
      • Creational
        • Singleton
        • Factory
        • Builder
        • Prototype
      • Structural
        • Proxy
        • Bridge
        • Decorator
        • Adapter
        • Facade
        • Composite
        • FlyWeight
      • Behavioral
        • Observer
        • Template Method
        • Strategy
        • State
        • Iterator
        • Chain of Responsibility
    • Distributed System
      • Protocol & Algorithm
      • Transcation
      • Theory
      • Resource Management
      • Scheduling
      • Computing
      • Message Queue
      • Cache
      • Consistent Hashing
  • database
    • InfluxDB
      • In-Memory Index
      • Meta
    • MySQL
      • SQL
      • Architecture
      • Log
      • Transaction
      • Indexing
      • Lock
      • Storage
    • Redis
    • Elasticsearch
      • Local Debug
    • HBase
    • Kafka
    • ZooKeeper
  • Reading
    • RocketMQ
    • 演说之禅
    • So Good They Can't Ignore You
    • 学会提问
    • Lecture
  • Other
    • v2ray
    • Kubernetes
    • Git
    • Maven
    • Anaconda And Conda
    • Fuck! Shit!
      • Remove Final by Reflection
      • Ingress Host
      • ExecuterService submit
  • Open source contribution
Powered by GitBook
On this page
  • 1. API
  • 滚动升级集群
  • Query API
  • Cat API
  • Index API
  • Cluster API
  • Shard Allocation
  • 2. Tuning

Was this helpful?

  1. database

Elasticsearch

PreviousRedisNextLocal Debug

Last updated 5 years ago

Was this helpful?

  • 选主流程分析

1. API

Elasticsearch 的 API 在上有很详细的介绍,我这里仅列举出平时用到的较多的 API,方便查询。

滚动升级集群

# 首先关闭 shard 的分配
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}

# 然后写入硬盘
POST _flush/synced

# 升级完成后打开
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}

Query API

# 时间范围查询
GET element-quantity-2/_search
{
  "query": {
    "range": {
      "timestamp": {
       "lte": "2018.11.13",
       "format": "yyyy.MM.dd"
      }
    }
  }
}

# 值的种类查询
GET element-quantity-2/_search
{
  "size": 0, 
  "query": {
    "match_all": {}
  },
  "aggs": {
    "count": {
      "cardinality": {
        "field": "databag_id"
      }
    }
  }
}

Cat API

# 通用参数
## 显示标题
?v

## 选择显示哪些列
?h=A,B,C

## 排序
?s=A:desc,B

# 获取素有的节点
GET _cat/nodes

# 获取所有的 index
GET _cat/indices?v

# 获取符合某个 pattern 的 index 的 shard
GET _cat/shards/${pattern}

# 获取所有的 shard,按照 state 排序
GET _cat/shards?v&s=state

# 获取所有的 shard,显示指定列
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason

# 获取特定类型的 thread_pool
GET /_cat/thread_pool/bulk,flush,refresh?v&h=node_name,name,active,queue,rejected,completed
​
# 获取所有的 thread_pool
GET /_cat/thread_pool?v&h=node_name,name,active,queue,rejected,completed

Index API

# 关闭 Index
POST logstash-bfaccesslog-2018.12/_close

# 删除 Index,匹配格式
DELETE .monitoring-*

# 统一修改所有 index 的配置
PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}

# 修改某个 index 的配置
PUT portal/_settings
{
  "index": {
    "number_of_replicas":1
  }
}

# 获取所有 index 的配置
GET _settings

# 获取所有 template
GET _template

# 增加自己的 template
PUT _template/my_logstash
{
  "index_patterns": ["logstash-*"],
  "order": 1,
  "settings": {
    "number_of_shards": 1
  }
}

# 当 node 丢失的时候,shard 重新分配延迟时间
PUT _all/_settings
{
  "settings": {
    "index.unassigned.node_left.delayed_timeout": "5m"
  }
}

Cluster API

# 获取所有关闭的 index
GET _cluster/state/blocks?pretty

# 获取集群的配置
GET /_cluster/settings

# 分析 Shard 分配问题
GET /_cluster/allocation/explain

# 获取各个节点上的 task
GET _tasks

# 获取所有数据节点和存储空间
GET _cat/allocation?v

Shard Allocation

  • cluster.routing.allocation.enable

    • all - (default) Allows shard allocation for all kinds of shards.

    • primaries - Allows shard allocation only for primary shards.

    • new_primaries - Allows shard allocation only for primary shards for new indices.

    • none - No shard allocations of any kind are allowed for any indices.

  • cluster.routing.rebalance.enable

    • all - (default) Allows shard balancing for all kinds of shards.

    • primaries - Allows shard balancing only for primary shards.

    • replicas - Allows shard balancing only for replica shards.

    • none - No shard balancing of any kind are allowed for any indices.

  • cluster.routing.allocation.allow_rebalance

    • always - Always allow rebalancing.

    • indices_primaries_active - Only when all primaries in the cluster are allocated.

    • indices_all_active - (default) Only when all shards (primaries and replicas) in the cluster are allocated.

  • cluster.routing.allocation.disk.threshold_enabled

    • true - default

    • false

  • index.unassigned.node_left.delayed_timeout

    • 见 Index API。

    • 注意:此配置是 index 级别的,所以就算配置的时候指定为_all,新建的 index 也不会有这个配置。可以用 template 的方式增加此配置。

2. Tuning

  • index.refresh_interval:由 Buffer 写入 Segment(在 OS cache 中) 的频率,默认 1s。

  • index.translog.sync_interval:默认 5s,表示 translog 由 OS cache 写入硬盘的频率。需要 close index 才能修改。

对于已存在的 index:

PUT .monitoring-*/_settings
{
  "index": {
  	"refresh_interval": "30s",
    "translog.durability": "async",
    "merge.scheduler.max_thread_count": 1
  }
}

对于以后创建的 index:

  • order:数值越大越后执行,会覆盖数值小的。

PUT _template/my_config
{
  "index_patterns": ["*"],
  "order" : 1, 
  "settings": {
    "index": {
        "refresh_interval": "30s",
        "translog.durability": "async",
        "translog.sync_interval": "100s",
        "merge.scheduler.max_thread_count": 1
      }
  }
}

参考 、。

index.merge.scheduler.max_thread_count:merge 操作的最大线程,默认有个公式,详情。

index.translog.durability:默认 request,表示同步写入,参考。

本地调试 Elasticsearch
官方文档
Cluster level shard allocation
Disk-based shard allocation
参考官方
官方文档