一、存在问题
用了一段时间elk发现如果索引长时间不删除,elk会越来越慢,重启elasticsearch服务器节点之前同步时间也会很长
二、解决方法(定期删除索引)
1.在elasticsearch节点上使用curl -XGET 'http://192.168.X.XX:9200/_cat/shards'查看索引
[root@192-168-x-x scripts]# curl -XGET 'http://192.168.x.x:9200/_cat/shards' | more % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0freetrip-2019.01.29 2 r STARTED 34 59kb 192.168.x.x node3-192.168.x.x freetrip-2019.01.29 2 p STARTED 34 59kb 192.168.x.x node1-192.168.x.x freetrip-2019.01.29 1 p STARTED 47 122.7kb 192.168.x.x node3-192.168.x.x freetrip-2019.01.29 1 r STARTED 47 122.7kb 192.168.x.x node1-192.168.x.x freetrip-2019.01.29 3 p STARTED 58 90.5kb 192.168.x.x node2-192.168.x.x freetrip-2019.01.29 3 r STARTED 58 90.5kb 192.168.x.x node1-192.168.x.x freetrip-2019.01.29 4 r STARTED 41 45.5kb 192.168.x.x node2-192.168.x.x freetrip-2019.01.29 4 p STARTED 41 45.5kb 192.168.x.x node3-192.168.x.x freetrip-2019.01.29 0 p STARTED 36 97.9kb 192.168.x.x node2-192.168.x.x freetrip-2019.01.29 0 r STARTED 36 97.9kb 192.168.x.x node3-192.168.x.x
2.将过滤出来要删除的索引存到临时文件夹里面
[root@192-168-x-x ~]# curl -XGET 'http://192.168.x.x:9200/_cat/shards' | grep 192.168.x.x | grep 2018 | awk '{print $1}' |uniq > elk-index.tmp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 81832 100 81832 0 0 142k 0 --:--:-- --:--:-- --:--:-- 142k [root@192-168-3-163 ~]# cat elk-index.tmp otter-canal-195-2018.12.10 logstash-nginx-chm-accesslog-2018.11.20 logstash-nginx-oms-accesslog-2018.10.14 laravel-chm248-online-2018.12.10 otter-canal-195-2018.11.20 logstash-nginx-oms-accesslog-2018.11.20 otter-communication-195-2018.11.23 otter-communication-195-2018.12.10 otter-node-195-2018.12.10 logstash-nginx-chvisa-accesslog-2018.10.14 wanmei219-online-2018.10.14 logstash-nginx-chm-accesslog-2018.11.19 otter-canal-195-2018.11.23 otter-canal-195-2018.11.19 logstash-nginx-oms-accesslog-2018.11.19 laravel-chm248-online-2018.11.19 logstash-nginx-oms-accesslog-2018.12.10 otter-manager-195-2018.12.10 logstash-nginx-chvisa-accesslog-2018.11.20 wanmei219-online-2018.12.10 logstash-nginx-chvisa-accesslog-2018.11.19 otter-communication-195-2018.11.19 otter-communication-195-2018.11.20 laravel-chm248-online-2018.11.20 laravel-chm248-online-2018.10.14 logstash-nginx-chm-accesslog-2018.12.10 wanmei219-online-2018.11.19 logstash-nginx-chvisa-accesslog-2018.12.10 wanmei219-online-2018.11.20 logstash-nginx-chm-accesslog-2018.10.14
3. 删除过滤出来的索引文件
for i in `cat elk-index.tmp` do curl -XDELETE http://192.168.3.163:9200/$i done
4. 使用脚本加定时任务,每一天删除前三天的索引
[root@192-168-x-x ~]# cat /home/scripts/del_elasticseatch_index.sh #!/bin/bash curl -XGET 'http://192.168.x.x:9200/_cat/shards' |grep 192.168.x.x | awk '{print $1}' |grep `date -d "5 days ago" +%Y.%m.%d` | uniq > /tmp/index_name.tmp for index_name in `cat /tmp/index_name.tmp` do echo $index_name curl -XDELETE http://192.168.x.x:9200/$index_name echo "${index_name} delete success" >> /home/scripts/del_elasticseatch_index.log done
5. 定时任务
[root@192-168-x-x ~]# crontab -l 0 3 * * * bash /home/scripts/del_elasticseatch_index.sh