2016-04-15 97 views
0

我经过大约370次备份(大约15天)后每小时有一次Elasticsearch备份,我的备份存储库大于15G!但总指数的大小只有500M左右! Elasticsearch是增量备份,但15G VS 500M,差别非常巨大!我想知道索引和备份存储库之间是否有这么大的差异是正常的?备份存储库大小远大于索引大小

是否由我经常备份(每小时)造成的?我使用群集1中的每小时备份和群集2中的每小时还原来实时保持两个ES群集数据相同。

我使用Elasticsearch备份API来备份

  1. 设置回购: 卷曲-XPUT “http://IP:9200/_snapshot/backup” -d “{\” 类型\ “:\” FS \”,\ “设置\” :{\ “压缩\”:真,\ “位置\”:\ “备份\”}}”

  2. 备份: CURTIME = date +"%Y-%m-%d %H:%M:%S" BKTIME = $ {CURTIME // [ - :] /} curl -XPUT“http://IP:9200/snapshot/backup/snapshot $ BKTIME?wait_for_completion = true”

我Elasticsearch设置:2个节点,12碎片/节点,2个索引,备份快照存储到NAS

在Elasticsearch数据目录

的FS型,指数大小:

node1的索引尺寸: [root @ esnode1 indices] $ du -sh

307M。

节点2指数大小 [根@ esnode2指数] $杜-SH

238M。

[根@ esnode1指数] $杜-LH

8.0K ./index1/10/translog 8.0K ./index1/10/_state 2.9M ./index1/10/index 2.9M ./index1/10 12K ./index1/5/translog 8.0K ./index1/5/_state 1.5M ./index1/5/index 1.5M ./index1/5 8.0K ./index1/4 /超越对 8.0K ./index1/4/_state 2.9M ./index1/4/index 2.9M ./index1/4 8.0K ./index1/_state 8.0K ./index1/7/translog 8.0K ./index1/7/_state 2.9M ./index1/7/index 2.9M ./index1/7 8.0K ./index1/1/translog 8.0K ./index1/1/_state 2.9中号./index1/1/index 2.9M ./index1/1 8.0K ./index1/2/translog 8.0K ./index1/2/_state 2.9M ./index1/2/index 2.9M。 /索引1/2 8.0K ./index1/6/translog 8.0K ./index1/6/_state 3.0M ./index1/6/index 3.0M ./index1/6 8.0K ./index1/0/translog 8.0K ./index1/0/_state 1.5M ./index1/0/index 1.5M ./index1/0 8.0K ./index1/8/translog 8.0K。/索引1/8/_STATE 1.5M ./index1/8/index 1.5M ./index1/8 8.0K ./index1/11/translog 8.0K ./index1/11/_state 2.9M ./index1/11 /索引 2.9M ./index1/11 12K ./index1/9/translog 8.0K ./index1/9/_state 3.0M ./index1/9/index 3.0M ./index1/9 8.0K ./index1/3/translog 8.0K ./index1/3/_state 3.0M ./index1/3/index 3.0M ./index1/3 31M ./index1 16K ./index2/10/ translog 8.0K ./index2/10/_state 16M ./index2/10/index 16M ./index2/10 36K ./index2/5/translog 8.0K ./index2/5/_state 43M ./index2/5/index 43M ./index2/5 20K ./index2/4/超越对 8.0K ./index2/4/_state 17M ./index2/4/index 18M ./index2/4 8.0K ./index2/_state 40K ./index2/7/translog 8.0K ./index2/7/_STATE 32M ./index2/7/index 32M ./index2/7 68K ./index2/1/translog 8.0K ./index2/1/_state 21M ./index2/1/index 21M ./index2/1 64K ./index2/2/translog 8.0K ./index2/2/_state 19M ./index2/2/index 19M ./index2/2 116K ./index2/6/translog 8.0K ./index2/6/_state 22M ./index2/6 /索引 22M ./index2/6 24K ./index2/0/translog 8.0K ./index2/0/_state 17M ./index2/0/index 17M ./index2/0 128K ./索引2/8 /超越对 8.0K ./index2/8/_state 34M ./index2/8/index 34M ./index2/8 72K ./index2/11/translog 8.0K ./index2/11/_state 20M ./index2/11/index 20M ./index2/11 88K ./index2/9/translog 8.0K ./index2/9/_state 22M ./index2/9/index 22M ./index2/9 76K ./index2/3/translog 8.0K ./index2/3/_state 16M ./index2/3/index 16M ./index2/3 277M ./index2 307M。

在备份库

,大小: [根@ esnode1备份] $杜-lh

114M ./backup/indices/index1/0

112M ./backup/indices/index1/5

114M ./backup/indices/index1/11

114M ./backup/indices/index1/10

111M ./backup/indices/index1/8

116M ./backup/indices/index1/4

120M ./backup/indices/index1/9

118M。/备份/索引/索引1/3

114M ./backup/indices/index1/2

115M ./backup/indices/index1/7

115M ./backup/indices/index1/1

112M ./backup/indices/index1/6

1.4G ./backup/indices/index1

747M ./backup/indices/index2/0

1.6G ./backup/indices/index2/5

887M ./backup/indices/index2/11

743M ./backup/indices/index2/10

2.1G ./备份/索引/索引2/8

801M ./backup/indices/index2/4

1.3G ./backup/indices/index2/9

8.78 ./backup/ind冰/索引2/3

951M ./backup/indices/index2/2

1.2G ./backup/indices/index2/7

953M ./backup/indices/index2/1

943M ./backup/indices/index2/6

13G ./backup/indices/index2

15G ./backup/indices

15G ./backup

1.1M ./backuplogs

15G。

====== https://www.elastic.co/blog/introducing-snapshot-restore 备份和恢复操作是增量,这意味着只有自上次快照更改的文件将被复制到存储库或恢复成一个索引。 增量快照允许按照需要频繁执行快照操作,而无需太多的磁盘空间开销。用户现在可以轻松地在升级之前创建快照,或者在群集中进行有风险的更改,并在出现问题时快速回滚到以前的索引状态。快照/恢复机制还可用于同步“热”集群和不同地理位置的远程“冷”备份集群之间的数据,以实现快速灾难恢复。

从上面,我的情况是一个真正的问题,任何人都可以帮助我吗?提前致谢 !

+0

你如何触发你的小时备份? – Val

+0

在/ var/spool/cron/root中添加一个linux cron作业 0 1 * * * /opt/es/ESBackup.sh >>/backup/log/backup _ $(date + \%Y - \%m- \ %d).log – caiyufei

+0

您的shell脚本是使用快照/还原还是只是复制数据文件夹?在后一种情况下,请知道,如果不先复制索引,则不应复制数据文件夹,否则会冒着复制不一致数据的风险。 – Val

回答

1

在Elasticsearch官方论坛确认

1)这是一个正常的结果是索引和备份库在我的情况

2)一些冗余数据有很大的不同尺寸(500G VS 15G)备份快照是由Lucene的段合并引起的

来自Elasticsearch专家:如果您持续索引到集群,则段的合并将持续发生在后台,并且相同的记录将随着时间的推移而以多个段结束,导致一个比索引大小大得多的存储库。

https://discuss.elastic.co/t/backup-repository-size-is-much-bigger-than-indices-size/47469/7

0

快照和恢复在段级别的作品它是不是在弹性提到的“快照和恢复”文件非常重要的信息。 我向Elastic报告了这个问题,并询问他们是否可以更新文档。