ceph rados对象存储索引残留问题排查与处理

2023-11-11 06:45

本文主要是介绍ceph rados对象存储索引残留问题排查与处理,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

问题现象

对象存储存储桶无法删除,检查发现生命周期过期后存储桶中有文件残留,未完全删除,但实际访问文件时为404,通过s3cmd无法删除对象,且无报错。

问题定位

检查bucket当前状态,发现桶内有大量object残留:

# radosgw-admin bucket stats --bucket=xxx
{"bucket": "xxx","zonegroup": "97cde842-9840-46d2-98f5-2f2ff2b356b9","placement_rule": "default-placement","explicit_placement": {"data_pool": "","data_extra_pool": "","index_pool": ""},"id": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","marker": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","index_type": "Normal","owner": "e638db8fbfbd4eeba23f4e7f386e2234","ver": "0#115872937,1#124262992,2#124096868,3#113933319,4#113752484,5#115978810,6#119804274,7#122088764,8#117238420,9#114292015,10#122281621,11#112069481,12#117514195,13#113249706,14#114992906,15#119836492,16#109344437,17#115584159,18#113904518,19#118475333,20#118442717,21#119161849,22#119636967,23#115195682,24#114825666,25#116687890,26#110362647,27#114200698,28#122991031,29#111118759,30#122337033,31#114870449,32#115295855,33#117904918,34#121221126,35#116571732,36#124130503,37#118485487,38#110489698,39#113517117,40#119301165,41#115423515,42#118595461,43#115347045,44#120019831,45#113194194,46#114868275,47#123410209,48#119010861,49#114866634,50#111604486,51#117146989,52#113854552,53#111169375,54#120663269,55#116393793,56#114043425,57#117807012,58#122349615,59#113390481,60#124332341,61#114732340,62#114933373,63#114692587,64#118841860,65#118823413,66#115203887,67#122762813,68#109180753,69#115918036,70#116907631,71#115166297,72#126285010,73#117308131,74#118488176,75#117960296,76#115163648,77#116434667,78#114904720,79#128437168,80#119323730,81#114563504,82#117850906,83#119937587,84#121641776,85#111226459,86#115483044,87#119067683,88#115415015,89#112365103,90#119033043,91#111300726,92#117803502,93#116570158,94#110132992,95#117143542,96#118852347,97#121028481,98#111041704,99#109977398,100#116800374,101#121455617,102#119020462,103#116583479,104#117083862,105#119048007,106#118643510,107#115722285,108#114712589,109#119526889,110#122204433,111#118918806,112#119785343,113#123881328,114#114785575,115#116062597,116#120837772,117#118009592,118#112875590,119#119865516,120#111902759,121#112887532,122#112571750,123#119188027,124#115919787,125#118978645,126#114970199,127#116637842","master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0","mtime": "2022-09-19 23:05:51.793737","max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#","usage": {"rgw.main": {"size": 57217593456,"size_actual": 57233543168,"size_utilized": 57217593456,"size_kb": 55876557,"size_kb_actual": 55892132,"size_kb_utilized": 55876557,"num_objects": 7896,"": {"size": 515899392,"size_actual": 515899392,"size_utilized": 515899392,"size_kb": 503808,"size_kb_actual": 503808,"size_kb_utilized": 503808,"num_objects": 123},"STANDARD": {"size": 56701694064,"size_actual": 56717643776,"size_utilized": 56701694064,"size_kb": 55372749,"size_kb_actual": 55388324,"size_kb_utilized": 55372749,"num_objects": 7773}},"rgw.multimeta": {"size": 0,"size_actual": 0,"size_utilized": 216,"size_kb": 0,"size_kb_actual": 0,"size_kb_utilized": 1,"num_objects": 6,"STANDARD": {"size": 0,"size_actual": 0,"size_utilized": 216,"size_kb": 0,"size_kb_actual": 0,"size_kb_utilized": 1,"num_objects": 6}}},"bucket_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": -1,"max_objects": -1}
}

拉取残余文件列表,首先获取pool、bucket idnum_shards

# 获取pool,ningbo1-zone1.rgw.buckets.index
rados lspools | grep "rgw.buckets.index"
# 获取bucket id,a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242
radosgw-admin bucket stats --bucket {bucket name} | egrep "id"
# 获取shard数量,128
radosgw-admin metadata get bucket.instance:{bucket name}:{bucket id} | grep num_shards

然后拉取文件列表:

for i in {0..127}
dorados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i >> objects.txt
done

检查object状态,发现相应数据底层均已删除:

for object in `cat objects.txt`
doradosgw-admin object stat --bucket=xxx --object=$object
done

均报错ERROR: failed to stat object, returned error: (2) No such file or directory
列出bucket索引,发现所有索引"exists": "true",判断为索引残留问题,及索引存在但数据已删除。

问题处置

可通过rados rmomapkey删除无效索引:

for i in {0..127}
dofor object in `rados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i`dorados -c /etc/ceph/ceph.conf -p ningbo1-zone1.rgw.buckets.index rmomapkey .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i $objectdone
done

重新扫描修复桶状态(可多次执行):

# radosgw-admin bucket check --fix --check-objects --bucket=xxx
[]
{}
{"existing_header": {"usage": {}},"calculated_header": {"usage": {}}
}

再次检查,发现桶已正常清空:

# radosgw-admin bucket stats --bucket=xxx
{"bucket": "xxx","zonegroup": "97cde842-9840-46d2-98f5-2f2ff2b356b9","placement_rule": "default-placement","explicit_placement": {"data_pool": "","data_extra_pool": "","index_pool": ""},"id": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","marker": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","index_type": "Normal","owner": "e638db8fbfbd4eeba23f4e7f386e2234","ver": "0#115872943,1#124262998,2#124096874,3#113933325,4#113752490,5#115978816,6#119804280,7#122088770,8#117238426,9#114292021,10#122281627,11#112069487,12#117514201,13#113249712,14#114992912,15#119836498,16#109344443,17#115584165,18#113904524,19#118475339,20#118442723,21#119161855,22#119636973,23#115195688,24#114825672,25#116687896,26#110362653,27#114200704,28#122991037,29#111118765,30#122337039,31#114870455,32#115295861,33#117904924,34#121221132,35#116571738,36#124130509,37#118485493,38#110489704,39#113517123,40#119301171,41#115423521,42#118595467,43#115347051,44#120019837,45#113194200,46#114868281,47#123410215,48#119010867,49#114866640,50#111604492,51#117146995,52#113854558,53#111169381,54#120663275,55#116393799,56#114043431,57#117807018,58#122349621,59#113390487,60#124332347,61#114732346,62#114933379,63#114692593,64#118841866,65#118823419,66#115203893,67#122762819,68#109180759,69#115918042,70#116907637,71#115166303,72#126285016,73#117308137,74#118488182,75#117960302,76#115163654,77#116434673,78#114904726,79#128437174,80#119323736,81#114563510,82#117850912,83#119937593,84#121641782,85#111226465,86#115483050,87#119067689,88#115415021,89#112365109,90#119033049,91#111300732,92#117803508,93#116570164,94#110132998,95#117143548,96#118852353,97#121028487,98#111041710,99#109977404,100#116800380,101#121455623,102#119020468,103#116583485,104#117083868,105#119048013,106#118643516,107#115722291,108#114712595,109#119526895,110#122204439,111#118918812,112#119785349,113#123881334,114#114785581,115#116062603,116#120837778,117#118009598,118#112875596,119#119865522,120#111902765,121#112887538,122#112571756,123#119188033,124#115919793,125#118978651,126#114970205,127#116637848","master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0","mtime": "2022-09-19 23:05:51.793737","max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#","usage": {},"bucket_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": -1,"max_objects": -1}
}

处理完毕。

这篇关于ceph rados对象存储索引残留问题排查与处理的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/388256

相关文章

C语言中的浮点数存储详解

《C语言中的浮点数存储详解》:本文主要介绍C语言中的浮点数存储详解,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录1、首先明确一个概念2、接下来,讲解C语言中浮点型数存储的规则2.1、可以将上述公式分为两部分来看2.2、问:十进制小数0.5该如何存储?2.3 浮点

Flask解决指定端口无法生效问题

《Flask解决指定端口无法生效问题》文章讲述了在使用PyCharm开发Flask应用时,启动地址与手动指定的IP端口不一致的问题,通过修改PyCharm的运行配置,将Flask项目的运行模式从Fla... 目录android问题重现解决方案问题重现手动指定的IP端口是app.run(host='0.0.

Java实现将byte[]转换为File对象

《Java实现将byte[]转换为File对象》这篇文章将通过一个简单的例子为大家演示Java如何实现byte[]转换为File对象,并将其上传到外部服务器,感兴趣的小伙伴可以跟随小编一起学习一下... 目录前言1. 问题背景2. 环境准备3. 实现步骤3.1 从 URL 获取图片字节数据3.2 将字节数组

Seata之分布式事务问题及解决方案

《Seata之分布式事务问题及解决方案》:本文主要介绍Seata之分布式事务问题及解决方案,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录Seata–分布式事务解决方案简介同类产品对比环境搭建1.微服务2.SQL3.seata-server4.微服务配置事务模式1

mysql关联查询速度慢的问题及解决

《mysql关联查询速度慢的问题及解决》:本文主要介绍mysql关联查询速度慢的问题及解决方案,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录mysql关联查询速度慢1. 记录原因1.1 在一次线上的服务中1.2 最终发现2. 解决方案3. 具体操作总结mysql

一文教你解决Python不支持中文路径的问题

《一文教你解决Python不支持中文路径的问题》Python是一种广泛使用的高级编程语言,然而在处理包含中文字符的文件路径时,Python有时会表现出一些不友好的行为,下面小编就来为大家介绍一下具体的... 目录问题背景解决方案1. 设置正确的文件编码2. 使用pathlib模块3. 转换路径为Unicod

Spring MVC跨域问题及解决

《SpringMVC跨域问题及解决》:本文主要介绍SpringMVC跨域问题及解决方案,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录跨域问题不同的域同源策略解决方法1.CORS2.jsONP3.局部解决方案4.全局解决方法总结跨域问题不同的域协议、域名、端口

Javascript访问Promise对象返回值的操作方法

《Javascript访问Promise对象返回值的操作方法》这篇文章介绍了如何在JavaScript中使用Promise对象来处理异步操作,通过使用fetch()方法和Promise对象,我们可以从... 目录在Javascript中,什么是Promise1- then() 链式操作2- 在之后的代码中使

MySQL常见的存储引擎和区别说明

《MySQL常见的存储引擎和区别说明》MySQL支持多种存储引擎,如InnoDB、MyISAM、MEMORY、Archive、CSV和Blackhole,每种引擎有其特点和适用场景,选择存储引擎时需根... 目录mysql常见的存储引擎和区别说明1. InnoDB2. MyISAM3. MEMORY4. A

MyBatis的配置对象Configuration作用及说明

《MyBatis的配置对象Configuration作用及说明》MyBatis的Configuration对象是MyBatis的核心配置对象,它包含了MyBatis运行时所需的几乎所有配置信息,这个对... 目录MyBATis配置对象Configuration作用Configuration 对象的主要作用C