Details
Major Description
This is because we remove the SnapshotSentinel from snapshotHandlers in SnapshotManager#cleanupSentinels. Only when the following 3 case, the cleanupSentinels will be called:
1. SnapshotManager#isSnapshotDone;
2. SnapshotManager#takeSnapshot;
3. SnapshotManager#restoreOrCloneSnapshot
So if no isSnapshotDone called, or no further snapshot taking, or snapshot restore/clone. the SnapshotSentinel will always be keep in snapshotHandlers.
But after HBASE-21387, Only when no snapshot taking, the SnapshotHFileCleaner will check the unref files and clean.
I found this bug, because in our XiaoMi branch-2, we implement the soft delete feature, which means if someone delete a table, then master will create a snapshot firstly, after that, the table deletion begain. the implementation is quite simple, we use the snapshotManager to create a snapshot.
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java index 8f42e4a..6da6a64 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java @@ -2385,12 +2385,6 @@ public class HMaster extends HRegionServer implements MasterServices { protected void run() throws IOException { getMaster().getMasterCoprocessorHost().preDeleteTable(tableName); + if (snapshotBeforeDelete) { + LOG.info("Take snaposhot for " + tableName + " before deleting"); + snapshotManager + .takeSnapshot(SnapshotDescriptionUtils.getSnapshotNameForDeletedTable(tableName)); + } + LOG.info(getClientIdAuditPrefix() + " delete " + tableName); // TODO: We can handle/merge duplicate request
In the master, I found the endless log after delete a table:
org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache: Not checking unreferenced files since snapshot is running, it will skip to clean the HFiles this time
This is because the snapshotHandlers never be cleaned after call the snapshotManager#takeSnapshot. I think the asynSnapshot may has the same problem.
Attachments
Attachments
Issue Links
- links to
