Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22200

WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir when checking for recovered.edits files, instead of taking FS instance as additional method parameter. When specifying different file systems for wal dir and root dir, WALSplitter.hasRecoveredEdits current implementation will crash or give wrong results. As of now, it's being used indirectly by SplitTableRegionProcedure. When running tests with WAL dir on HDFS and root dir on S3, for example, noticed region split failing with below error:

      2019-04-08 13:53:58,064 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=98, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=test-tbl, parent=4c5db01611e97e3abbe02e781e867212, daughterA=28a0a5e4ef7618899f6bd6dfb5335fe7, daughterB=05fa26feaf03ebf9e87e099cbd1eabac
java.lang.IllegalArgumentException: Path hdfs://host-1.example.com:8020/wal_dir/default/test-tbl/4c5db01611e97e3abbe02e781e867212/recovered.edits scheme must be s3a
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:1127)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:437)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2110)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1668)
        at org.apache.hadoop.hbase.wal.WALSplitter.getSplitEditFilesSorted(WALSplitter.java:576)
        at org.apache.hadoop.hbase.wal.WALSplitter.hasRecoveredEdits(WALSplitter.java:558)
        at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.hasRecoveredEdits(SplitTableRegionProcedure.java:148)
        at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:255)

      Since WALSplitter.hasRecoveredEdits already resolves the proper WAL dir for the region, we can simply re-use FS instance from the path instance for the WAL dir region, when searching for recovered.edits.

      Attachments

        1. HBASE-22200-branch-2.1-003.patch
          7 kB
          Wellington Chevreuil
        2. HBASE-22200-master-002.patch
          7 kB
          Wellington Chevreuil
        3. HBASE-22200-branch-2.1-002.patch
          7 kB
          Wellington Chevreuil
        4. HBASE-22200-branch-2.1-001.patch
          6 kB
          Wellington Chevreuil
        5. HBASE-22200-master-001.patch
          6 kB
          Wellington Chevreuil

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-20734 Colocate recovered edits directory with hbase.wal.dir

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21688 Address WAL filesystem issues

          • Major - Major loss of function.
          • Closed

          Activity

            People

              wchevreuil Wellington Chevreuil
              wchevreuil Wellington Chevreuil
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: