Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17437

Support specifying a WAL directory outside of the root directory

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      This patch adds support for specifying a WAL directory outside of the HBase root directory.

      Multiple configuration variables were added to accomplish this:
      hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset.

      hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default.

      hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.
      Show
      This patch adds support for specifying a WAL directory outside of the HBase root directory. Multiple configuration variables were added to accomplish this: hbase.wal.dir: used to configure where the root WAL directory is located. Could be on a different FileSystem than the root directory. WAL directory can not be set to a subdirectory of the root directory. The default value of this is the root directory if unset. hbase.rootdir.perms: Configures FileSystem permissions to set on the root directory. This is '700' by default. hbase.wal.dir.perms: Configures FileSystem permissions to set on the WAL directory FileSystem. This is '700' by default.
    • Patch

    Description

      Currently, the WAL and the StoreFiles need to be on the same FileSystem. Some FileSystems (such as Amazon S3) don’t support append or consistent writes. These two properties are imperative for the WAL in order to avoid loss of writes. However, StoreFiles don’t necessarily need the same consistency guarantees (since writes are cached locally and if writes fail, they can always be replayed from the WAL).

      This JIRA aims to allow users to configure a log directory (for WALs) that is outside of the root directory or even in a different FileSystem. The default value will still put the log directory under the root directory.

      Attachments

        1. hbase-17437-master.patch
          74 kB
          Yishan Yang
        2. hbase-17437-branch-1.2.patch
          134 kB
          Yishan Yang
        3. HBASE-17437.master.012.patch
          95 kB
          Zach York
        4. HBASE-17437.master.011.patch
          96 kB
          Zach York
        5. HBASE-17437.master.010.patch
          100 kB
          Zach York
        6. HBASE-17437.master.009.patch
          100 kB
          Zach York
        7. HBASE-17437.master.008.patch
          100 kB
          Zach York
        8. HBASE-17437.master.007.patch
          102 kB
          Zach York
        9. HBASE-17437.master.006.patch
          102 kB
          Zach York
        10. HBASE-17437.master.005.patch
          103 kB
          Zach York
        11. HBASE-17437.master.004.patch
          94 kB
          Zach York
        12. HBASE-17437.master.003.patch
          94 kB
          Zach York
        13. HBASE-17437.master.002.patch
          94 kB
          Zach York
        14. HBASE-17437.master.001.patch
          94 kB
          Zach York
        15. HBASE-17437.branch-1.004.patch
          128 kB
          Zach York
        16. HBASE-17437.branch-1.003.patch
          128 kB
          Zach York
        17. HBASE-17437.branch-1.002.patch
          128 kB
          Zach York
        18. HBASE-17437.branch-1.001.patch
          104 kB
          Zach York

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21755 RS aborts while performing replication with wal dir on hdfs, root dir on s3

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-21688 Address WAL filesystem issues

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-17878 java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter when starting HBase with hbase.rootdir on S3

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-20734 Colocate recovered edits directory with hbase.wal.dir

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-17588 Remove unused imports brought in by HBASE-17437

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed
          links to

          Web Link Review Board (branch-1)

          Web Link Review Board (master)

          Activity

            People

              zyork Zach York
              yangyishan0901m Yishan Yang
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: