Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22567

[HBCK2] Add new methods for dealing with missing regions in META while Master is online

    XMLWordPrintableJSON

Details

    • Hide
      Adds new, lightweight option for recovering missing regions in meta, "addFsRegionsMissingInMeta" command, as well "readonly" "reportMissingRegionsInMeta" to show the list of regions to be readded in meta. This is a less intrusive alternative to OfflineMetaRepair, can be used when master and meta are still capable of coming online.

      Detailed description as provided by the command usage help:

      {noformat}
       addFsRegionsMissingInMeta <NAMESPACE|NAMESPACE:TABLENAME>...
         Options:
          -d,--force_disable aborts fix for table if disable fails.
         To be used in scenarios where some regions may be missing in META,
         but there's still a valid 'regioninfo' metadata file on HDFS.
         This is a lighter version of 'OfflineMetaRepair tool commonly used for
         similar issues on 1.x release line.
         This command needs META to be online. For each table name passed as
         parameter, it performs a diff between regions available in META,
         against existing regions dirs on HDFS. Then, for region dirs with
         no matches in META, it reads regioninfo metadata file and
         re-creates given region in META. Regions are re-created in 'CLOSED'
         state at META table only, but not in Masters' cache, and are not
         assigned either. To get these regions online, run HBCK2
      'assigns'command
         printed at the end of this command results for convenience.

         NOTE: If using hbase releases older than 2.3.0, a rolling restart of
         HMasters is needed prior to executing the provided 'assigns' command.

         An example adding missing regions for tables 'tbl_1' on default
         namespace, 'tbl_2' on namespace 'n1' and for all tables from
         namespace 'n2':
           $ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2
         Returns HBCK2 'assigns' command with all re-inserted regions.
         SEE ALSO: reportMissingRegionsInMeta
      ...
       reportMissingRegionsInMeta <NAMESPACE|NAMESPACE:TABLENAME>...
         To be used in scenarios where some regions may be missing in META,
         but there's still a valid 'regioninfo metadata file on HDFS.
         This is a checking only method, designed for reporting purposes and
         doesn't perform any fixes, providing a view of which regions (if any)
         would get re-added to meta, grouped by respective table/namespace.
         To effectively re-add regions in meta, addFsRegionsMissingInMeta should be executed.
         This command needs META to be online. For each namespace/table passed
         as parameter, it performs a diff between regions available in META,
         against existing regions dirs on HDFS. Region dirs with no matches
         are printed grouped under its related table name. Tables with no
         missing regions will show a 'no missing regions' message. If no
         namespace or table is specified, it will verify all existing regions.
         It accepts a combination of multiple namespace and tables. Table names
         should include the namespace portion, even for tables in the default
         namespace, otherwise it will assume as a namespace value.
         An example triggering missing regions report for tables 'table_1'
         and 'table_2', under default namespace:
           $ HBCK2 reportMissingRegionsInMeta default:table_1 default:table_2
         An example triggering missing regions report for table 'table_1'
         under default namespace, and for all tables from namespace 'ns1':
           $ HBCK2 reportMissingRegionsInMeta default:table_1 ns1
         Returns list of missing regions for each table passed as parameter, or
         for each table on namespaces specified as parameter.
      ...
      {noformat}
      Show
      Adds new, lightweight option for recovering missing regions in meta, "addFsRegionsMissingInMeta" command, as well "readonly" "reportMissingRegionsInMeta" to show the list of regions to be readded in meta. This is a less intrusive alternative to OfflineMetaRepair, can be used when master and meta are still capable of coming online. Detailed description as provided by the command usage help: {noformat}  addFsRegionsMissingInMeta <NAMESPACE|NAMESPACE:TABLENAME>...    Options:     -d,--force_disable aborts fix for table if disable fails.    To be used in scenarios where some regions may be missing in META,    but there's still a valid 'regioninfo' metadata file on HDFS.    This is a lighter version of 'OfflineMetaRepair tool commonly used for    similar issues on 1.x release line.    This command needs META to be online. For each table name passed as    parameter, it performs a diff between regions available in META,    against existing regions dirs on HDFS. Then, for region dirs with    no matches in META, it reads regioninfo metadata file and    re-creates given region in META. Regions are re-created in 'CLOSED'    state at META table only, but not in Masters' cache, and are not    assigned either. To get these regions online, run HBCK2 'assigns'command    printed at the end of this command results for convenience.    NOTE: If using hbase releases older than 2.3.0, a rolling restart of    HMasters is needed prior to executing the provided 'assigns' command.    An example adding missing regions for tables 'tbl_1' on default    namespace, 'tbl_2' on namespace 'n1' and for all tables from    namespace 'n2':      $ HBCK2 addFsRegionsMissingInMeta default:tbl_1 n1:tbl_2 n2    Returns HBCK2 'assigns' command with all re-inserted regions.    SEE ALSO: reportMissingRegionsInMeta ...  reportMissingRegionsInMeta <NAMESPACE|NAMESPACE:TABLENAME>...    To be used in scenarios where some regions may be missing in META,    but there's still a valid 'regioninfo metadata file on HDFS.    This is a checking only method, designed for reporting purposes and    doesn't perform any fixes, providing a view of which regions (if any)    would get re-added to meta, grouped by respective table/namespace.    To effectively re-add regions in meta, addFsRegionsMissingInMeta should be executed.    This command needs META to be online. For each namespace/table passed    as parameter, it performs a diff between regions available in META,    against existing regions dirs on HDFS. Region dirs with no matches    are printed grouped under its related table name. Tables with no    missing regions will show a 'no missing regions' message. If no    namespace or table is specified, it will verify all existing regions.    It accepts a combination of multiple namespace and tables. Table names    should include the namespace portion, even for tables in the default    namespace, otherwise it will assume as a namespace value.    An example triggering missing regions report for tables 'table_1'    and 'table_2', under default namespace:      $ HBCK2 reportMissingRegionsInMeta default:table_1 default:table_2    An example triggering missing regions report for table 'table_1'    under default namespace, and for all tables from namespace 'ns1':      $ HBCK2 reportMissingRegionsInMeta default:table_1 ns1    Returns list of missing regions for each table passed as parameter, or    for each table on namespaces specified as parameter. ... {noformat}

    Description

      Following latest discussion on HBASE-21745, this proposes an hbck2 command that allows for inserting back regions missing in META that still have regioninfo available in HDFS. Although this is still an interactive and simpler version than the old OfflineMetaRepair, it still relies on hdfs state as the source of truth, and performs META updates mostly independently from Master (apart from requiring Meta table been online).

      For a more detailed explanation on this command behaviour, pasting command usage text:

      To be used for scenarios where some regions may be missing in META,
      but there's still a valid 'regioninfo' metadata file on HDFS.
      This is a lighter version of 'OfflineMetaRepair' tool commonly used for
      similar issues on 1.x release line.
      This command needs META to be online. For each table name passed as
      parameter, it performs a diff between regions available in META,
      against existing regions dirs on HDFS. Then, for region dirs with
      no matches in META, it reads regioninfo metadata file and
      re-creates given region in META. Regions are re-created in 'CLOSED'
      state at META table only, but not in Masters' cache, and are not
      assigned either. A rolling Masters restart, followed by a
      hbck2 'assigns' command with all re-inserted regions is required.
      This hbck2 'assigns' command is printed for user convenience.
      WARNING: To avoid potential region overlapping problems due to ongoing
      splits, this command disables given tables while re-inserting regions.
      An example adding missing regions for tables 'table_1' and 'table_2':
      $ HBCK2 addMissingRegionsInMeta table_1 table_2
      Returns hbck2 'assigns' command with all re-inserted regions.

      Attachments

        Issue Links

          Activity

            People

              wchevreuil Wellington Chevreuil
              wchevreuil Wellington Chevreuil
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: