Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19064

Synchronous replication for HBase

    XMLWordPrintableJSON

Details

    Description

      The guys from Alibaba made a presentation on HBaseCon Asia about the synchronous replication for HBase. We(Xiaomi) think this is a very useful feature for HBase so we want to bring it into the community version.

      This is a big feature so we plan to do it in a feature branch.

      Attachments

        1. HBASE-19064.master.patch
          552 kB
          Duo Zhang
        2. HBASE-19064.master.v1.patch
          561 kB
          Duo Zhang
        3. HBASE-19064.master.v2.patch
          562 kB
          Duo Zhang

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. HBASE-9469 Synchronous replication

          • Major - Major loss of function.
          • Closed
          is blocked by

          New Feature - A new feature of the product, which has yet to be developed. HBASE-19397 Design procedures for ReplicationManager to notify peer change event from master

          • Major - Major loss of function.
          • Closed
          is related to

          Umbrella - An overarching type made of sub-tasks HBASE-20422 Synchronous replication for HBase phase II

          • Major - Major loss of function.
          • Open
          links to

          Web Link Design Doc

          Web Link Review Board

          1.
          		 Add a remote peer cluster wal directory config for synchronous replication Sub-task Resolved Guanghao Zhang
          2.
          		 Support setting up two clusters with A and S state Sub-task Resolved Duo Zhang
          3.
          		 Implement a procedure to convert RS from A to DA Sub-task Resolved Unassigned
          4.
          		 Implement a procedure to convert RS from S to DA Sub-task Resolved Guanghao Zhang
          5.
          		 Reject read/write from client but accept write from replication in state S Sub-task Resolved Duo Zhang
          6.
          		 Introduce a new log writer which can write to two HDFSes Sub-task Resolved Duo Zhang
          7.
          		 Introduce a special WALProvider for synchronous replication Sub-task Resolved Duo Zhang
          8.
          		 Add a new cluster state flag for synchronous replication Sub-task Resolved Guanghao Zhang
          9.
          		 Reject the replication request when peer is DA or A state Sub-task Resolved Zheng Hu
          10.
          		 Allow temporarily skipping the creation of remote writer for SynchronousReplicationWALProvider Sub-task Resolved Unassigned
          11.
          		 Complete the procedure for adding a sync replication peer Sub-task Resolved Duo Zhang
          12.
          		 Use protobuf instead of enum.ordinal to store SyncReplicationState Sub-task Resolved Guanghao Zhang
          13.
          		 Add UT for sync replication peer in DA state Sub-task Resolved Duo Zhang
          14.
          		 Only allow table replication for sync replication for now Sub-task Resolved Guanghao Zhang
          15.
          		 Introduce a new base class for replication peer procedure Sub-task Resolved Duo Zhang
          16.
          		 Allow write request from replication but reject write request from user client when S state. Sub-task Resolved Zheng Hu
          17.
          		 Only allow removing sync replication peer which is in DA state Sub-task Resolved Zheng Hu
          18.
          		 General framework to transit sync replication state Sub-task Resolved Duo Zhang
          19.
          		 General framework to transit sync replication state Sub-task Resolved Unassigned
          20.
          		 Implement a procedure to replay sync replication wal for standby cluster Sub-task Resolved Guanghao Zhang
          21.
          		 Create remote wal directory when transitting to state S Sub-task Resolved Duo Zhang
          22.
          		 Remove the SYNC_REPLICATION_ENABLED flag Sub-task Resolved Guanghao Zhang
          23.
          		 Forbid major compaction when standby cluster replay the remote wals Sub-task Resolved Guanghao Zhang
          24.
          		 Also remove the wal file in remote cluster when we finish replicating a file Sub-task Resolved Duo Zhang
          25.
          		 Operational documents for synchronous replication. Sub-task Resolved Zheng Hu
          26.
          		 Do not write the cluster id of the current active cluster when writing remote WAL Sub-task Resolved Zheng Hu
          27.
          		 Give up replicating anything in S state Sub-task Resolved Duo Zhang
          28.
          		 Cleanup related resources when remove a sync replication peer Sub-task Resolved Zheng Hu
          29.
          		 Also remove remote wals when peer is in DA state Sub-task Resolved Duo Zhang
          30.
          		 Support removing a ReplicationSourceShipper for a special wal group Sub-task Resolved Duo Zhang
          31.
          		 Support removing a WAL from LogRoller Sub-task Resolved Guanghao Zhang
          32.
          		 Concurrency issue for CombinedAsyncWriter Sub-task Resolved Unassigned
          33.
          		 NPE in RecoverStandbyProcedure.execute Sub-task Resolved Guanghao Zhang
          34.
          		 Check remote WAL directory when creating peer and transiting peer to A Sub-task Resolved Duo Zhang
          35.
          		 Allow writing WAL to local and remote cluster concurrently Sub-task Resolved Duo Zhang
          36.
          		 Polish the WAL switching when transiting from A to S Sub-task Resolved Duo Zhang
          37.
          		 Reopen regions using ReopenTableRegionsProcedure Sub-task Resolved Duo Zhang
          38.
          		 Performance comparison of synchronous replication branch and master branch Sub-task Resolved Zheng Hu
          39.
          		 NPE encountered when rolling update from master with an async peer to branch HBASE-19064 Sub-task Resolved Zheng Hu
          40.
          		 Fix the style issues on branch HBASE-19064 before merging back to master Sub-task Resolved Duo Zhang
          41.
          		 Fix broken TestReplicationShell Sub-task Resolved Duo Zhang

          Activity

            People

              zhangduo Duo Zhang
              zhangduo Duo Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: