Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-28565

Make map reduce jobs accept connection uri when specifying peer cluster

    XMLWordPrintableJSON

Details

    • Reviewed
    • Hide
      For TableOutputFormat
      The OUTPUT_CONF_PREFIX and QUORUM_ADDRESS are deprecated, a new OUTPUT_CLUSTER field is introduced. Please use OUTPUT_CLUSTER to specify the target cluster through connection URI. You can also carry the extra configuration values through the queries of the connection URI, which is the same with what OUTPUT_CONF_PREFIX provides.

      For CopyTable
      A new '--peer.uri' command line option is introduced, for specifying target cluster through connection URI, the old '--peer,adr' option is deprecated. If both options are presented, only '--peer.uri' will take effect.

      For SyncTable
      Introduce '--sourceuri' and '--targeturi' command line options for specifying source/target cluster through connection URI. The old '--sourcezkcluster' and '--targetzkcluster' are deprecated. If '--sourceuri' and '--sourcezkcluster' are both presented, only '--sourceuri' will take effect. It is the same for '--targeturi' and '--targetzkcluster'.

      For VerifyReplication
      Since we may load the peer cluster from ReplicationPeerConfig, so here we do not introduce new command line options, you can just use connection uri as the peer quorum address. The support for old cluster key format will be removed in future releases.

      For TableMapReduceUtil
      Add a new initCredentialsForCluster method which takes a URI as parameter
      public static void initCredentialsForCluster(Job job, Configuration conf, URI uri) throws IOException;

      Deprecate two methods which take quorumAddress as a parameter
      public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, String quorumAddress) throws IOException
      public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, String quorumAddress, boolean addDependencyJars) throws IOException;

      Introduce two new methods which take a URI as parameter
      public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, URI outputCluster) throws IOException;
      public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, URI outputCluster, boolean addDependencyJars) throws IOException;
      Show
      For TableOutputFormat The OUTPUT_CONF_PREFIX and QUORUM_ADDRESS are deprecated, a new OUTPUT_CLUSTER field is introduced. Please use OUTPUT_CLUSTER to specify the target cluster through connection URI. You can also carry the extra configuration values through the queries of the connection URI, which is the same with what OUTPUT_CONF_PREFIX provides. For CopyTable A new '--peer.uri' command line option is introduced, for specifying target cluster through connection URI, the old '--peer,adr' option is deprecated. If both options are presented, only '--peer.uri' will take effect. For SyncTable Introduce '--sourceuri' and '--targeturi' command line options for specifying source/target cluster through connection URI. The old '--sourcezkcluster' and '--targetzkcluster' are deprecated. If '--sourceuri' and '--sourcezkcluster' are both presented, only '--sourceuri' will take effect. It is the same for '--targeturi' and '--targetzkcluster'. For VerifyReplication Since we may load the peer cluster from ReplicationPeerConfig, so here we do not introduce new command line options, you can just use connection uri as the peer quorum address. The support for old cluster key format will be removed in future releases. For TableMapReduceUtil Add a new initCredentialsForCluster method which takes a URI as parameter public static void initCredentialsForCluster(Job job, Configuration conf, URI uri) throws IOException; Deprecate two methods which take quorumAddress as a parameter public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, String quorumAddress) throws IOException public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, String quorumAddress, boolean addDependencyJars) throws IOException; Introduce two new methods which take a URI as parameter public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, URI outputCluster) throws IOException; public static void initTableReducerJob(String table, Class<? extends TableReducer> reducer, Job job, Class partitioner, URI outputCluster, boolean addDependencyJars) throws IOException;

    Attachments

      Issue Links

        Activity

          People

            zhangduo Duo Zhang
            zhangduo Duo Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: