Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11061

Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.2, 0.96.3
    • None
    • None
    • Reviewed

    Description

      HBASE-10948 fixed the issue where hbase table files all have 'x' mode.

      This JIRA is to port the fix to 0.96 / 0.98 branches.
      The port doesn't rely on API specific to hadoop 2.0

      Attachments

        1. 11061.txt
          4 kB
          Michael Stack
        2. HBASE-10948-0.98.patch
          4 kB
          Jerry He
        3. HBASE-10948-0.96.patch
          4 kB
          Jerry He

        Activity

          +1 for 0.98

          apurtell Andrew Kyle Purtell added a comment - +1 for 0.98
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641601/HBASE-10948-0.98.patch
          against trunk revision .
          ATTACHMENT ID: 12641601

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9383//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641601/HBASE-10948-0.98.patch against trunk revision . ATTACHMENT ID: 12641601 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9383//console This message is automatically generated.
          stack Michael Stack added a comment -

          So, we always get the hadoop1 'implementation':

          + /**
          + * Get the default permission for file.
          + * This is the same method as FsPermission.getFileDefault() in Hadoop 2.
          + * We provide the method here to support compatibility with Hadoop 1.
          + */
          + public static FsPermission getFileDefault()

          { + return new FsPermission((short)00666); }

          Should this be using the hadoop-compat modules in hbase? i.e. add this default to the hadoop1 implementation but if we are running on hadoop2, we pick up the hadoop2 implementation?

          Is the method name from hadoop? Seems like it should be getFileDefaultPermission... but if from hadoop you can't change it.
          Thanks.

          stack Michael Stack added a comment - So, we always get the hadoop1 'implementation': + /** + * Get the default permission for file. + * This is the same method as FsPermission.getFileDefault() in Hadoop 2. + * We provide the method here to support compatibility with Hadoop 1. + */ + public static FsPermission getFileDefault() { + return new FsPermission((short)00666); } Should this be using the hadoop-compat modules in hbase? i.e. add this default to the hadoop1 implementation but if we are running on hadoop2, we pick up the hadoop2 implementation? Is the method name from hadoop? Seems like it should be getFileDefaultPermission... but if from hadoop you can't change it. Thanks.
          jinghe Jerry He added a comment -

          Hi, stack
          Here is the method in org.apache.hadoop.fs.permission in hadoop 2:
          /**

          • Get the default permission for file.
            */
            public static FsPermission getFileDefault() { return new FsPermission((short)00666); }

          This method is not in hadoop 1.
          I put the same name with the same implementation in our HBase FSUtils for 0.96/0.98 so that 0.96/0.98 can use this implementation, even on hadoop 1.
          In a sense e are actually always using hadoop 2 implementation.
          Of course there is a risk with this hard-code approach, if the implementation evolves in hadoop 2. But it is a very small risk that we can live with in 0.96/0.98

          To use the hadoop-compat modules, I think I will need to do these:
          1. Have a wrapper interface in hadoop-compact.
          2. In hadoop2-compact, the wrapper will directly call org.apache.hadoop.fs.permission#getFileDefault().
          3. In hadoop1-compact, the wrapper will call an internal method that has the same implementation as org.apache.hadoop.fs.permission#getFileDefault()
          Probably an over-kill?

          jinghe Jerry He added a comment - Hi, stack Here is the method in org.apache.hadoop.fs.permission in hadoop 2: /** Get the default permission for file. */ public static FsPermission getFileDefault() { return new FsPermission((short)00666); } This method is not in hadoop 1. I put the same name with the same implementation in our HBase FSUtils for 0.96/0.98 so that 0.96/0.98 can use this implementation, even on hadoop 1. In a sense e are actually always using hadoop 2 implementation. Of course there is a risk with this hard-code approach, if the implementation evolves in hadoop 2. But it is a very small risk that we can live with in 0.96/0.98 To use the hadoop-compat modules, I think I will need to do these: 1. Have a wrapper interface in hadoop-compact. 2. In hadoop2-compact, the wrapper will directly call org.apache.hadoop.fs.permission#getFileDefault(). 3. In hadoop1-compact, the wrapper will call an internal method that has the same implementation as org.apache.hadoop.fs.permission#getFileDefault() Probably an over-kill?
          stack Michael Stack added a comment -

          Probably an over-kill?

          Well, what you describe is what was necessary when we wanted to run on h1 and h2 (smile). For sure required a bunch of work.

          Let me commit with added note on the danger h2 could change its implementation under us (but agree small risk).

          stack Michael Stack added a comment - Probably an over-kill? Well, what you describe is what was necessary when we wanted to run on h1 and h2 (smile). For sure required a bunch of work. Let me commit with added note on the danger h2 could change its implementation under us (but agree small risk).
          stack Michael Stack added a comment -

          What I applied to 0.96 and to 0.98 (0.98 had a reject but it was just WS at EOF). Thanks for the patch jerryhe

          stack Michael Stack added a comment - What I applied to 0.96 and to 0.98 (0.98 had a reject but it was just WS at EOF). Thanks for the patch jerryhe
          jinghe Jerry He added a comment -

          thanks stack for the review and additional comment.

          jinghe Jerry He added a comment - thanks stack for the review and additional comment.
          hudson Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #299 (See https://builds.apache.org/job/HBase-0.98/299/)
          HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590771)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #299 (See https://builds.apache.org/job/HBase-0.98/299/ ) HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590771) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #284 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/284/)
          HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590771)

          • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #284 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/284/ ) HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590771) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment -

          FAILURE: Integrated in hbase-0.96 #396 (See https://builds.apache.org/job/hbase-0.96/396/)
          HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590770)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment - FAILURE: Integrated in hbase-0.96 #396 (See https://builds.apache.org/job/hbase-0.96/396/ ) HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590770) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment -

          FAILURE: Integrated in hbase-0.96-hadoop2 #275 (See https://builds.apache.org/job/hbase-0.96-hadoop2/275/)
          HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590770)

          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
          hudson Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #275 (See https://builds.apache.org/job/hbase-0.96-hadoop2/275/ ) HBASE-11061 Port HBASE-10948 Fix hbase table file 'x' mode to 0.96 / 0.98 (stack: rev 1590770) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java

          People

            jinghe Jerry He
            yuzhihong@gmail.com Ted Yu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: