Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6102

Lower the default maximum items per directory to fix PB fsimage loading

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • namenode
    • None
    • Incompatible change

    Description

      Found by schu during testing. We were creating a bunch of directories in a single directory to blow up the fsimage size, and it ends up we hit this error when trying to load a very large fsimage:

      2014-03-13 13:57:03,901 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 24523605 INodes.
2014-03-13 13:57:59,038 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Failed to load image from FSImageFile(file=/dfs/nn/current/fsimage_0000000000024532742, cpktTxId=0000000000024532742)
com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
        at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
        at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
        at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
        at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
        at com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9839)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9770)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9901)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9896)
        at 52)
...

      Some further research reveals there's a 64MB max size per PB message, which seems to be what we're hitting here.

      Attachments

        1. hdfs-6102-2.patch
          7 kB
          Andrew Wang
        2. hdfs-6102-1.patch
          4 kB
          Andrew Wang

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6104 TestFsLimits#testDefaultMaxComponentLength Fails on branch-2

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10729 Improve log message for edit loading failures caused by FS limit checks.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7473 Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-10593 MAX_DIR_ITEMS should not be hard coded since RPC buff size is configurable

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6099 HDFS file system limits not enforced on renames.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-6055 Change default configuration to limit file name length in HDFS

          • Major - Major loss of function.
          • Closed

          Activity

            People

              andrew.wang Andrew Wang
              andrew.wang Andrew Wang
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: