Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4850

fix OfflineImageViewer to work on fsimages with empty files or snapshots

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0-beta
    • tools
    • Reviewed

    Description

      I deployed hadoop-trunk HDFS and created /user/schu/. I then forced a checkpoint, fetched the fsimage, and ran the default OfflineImageViewer successfully on the fsimage.

      schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000004 -o oiv_out_1
      schu-mbp:~ schu$ cat oiv_out_1
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user
      drwxr-xr-x  -     schu supergroup          0 2013-05-24 16:59 /user/schu
      schu-mbp:~ schu$ 
      

      I then touched an empty file /user/schu/testFile1

      schu-mbp:~ schu$ hadoop fs -lsr /
      lsr: DEPRECATED: Please use 'ls -R' instead.
      drwxr-xr-x   - schu supergroup          0 2013-05-24 16:59 /user
      drwxr-xr-x   - schu supergroup          0 2013-05-24 17:00 /user/schu
      -rw-r--r--   1 schu supergroup          0 2013-05-24 17:00 /user/schu/testFile1
      

      and forced another checkpoint, fetched the fsimage, and reran the OfflineImageViewer. I encountered a NegativeArraySizeException:

      schu-mbp:~ schu$ hdfs oiv -i fsimage_0000000000000000008 -o oiv_out_2
      Input ended unexpectedly.
      2013-05-24 17:01:13,622 ERROR [main] offlineImageViewer.OfflineImageViewer (OfflineImageViewer.java:go(140)) - image loading failed at offset 402
      Exception in thread "main" java.lang.NegativeArraySizeException
      	at org.apache.hadoop.io.Text.readString(Text.java:458)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processPermission(ImageLoaderCurrent.java:370)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINode(ImageLoaderCurrent.java:671)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processChildren(ImageLoaderCurrent.java:557)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:464)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDirectoryWithSnapshot(ImageLoaderCurrent.java:470)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processLocalNameINodesWithSnapshot(ImageLoaderCurrent.java:444)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processINodes(ImageLoaderCurrent.java:398)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:199)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:136)
      	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.main(OfflineImageViewer.java:260)
      

      This is reproducible. I've reproduced this scenario after formatting HDFS and restarting and touching an empty file /testFile1.

      Attached are the data dirs, the fsimage before creating the empty file (fsimage_0000000000000000004) and the fsimage afterwards (fsimage_0000000000000000004) and their outputs, oiv_out_1 and oiv_out_2 respectively.

      The oiv_out_2 does not include the empty /user/schu/testFile1.

      I don't run into this problem using hadoop-2.0.4-alpha.

      Attachments

        1. datadirs.tar.gz
          3 kB
          Stephen Chu
        2. fsimage_0000000000000000004
          0.4 kB
          Stephen Chu
        3. fsimage_0000000000000000008
          0.4 kB
          Stephen Chu
        4. oiv_out_1
          0.2 kB
          Stephen Chu
        5. oiv_out_2
          0.2 kB
          Stephen Chu
        6. HDFS-4850.000.patch
          0.8 kB
          Jing Zhao
        7. HDFS-4850.001.patch
          18 kB
          Jing Zhao

        Issue Links

          Activity

            People

              jingzhao Jing Zhao
              schu Stephen Chu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: