Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8581

ContentSummary on / skips further counts on yielding lock

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      If one directory such as "/result" exists about 200000 files, then when execute "hdfs dfs -count /", the result will go wrong. For all directories whose name after "/result", file num will not be included.

      My cluster see as below, "/result_1433858936" is the directory exist huge files, and files in "/sparkJobHistory", "/tmp", "/user" are not included

      vm-221:/export1/BigData/current # hdfs dfs -ls /
      15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled.
      Found 9 items
      rw-rr- 3 hdfs supergroup 0 2015-06-08 12:10 /PRE_CREATE_DIR.SUCCESS
      drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume
      drwx------ - hbase hadoop 0 2015-06-10 15:25 /hbase
      drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt
      drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history
      drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 /result_1433858936
      drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory
      drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp
      drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /
      15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled.
      1043 171536 1756375688 /
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS
      15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled.
      0 1 0 /PRE_CREATE_DIR.SUCCESS
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /flume
      15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled.
      1 0 0 /flume
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /hbase
      15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled.
      36 18 14807 /hbase
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /hyt
      15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled.
      1 0 0 /hyt
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /mr-history
      15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled.
      3 0 0 /mr-history
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936
      15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled.
      1001 171517 1756360881 /result_1433858936
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory
      15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled.
      1 3 21785 /sparkJobHistory
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /tmp
      15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled.
      17 6 35958 /tmp
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /user
      15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled.
      12 1 19077 /user

      Attachments

        1. HDFS-8581.4.patch
          3 kB
          J.Andreina
        2. HDFS-8581.3.patch
          3 kB
          J.Andreina
        3. HDFS-8581.2.patch
          3 kB
          J.Andreina
        4. HDFS-8581.1.patch
          3 kB
          J.Andreina

        Issue Links

          Activity

            People

              andreina J.Andreina
              tongshiquan tongshiquan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: