Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14541

When evictableMmapped or evictable size is zero, do not throw NoSuchElementException

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Our XiaoMi HBase team are evaluating the performence improvement of HBASE-21879, and we have few CPU flame graph & heap flame graph by using async-profiler, and find that there're some performence issues in DFSClient .

      See the attached two flame graphs, we can conclude that the try catch block in ShortCircuitCache#trimEvictionMaps has some serious perf problem , we should remove the try catch from DFSClient.

        /**
         * Trim the eviction lists.
         */
        private void trimEvictionMaps() {
          long now = Time.monotonicNow();
          demoteOldEvictableMmaped(now);
      
          while (true) {
            long evictableSize = evictable.size();
            long evictableMmappedSize = evictableMmapped.size();
            if (evictableSize + evictableMmappedSize <= maxTotalSize) {
              return;
            }
            ShortCircuitReplica replica;
            try {
              if (evictableSize == 0) {
                replica = (ShortCircuitReplica)evictableMmapped.get(evictableMmapped
                    .firstKey());
              } else {
                replica = (ShortCircuitReplica)evictable.get(evictable.firstKey());
              }
            } catch (NoSuchElementException e) {
              break;
            }
            if (LOG.isTraceEnabled()) {
              LOG.trace(this + ": trimEvictionMaps is purging " + replica +
                  StringUtils.getStackTrace(Thread.currentThread()));
            }
            purge(replica);
          }
        }
      

      Our Xiaomi HDFS Team member leosun08 will prepare patch for this issue.

      Attachments

        1. HDFS-14541.002.patch
          3 kB
          Lisheng Sun
        2. HDFS-14541.001.patch
          3 kB
          Lisheng Sun
        3. after-heap-flame-graph.svg
          359 kB
          Zheng Hu
        4. after-cpu-flame-graph.svg
          378 kB
          Zheng Hu
        5. before-cpu-flame-graph.svg
          369 kB
          Zheng Hu
        6. before-heap-flame-graph.svg
          373 kB
          Zheng Hu
        7. after-QPS.png
          272 kB
          Zheng Hu
        8. before-QPS.png
          404 kB
          Zheng Hu
        9. HDFS-14541.000.patch
          3 kB
          Lisheng Sun
        10. async-prof-pid-94152-alloc-2.svg
          384 kB
          Zheng Hu
        11. async-prof-pid-94152-cpu-1.svg
          366 kB
          Zheng Hu

        Issue Links

          Activity

            People

              leosun08 Lisheng Sun
              openinx Zheng Hu
              Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: