Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-9843

Ozone client high memory (heap) utilization

    XMLWordPrintableJSON

Details

    Description

      Ozone client high memory utilization during write - should use pooled direct buffer for chunks

      Found this issue when working on S3.

      S3G memory footprint is too high when reading/writing small objects.

      Ran a test to write thousand of small (400 bytes) objects per seconds and memory utilization and gc is too high.

      This is issue is not S3 specific, but happens on ozone-client in general.

      Attachments

        1. s3_heap_metric.png
          1.43 MB
          Duong
        2. image-2023-12-05-16-35-37-893.png
          1.37 MB
          Duong
        3. image-2023-12-05-16-36-17-726.png
          1.37 MB
          Duong
        4. buffer_allocation.png
          3.21 MB
          Duong

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. HDDS-10288 Checksum to support direct buffers

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDDS-10283 Apply zero copy to Ozone client reads chunks

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDDS-10043 java.lang.OutOfMemoryError on FSDataOutputStream write ops

          • Major - Major loss of function.
          • Resolved
          split from

          Improvement - An improvement or enhancement to an existing feature or task. HDDS-9536 Datanode perf: Copying (heap) buffers is costly

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #6153

          Activity

            People

              duongnguyen Duong
              duongnguyen Duong
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: