Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16830 Add Public IOStatistics API
  3. HADOOP-17271

S3A statistics to support IOStatistics

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.1
    • fs/s3

    Description

      S3A to rework statistics with

      • API + Implementation split of the interfaces used by subcomponents when reporting stats
      • S3A Instrumentation to implement all the interfaces
      • streams, etc to all implement IOStatisticsSources and serve to callers
      • Add some tracking of durations of remote requests
      • RemoteIterator iterators returned from list* calls return list stats for callers to collect; toString will print them too

      Attachments

        Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-17451 Intermittent failure of S3A tests which make assertions on statistics/IOStatistics

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-17456 S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          causes

          Sub-task - The sub-task of the issue HADOOP-17774 bytesRead FS statistic showing twice the correct value in S3A

          • Major - Major loss of function.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-13845 s3a to instrument duration of HTTP calls

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15348 S3A Input Stream bytes read counter isn't getting through to StorageStatistics/instrumentation properly

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16133 S3A statistic collection underrecords bytes written in helper threads

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          fixes

          Sub-task - The sub-task of the issue HADOOP-16666 typo in TestNeworkBinding

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request #2324

          Web Link GitHub Pull Request #2553

          Web Link GitHub Pull Request #2580

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 11.5h
                  11.5h