Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13204 Über-jira: S3a phase III: scale and tuning
  3. HADOOP-14138

Remove S3A ref from META-INF service discovery, rely on existing core-default entry

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.9.0
    • 2.8.0, 2.7.4, 3.0.0-alpha4
    • fs/s3
    • None
    • Incompatible change
    • Hide
      The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.
      Show
      The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.

    Description

      As discussed in HADOOP-14132, the shaded AWS library is killing performance starting all hadoop operations, due to classloading on FS service discovery.

      This is despite the fact that there is an entry for fs.s3a.impl in core-default.xml, we don't need service discovery here

      Proposed:

      1. cut the entry from /hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
      2. when HADOOP-14132 is in, move to that, including declaring an XML file exclusively for s3a entries

      I want this one in first as its a major performance regression, and one we coula actually backport to 2.7.x, just to improve load time slightly there too

      Attachments

        1. HADOOP-14138.001.patch
          0.6 kB
          Jason Darrell Lowe
        2. HADOOP-14138-branch-2-001.patch
          0.6 kB
          Steve Loughran

        Issue Links

          is broken by

          Sub-task - The sub-task of the issue HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-14132 Filesystem discovery to stop loading implementation classes

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. AMBARI-21665 shaded AWS SDK in Hadoop JARs slowing ambari view loading

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: