Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9 features
  3. HADOOP-19295

S3A: fs.s3a.connection.request.timeout too low for large uploads over slow links

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0, 3.4.1
    • None
    • fs/s3
    • None

    Description

      The value of fs.s3a.connection.request.timeout (default = 60s} is too low for large uploads over slow connections.

      I suspect something changed between the v1 and v2 SDK versions so that put was exempt from the normal timeouts, It is not and now surfaces in failures to upload 1+ GB files over slower network connections. Smailer (for example 128 MB) files work.

      The parallel queuing of writes in the S3ABlockOutputStream is helping create this problem as it queues multiple blocks at the same time, so per-block bandwidth becomes available/blocks ; four blocks cuts the capacity down by a quarter.

      The fix is straightforward: use a much bigger timeout. I'm going to propose 15 minutes. We need to strike a balance between upload time allocation and other requests timing out.

      I do worry about other consequences; we've found that timeout exception happy to hide the underlying causes of retry failures -so in fact this may be better for all but a server hanging after the HTTP request is initiated.

      too bad we can't alter the timeout for different requests

      Attachments

        Activity

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: