Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25135 HBase downloads in e2e tests fail often
  3. FLINK-19158

Revisit java e2e download timeouts

    XMLWordPrintableJSON

Details

    Description

      Consider this failed test case

      Test testHBase(org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) is running.
--------------------------------------------------------------------------------
09:38:38,719 [                main] INFO  org.apache.flink.tests.util.cache.PersistingDownloadCache    [] - Downloading https://archive.apache.org/dist/hbase/1.4.3/hbase-1.4.3-bin.tar.gz.
09:40:38,732 [                main] ERROR org.apache.flink.tests.util.hbase.SQLClientHBaseITCase       [] - 
--------------------------------------------------------------------------------
Test testHBase(org.apache.flink.tests.util.hbase.SQLClientHBaseITCase) failed with:
java.io.IOException: Process ([wget, -q, -P, /home/vsts/work/1/e2e_cache/downloads/1598516010, https://archive.apache.org/dist/hbase/1.4.3/hbase-1.4.3-bin.tar.gz]) exceeded timeout (120000) or number of retries (3).
	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:148)
	at org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
	at org.apache.flink.tests.util.cache.PersistingDownloadCache.getOrDownload(PersistingDownloadCache.java:36)
	at org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:76)
	at org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:70)
	at org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)

      It seems that the download has not been retried. The download might be stuck? I would propose to set a timeout per try and increase the total time from 2 to 5 minutes.

      This example is from: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6267&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue FLINK-17510 StreamingKafkaITCase. testKafka timeouts on downloading Kafka

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. FLINK-17400 LocalStandaloneKafkaResource.setupKafkaDist fails due to download timeout

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Technical Debt - FLINK-25112 Remove TTL from e2e cache

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #13389

          Web Link GitHub Pull Request #14694

          Activity

            People

              martijnvisser Martijn Visser
              rmetzger Robert Metzger
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: