Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-6176

Fix flaky test testArchivalWithMultiWriters

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • tests-ci
    • None

    Description

      [ERROR] testArchivalWithMultiWriters{boolean}[1]  Time elapsed: 68.25 s  <<< ERROR!
      2023-04-30T03:49:36.0590893Z java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
      2023-04-30T03:49:36.0591382Z     at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
      2023-04-30T03:49:36.0591833Z     at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
      2023-04-30T03:49:36.0592347Z     at org.apache.hudi.io.TestHoodieTimelineArchiver.testArchivalWithMultiWriters(TestHoodieTimelineArchiver.java:683)
      2023-04-30T03:49:36.0592661Z     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2023-04-30T03:49:36.0592916Z     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      2023-04-30T03:49:36.0593587Z     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      2023-04-30T03:49:36.0593883Z     at java.lang.reflect.Method.invoke(Method.java:498)
      2023-04-30T03:49:36.0594151Z     at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
      2023-04-30T03:49:36.0594478Z     at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
      2023-04-30T03:49:36.0594853Z     at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
      2023-04-30T03:49:36.0595215Z     at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
      2023-04-30T03:49:36.0595564Z     at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
      2023-04-30T03:49:36.0595934Z     at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
      2023-04-30T03:49:36.0596316Z     at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
      2023-04-30T03:49:36.0596990Z     at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
      2023-04-30T03:49:36.0597381Z     at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
      2023-04-30T03:49:36.0597760Z     at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
      2023-04-30T03:49:36.0598140Z     at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
      2023-04-30T03:49:36.0598520Z     at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
      2023-04-30T03:49:36.0598866Z     at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
      2023-04-30T03:49:36.0599178Z     at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
      2023-04-30T03:49:36.0599556Z     at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
      2023-04-30T03:49:36.0599941Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0600300Z     at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
      2023-04-30T03:49:36.0600678Z     at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
      2023-04-30T03:49:36.0601041Z     at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
      2023-04-30T03:49:36.0601398Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
      2023-04-30T03:49:36.0601770Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0602145Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
      2023-04-30T03:49:36.0602470Z     at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
      2023-04-30T03:49:36.0602923Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
      2023-04-30T03:49:36.0603295Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0603637Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
      2023-04-30T03:49:36.0603985Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
      2023-04-30T03:49:36.0604387Z     at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
      2023-04-30T03:49:36.0604819Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:212)
      2023-04-30T03:49:36.0605313Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:192)
      2023-04-30T03:49:36.0605699Z     at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
      2023-04-30T03:49:36.0606109Z     at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
      2023-04-30T03:49:36.0606435Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      2023-04-30T03:49:36.0606732Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0607028Z     at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
      2023-04-30T03:49:36.0607307Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0607600Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      2023-04-30T03:49:36.0607998Z     at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:440)
      2023-04-30T03:49:36.0608528Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0608828Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0609122Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0609395Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      2023-04-30T03:49:36.0609688Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0609989Z     at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
      2023-04-30T03:49:36.0610273Z     at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      2023-04-30T03:49:36.0610570Z     at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      2023-04-30T03:49:36.0610881Z     at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
      2023-04-30T03:49:36.0611182Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
      2023-04-30T03:49:36.0611486Z     at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      2023-04-30T03:49:36.0611775Z     at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
      2023-04-30T03:49:36.0612053Z     at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
      2023-04-30T03:49:36.0612348Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0612641Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0612918Z     at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      2023-04-30T03:49:36.0613219Z     at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
      2023-04-30T03:49:36.0613601Z     at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      2023-04-30T03:49:36.0614672Z     at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      2023-04-30T03:49:36.0615065Z     at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
      2023-04-30T03:49:36.0615418Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
      2023-04-30T03:49:36.0615709Z     at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      2023-04-30T03:49:36.0615994Z     at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
      2023-04-30T03:49:36.0616287Z     at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
      2023-04-30T03:49:36.0616572Z     at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
      2023-04-30T03:49:36.0616870Z     at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      2023-04-30T03:49:36.0617162Z     at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      2023-04-30T03:49:36.0617655Z     at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
      2023-04-30T03:49:36.0617976Z     at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
      2023-04-30T03:49:36.0618279Z     at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      2023-04-30T03:49:36.0618550Z     at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
      2023-04-30T03:49:36.0618881Z     at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
      2023-04-30T03:49:36.0619254Z     at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
      2023-04-30T03:49:36.0619614Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
      2023-04-30T03:49:36.0619989Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0620363Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
      2023-04-30T03:49:36.0620810Z     at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
      2023-04-30T03:49:36.0621149Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
      2023-04-30T03:49:36.0621519Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0621878Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
      2023-04-30T03:49:36.0622560Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
      2023-04-30T03:49:36.0622871Z     at java.util.ArrayList.forEach(ArrayList.java:1259)
      2023-04-30T03:49:36.0623232Z     at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
      2023-04-30T03:49:36.0623647Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
      2023-04-30T03:49:36.0624068Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0624433Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
      2023-04-30T03:49:36.0624769Z     at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
      2023-04-30T03:49:36.0625090Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
      2023-04-30T03:49:36.0625461Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0625818Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
      2023-04-30T03:49:36.0626149Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
      2023-04-30T03:49:36.0626446Z     at java.util.ArrayList.forEach(ArrayList.java:1259)
      2023-04-30T03:49:36.0626805Z     at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
      2023-04-30T03:49:36.0627212Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
      2023-04-30T03:49:36.0627581Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0627949Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
      2023-04-30T03:49:36.0628287Z     at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
      2023-04-30T03:49:36.0628607Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
      2023-04-30T03:49:36.0628971Z     at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
      2023-04-30T03:49:36.0629477Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
      2023-04-30T03:49:36.0629806Z     at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
      2023-04-30T03:49:36.0630205Z     at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
      2023-04-30T03:49:36.0630630Z     at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
      2023-04-30T03:49:36.0630995Z     at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
      2023-04-30T03:49:36.0631369Z     at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
      2023-04-30T03:49:36.0631931Z     at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
      2023-04-30T03:49:36.0632707Z     at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
      2023-04-30T03:49:36.0633093Z     at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
      2023-04-30T03:49:36.0633464Z     at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
      2023-04-30T03:49:36.0633960Z     at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
      2023-04-30T03:49:36.0634610Z     at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
      2023-04-30T03:49:36.0634960Z     at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
      2023-04-30T03:49:36.0635331Z     at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
      2023-04-30T03:49:36.0635825Z     at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
      2023-04-30T03:49:36.0636159Z     at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
      2023-04-30T03:49:36.0636466Z     at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
      2023-04-30T03:49:36.0636739Z     at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
      2023-04-30T03:49:36.0637195Z Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
      2023-04-30T03:49:36.0637494Z     at org.apache.hudi.client.HoodieTimelineArchiver.archive(HoodieTimelineArchiver.java:701)
      2023-04-30T03:49:36.0637817Z     at org.apache.hudi.client.HoodieTimelineArchiver.archiveIfRequired(HoodieTimelineArchiver.java:182)
      2023-04-30T03:49:36.0638168Z     at org.apache.hudi.io.TestHoodieTimelineArchiver.lambda$null$8(TestHoodieTimelineArchiver.java:655)
      2023-04-30T03:49:36.0638499Z     at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
      2023-04-30T03:49:36.0638805Z     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2023-04-30T03:49:36.0639114Z     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2023-04-30T03:49:36.0640515Z     at java.lang.Thread.run(Thread.java:750)
      2023-04-30T03:49:36.0640852Z Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/tmp/junit7571097132575561902/dataset/.hoodie/archived/.commits_.archive.5_1-0-1
      2023-04-30T03:49:36.0641208Z     at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
      2023-04-30T03:49:36.0641511Z     at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:334)
      2023-04-30T03:49:36.0641826Z     at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:399)
      2023-04-30T03:49:36.0642155Z     at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:462)
      2023-04-30T03:49:36.0642656Z     at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:441)
      2023-04-30T03:49:36.0642932Z     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1072)
      2023-04-30T03:49:36.0643244Z     at org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$create$9(HoodieWrapperFileSystem.java:290)
      2023-04-30T03:49:36.0643650Z     at org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:114)
      2023-04-30T03:49:36.0643992Z     at org.apache.hudi.common.fs.HoodieWrapperFileSystem.create(HoodieWrapperFileSystem.java:288)
      2023-04-30T03:49:36.0644346Z     at org.apache.hudi.common.table.log.HoodieLogFormatWriter.createNewFile(HoodieLogFormatWriter.java:234)
      2023-04-30T03:49:36.0644708Z     at org.apache.hudi.common.table.log.HoodieLogFormatWriter.getOutputStream(HoodieLogFormatWriter.java:121)
      2023-04-30T03:49:36.0645051Z     at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:144)
      2023-04-30T03:49:36.0645410Z     at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlock(HoodieLogFormatWriter.java:135)
      2023-04-30T03:49:36.0645877Z     at org.apache.hudi.client.HoodieTimelineArchiver.writeToFile(HoodieTimelineArchiver.java:719)
      2023-04-30T03:49:36.0646206Z     at org.apache.hudi.client.HoodieTimelineArchiver.archive(HoodieTimelineArchiver.java:699)
      2023-04-30T03:49:36.0646425Z     ... 6 more 

      Attachments

        Issue Links

          Activity

            People

              guoyihua Ethan Guo (this is the old account; please use "yihua")
              guoyihua Ethan Guo (this is the old account; please use "yihua")
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: