Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
TestCOWDataSource.testCopyOnWriteStorage will failed random. Because before the incremental read, add a new upsert commit.
// pull the latest commit
val hoodieIncViewDF2 = spark.read.format("org.apache.hudi")
.option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
.option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, commitInstantTime2)
.load(basePath)
the new commit is :
// Upsert based on the written table with Hudi metadata columns
val verificationRowKey = snapshotDF1.limit(1).select("_row_key").first.getString(0)
as verificationRowKey will contains in "uniqueKeyCnt", so will failed as : "expected: <65> but was: <66>"
https://travis-ci.com/github/apache/hudi/jobs/463879606
org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:173)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlices(RemoteHoodieTableFileSystemView.java:275)
... 30 more
[WARN ] 2020-12-22 12:32:40,788 org.apache.hudi.testutils.HoodieClientTestHarness - Closing file-system instance used in previous test-run
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.352 s <<< FAILURE! - in org.apache.hudi.functional.TestCOWDataSource
[ERROR] org.apache.hudi.functional.TestCOWDataSource.testCopyOnWriteStorage Time elapsed: 15.275 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <65> but was: <66>
at org.apache.hudi.functional.TestCOWDataSource.testCopyOnWriteStorage(TestCOWDataSource.scala:160)
[INFO] Running org.apache.hudi.functional.TestDataSourceForBootstrap
[WARN ] 2020-12-22 12:32:43,641 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:47,818 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:50,921 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:56,169 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:56,793 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:57,388 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:05,191 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:10,221 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:17,985 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:22,498 org.apache.hudi.DefaultSource - Loading Base File Only View.
Attachments
Issue Links
- links to