[TAJO-231] In StorageManger v2, TestCSVCompression incurs unexpected end of stream occasionally - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Incomplete
Affects Version/s: None
Fix Version/s: 0.2-incubating
Component/s: Storage
Labels:
None

Description

In StorageManger v2, TestCSVCompression incurs 'unexpected end of stream occasionally'.

https://builds.apache.org/job/Tajo-trunk-postcommit/496/testReport/junit/org.apache.tajo.storage.v2/TestCSVCompression/testSplitCompressionData_0_/

2013-10-05 13:21:32,289 INFO  [DiskFileScanner:0,sda] v2.DiskFileScanScheduler (DiskFileScanScheduler.java:run(104)) - Exit from Disk Queue:SplitCompression.bz2:0:65301_1380946892289
2013-10-05 13:21:32,289 INFO  [main] v2.ScheduledInputStream (ScheduledInputStream.java:readNext(117)) - EOF:SplitCompression.bz2:0:65301
2013-10-05 13:21:32,296 ERROR [main] v2.CSVFileScanner (CSVFileScanner.java:nextTuple(299)) - unexpected end of stream
java.io.IOException: unexpected end of stream
	at org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:969)
	at org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:504)
	at org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:333)
	at org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:399)
	at org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:483)
	at java.io.InputStream.read(InputStream.java:101)
	at org.apache.tajo.storage.v2.CSVFileScanner.page(CSVFileScanner.java:214)
	at org.apache.tajo.storage.v2.CSVFileScanner.nextTuple(CSVFileScanner.java:282)
	at org.apache.tajo.storage.v2.FileScannerV2.next(FileScannerV2.java:200)
	at org.apache.tajo.storage.v2.TestCSVCompression.testSplitCompressionData(TestCSVCompression.java:147)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
2013-10-05 13:21:32,298 INFO  [main] v2.ScheduledInputStream (ScheduledInputStream.java:close(348)) - Close:SplitCompression.bz2:0:65301
2013-10-05 13:21:32,298 INFO  [main] v2.FileScannerV2 (FileScannerV2.java:close(184)) - file:/home/hyunsik/Code/tajo/tajo-4/tajo-core/tajo-core-backend/target/test-data/v2/TestCSVCompression/SplitCompression.bz2:0 closed, totalScanTime=2
2013-10-05 13:21:32,299 INFO  [FileScanRunner:SplitCompression.bz2:65301:60218_1380946892299] v2.ScheduledInputStream (ScheduledInputStream.java:<init>(81)) - Open:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,301 INFO  [FileScanRunner:SplitCompression.bz2:65301:60218_1380946892301] v2.ScheduledInputStream (ScheduledInputStream.java:readNext(117)) - EOF:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,302 INFO  [FileScanRunner:SplitCompression.bz2:65301:60218_1380946892301] v2.ScheduledInputStream (ScheduledInputStream.java:readNext(121)) - EndOfStream:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,303 INFO  [main] v2.ScheduledInputStream (ScheduledInputStream.java:readNext(117)) - EOF:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,305 INFO  [main] v2.ScheduledInputStream (ScheduledInputStream.java:readNext(117)) - EOF:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,307 INFO  [main] v2.ScheduledInputStream (ScheduledInputStream.java:close(348)) - Close:SplitCompression.bz2:65301:60218
2013-10-05 13:21:32,307 INFO  [main] v2.FileScannerV2 (FileScannerV2.java:close(184)) - file:/home/hyunsik/Code/tajo/tajo-4/tajo-core/tajo-core-backend/target/test-data/v2/TestCSVCompression/SplitCompression.bz2:65301 closed, totalScanTime=2

java.lang.AssertionError: 
Expected :100000
Actual   :0
 <Click to see difference>
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.tajo.storage.v2.TestCSVCompression.testSplitCompressionData(TestCSVCompression.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:77)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

TAJO-231.patch
14/Oct/13 11:19
0.7 kB
Hyunsik Choi

Activity

People

Assignee:: Hyunsik Choi

Reporter:: Hyunsik Choi

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 05/Oct/13 04:24

Updated:: 14/Oct/13 15:34

Resolved:: 14/Oct/13 15:24