Major Description
This patch looks to have exposed a problem in our HStore commit logic. We are supposed to crash out if we fail to write but we keep going here. I am having trouble figuring why. Let me write a little test:
2014-03-14 01:58:48,647 DEBUG [Thread-3] regionserver.HRegionFileSystem(339): Committing store file /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0 as /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0 2014-03-14 01:58:48,647 INFO [Thread-2] regionserver.HRegion(5779): writing data to region testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab. with WAL disabled. Data may be lost in the event of a crash. 2014-03-14 01:58:48,648 ERROR [Thread-3] regionserver.HStore$StoreFlusherImpl(1964): Failed to commit store file /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0 org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file file:/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0 at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:580) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1019) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:211) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:350) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:445) at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:551) at org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:842) at org.apache.hadoop.hbase.regionserver.HStore.access$200(HStore.java:118) at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:1961) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1706) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1583) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1498) at org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3034) Caused by: java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:282) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.seek(RawLocalFileSystem.java:111) at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:78) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:206) at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237) at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158) at org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:384) at org.apache.hadoop.fs.FSInputChecker.seek(FSInputChecker.java:365) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.seek(ChecksumFileSystem.java:271) at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37) at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:389) at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:537) ... 13 more 2014-03-14 01:58:48,657 DEBUG [pool-1-thread-1] regionserver.HRegion(1037): Closing testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab.: disabling compactions & flushes 2014-03-14 01:58:48,657 INFO [pool-1-thread-1] regionserver.HRegion(1045): Running close preflush of testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab.