[CASSANDRA-17116] When streaming sees a ClosedChannelException this triggers the disk failure policy - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Normal
Resolution: Fixed
Fix Version/s: 4.1-alpha1, 4.1
Component/s: Consistency/Streaming
Labels:
None

Bug Category:
Availability - Process Crash
Severity:
Normal
Complexity:
Normal
Discovered By:
Unit Test
Platform:

All
Impacts:

None
Since Version:

4.0
Source Control Link:

https://github.com/apache/cassandra/commit/0859b3c4cfa690dd270030c4bc7ff93eadf6a732
Test and Documentation Plan:

Hide

tests added

Show
tests added

Description

Found in ~~CASSANDRA-17085~~.

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7264
https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7256

ERROR [Stream-Deserializer-/127.0.0.1:7000-f2eb1a15] 2021-11-02 21:35:40,983 DefaultFSErrorHandler.java:104 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.FSWriteError: java.nio.channels.ClosedChannelException
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:227)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:206)
	at org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:125)
	at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:51)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:37)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
	at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:62)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException: null
	at org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:136)
	at org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:155)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:217)
	... 9 common frames omitted

When bootstrap fails and streaming is closed, this triggers the disk failure policy which causes the JVM to halt by default (if this happens outside of bootstrap, then we stop transports and keep the JVM up).

org.apache.cassandra.streaming.StreamDeserializingTask attempts to handle this by ignoring this exception, but the call to org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize
Does try/catch and inspects exception; triggering this condition.

Attachments

Issue Links

is related to

CASSANDRA-17076 Fix failing test: test_bootstrap_with_reset_bootstrap_state - bootstrap_test.TestBootstrap

Resolved

CASSANDRA-17081 Fix test: bootstrap_test.py::TestBootstrap::test_bootstrap_with_reset_bootstrap_state

Resolved

CASSANDRA-17085 commit log was switched from non-daemon to daemon threads, which causes the JVM to exit in some case as no non-daemon threads are active

Resolved

CASSANDRA-18733 Waiting indefinitely on ReceivedMessage response in StreamSession#receive() can cause deadlock

Resolved

links to

Circle CI: trunk

GH PR: dtest trunk

GH PR: trunk

(2 links to)

Activity

People

Assignee:: David Capwell

Reporter:: David Capwell

Authors:: David Capwell, Francisco Guerrero

Reviewers:: Caleb Rackliffe, Dinesh Joshi

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 02/Nov/21 23:38

Updated:: 08/Aug/23 16:42

Resolved:: 04/Feb/22 19:53

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

20m