Major Description
during a scan of many rows(each row may be more than 64k), we encountered "RpcClient java.io.IOException: Unexpected closed connection" a lot, this is breaking kylin's normal functions.
if you open my log attached and grep "http-bio-7070-exec-11", you will find this thread is:
1. launching several scans, this is because the kylin cube has several segments, each segment correspond to a htable. A complete cube visit requires to visit every htable related, so there're many scans established
2. One of the scanner(scanner_id: 538064847515747456 ) failed due to:java.io.IOException: Unexpected closed connection
3. the error in step 2 lead to hbase client side/server side out of sync, casuing org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException
4. hbase client retried with a different scanner id (still same htable, same region) but still failed
can you please check at server side on what on earth is causing the " Unexpected closed connection" it seems to be the root cause.