Concurrent Data Loading Issue

Even carbon claim that it can support the concurrent data loading together with table compaction, in fact it cannot. We have faced data inconsistent issue in carbon 1.6.1 when loading data concurrently into the table together with compaction. That is why we implement table locking for load data, compact and clean files command. All this is due to the manipulation of the table’s metadata file, i.e .tablestatus.

We are facing issue in concurrent data loading together with compaction - gets us to data inconsistency - what is the way out to fix this as we want concurrent loading with compaction