Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-5639

[compaction]Fix file not found exception in cross space compaction selector

    XMLWordPrintableJSON

Details

    • 2023-3-Storage

    Description

      master 0302_33cd749
      对齐序列合并过程中,有NPE 异常,可以处理一下:
      2023-03-07 09:57:23,123 [pool-62-IoTDB-Compaction_Schedule-root.test.g_0-9-1] ERROR o.a.i.t.f.f.LocalFSFactory:98 - Failed to get buffered input stream for /data1/iotdb/./opt_0302_33cd749/sbin/../data/datanode/data/sequence/root.test.g_0/9/253/1678092688862-303-0-0.tsfile.resource.
      java.io.FileNotFoundException: /data1/iotdb/./opt_0302_33cd749/sbin/../data/datanode/data/sequence/root.test.g_0/9/253/1678092688862-303-0-0.tsfile.resource (No such file or directory)
      at java.io.FileInputStream.open0(Native Method)
      at java.io.FileInputStream.open(FileInputStream.java:195)
      at java.io.FileInputStream.<init>(FileInputStream.java:138)
      at java.io.FileInputStream.<init>(FileInputStream.java:93)
      at org.apache.iotdb.tsfile.fileSystem.fsFactory.LocalFSFactory.getBufferedInputStream(LocalFSFactory.java:96)
      at org.apache.iotdb.db.engine.storagegroup.TsFileResource.buildDeviceTimeIndex(TsFileResource.java:423)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate$TsFileResourceCandidate.prepareDeviceInfos(CrossSpaceCompactionCandidate.java:209)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate$TsFileResourceCandidate.containsDevice(CrossSpaceCompactionCandidate.java:241)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate.prepareNextSplit(CrossSpaceCompactionCandidate.java:86)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate.hasNextSplit(CrossSpaceCompactionCandidate.java:68)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.executeTaskResourceSelection(RewriteCrossSpaceCompactionSelector.java:152)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.selectOneTaskResources(RewriteCrossSpaceCompactionSelector.java:117)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.selectCrossSpaceTask(RewriteCrossSpaceCompactionSelector.java:240)
      at org.apache.iotdb.db.engine.compaction.schedule.CompactionScheduler.tryToSubmitCrossSpaceCompactionTask(CompactionScheduler.java:165)
      at org.apache.iotdb.db.engine.compaction.schedule.CompactionScheduler.scheduleCompaction(CompactionScheduler.java:61)
      at org.apache.iotdb.db.engine.storagegroup.DataRegion.executeCompaction(DataRegion.java:2150)
      at org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil.lambda$scheduleWithFixedDelay$1(ScheduledExecutorUtil.java:177)
      at org.apache.iotdb.commons.concurrent.WrappedRunnable$1.runMayThrow(WrappedRunnable.java:44)
      at org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      2023-03-07 09:57:23,153 [pool-62-IoTDB-Compaction_Schedule-root.test.g_0-9-1] ERROR o.a.i.d.e.c.s.i.RewriteCrossSpaceCompactionSelector:269 - root.test.g_0 cannot select file for cross space compaction
      org.apache.iotdb.db.exception.MergeException: Can't read file /data1/iotdb/./opt_0302_33cd749/sbin/../data/datanode/data/sequence/root.test.g_0/9/253/1678092688862-303-0-0.tsfile.resource from disk
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.selectOneTaskResources(RewriteCrossSpaceCompactionSelector.java:121)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.selectCrossSpaceTask(RewriteCrossSpaceCompactionSelector.java:240)
      at org.apache.iotdb.db.engine.compaction.schedule.CompactionScheduler.tryToSubmitCrossSpaceCompactionTask(CompactionScheduler.java:165)
      at org.apache.iotdb.db.engine.compaction.schedule.CompactionScheduler.scheduleCompaction(CompactionScheduler.java:61)
      at org.apache.iotdb.db.engine.storagegroup.DataRegion.executeCompaction(DataRegion.java:2150)
      at org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil.lambda$scheduleWithFixedDelay$1(ScheduledExecutorUtil.java:177)
      at org.apache.iotdb.commons.concurrent.WrappedRunnable$1.runMayThrow(WrappedRunnable.java:44)
      at org.apache.iotdb.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:29)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Can't read file /data1/iotdb/./opt_0302_33cd749/sbin/../data/datanode/data/sequence/root.test.g_0/9/253/1678092688862-303-0-0.tsfile.resource from disk
      at org.apache.iotdb.db.engine.storagegroup.TsFileResource.buildDeviceTimeIndex(TsFileResource.java:432)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate$TsFileResourceCandidate.prepareDeviceInfos(CrossSpaceCompactionCandidate.java:209)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate$TsFileResourceCandidate.containsDevice(CrossSpaceCompactionCandidate.java:241)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate.prepareNextSplit(CrossSpaceCompactionCandidate.java:86)
      at org.apache.iotdb.db.engine.compaction.selector.utils.CrossSpaceCompactionCandidate.hasNextSplit(CrossSpaceCompactionCandidate.java:68)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.executeTaskResourceSelection(RewriteCrossSpaceCompactionSelector.java:152)
      at org.apache.iotdb.db.engine.compaction.selector.impl.RewriteCrossSpaceCompactionSelector.selectOneTaskResources(RewriteCrossSpaceCompactionSelector.java:117)
      ... 14 common frames omitted
      Caused by: java.lang.NullPointerException: null
      at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readByte(ReadWriteIOUtils.java:477)
      at org.apache.iotdb.db.engine.storagegroup.TsFileResource.buildDeviceTimeIndex(TsFileResource.java:424)
      ... 20 common frames omitted

      测试环境
      192.168.130.1 16CPU 32GB内存
      ~关合并生成数据备份
      备份路径在:/data1/iotdb/m_0305_1df6a76/data_xx
      CN :MAX_HEAP_SIZE="2G"
      DN:MAX_HEAP_SIZE="20G"
      MAX_DIRECT_MEMORY_SIZE="6G"
      COMMON :
      time_partition_interval=6048000000
      max_waiting_time_when_insert_blocked=3600000
      enable_seq_space_compaction=false
      enable_unseq_space_compaction=false
      enable_cross_space_compaction=false
      benchmark配置见附件,运行脚本:run_2_comp_aligned.sh

      ~开合并 ,测试合并
      CN :
      MAX_HEAP_SIZE="2G"
      DN :
      MAX_HEAP_SIZE="20G"
      MAX_DIRECT_MEMORY_SIZE="6G"
      COMMON:
      time_partition_interval=6048000000
      query_timeout_threshold=36000000
      max_waiting_time_when_insert_blocked=3600000
      enable_seq_space_compaction=true
      enable_unseq_space_compaction=true
      enable_cross_space_compaction=true
      compaction_write_throughput_mb_per_sec=1024

      Attachments

        1. comp1_aligned.conf
          14 kB
          刘珍
        2. comp2_aligned.conf
          14 kB
          刘珍
        3. run_2_comp_aligned.sh
          0.4 kB
          刘珍
        4. test.sh
          0.2 kB
          刘珍

        Activity

          People

            ChouBenson 周沛辰
            刘珍 刘珍
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: