Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-842

Better Export/Import-CSV Tool

    XMLWordPrintableJSON

Details

    • 2021-09

    Description

      Hi, our import-csv tool is currently implemented by JDBC and requires a fossil format:

      e.g., 

      Time,root.sg.d1.s1,root.sg.d1.s2,root.sg.d2.s1,root.sg.d2.s2,root.sg.d2.s3
2020-08-18T10:22:31.603+08:00,1,2.0,null,null,null
2020-08-18T10:22:35.631+08:00,1,2.0,null,null,null
2020-08-18T10:22:41.093+08:00,null,null,1,2.0,null
2020-08-18T10:22:52.603+08:00,null,null,1,2.0,true

      Requirement 1:

      As we support 3 kinds of output format: align all series (by default), align by device, without alignment, it is better to support such 3 kinds of import-csv format:

      a. 

      Time,root.sg.d1.s1,root.sg.d1.s2,root.sg.d2.s1,root.sg.d2.s2,root.sg.d2.s3
2020-08-18T10:22:31.603+08:00,1,2.0,null,null,null
2020-08-18T10:22:35.631+08:00,1,2.0,null,null,null
2020-08-18T10:22:41.093+08:00,null,null,1,2.0,null
2020-08-18T10:22:52.603+08:00,null,null,1,2.0,true

      b. 

      Time,Device,s1,s2,s3
2020-08-18T10:22:31.603+08:00,root.sg.d1,1,2.0,null
2020-08-18T10:22:35.631+08:00,root.sg.d1,1,2.0,null
2020-08-18T10:22:41.093+08:00,root.sg.d2,1,2.0,null
2020-08-18T10:22:52.603+08:00,root.sg.d2,1,2.0,true

      c.
      (it is strange, I'd like to do not support such format.)

      Requment2:
      Different users may have different time formats for the first column.
      So, we'd better support different kinds of time format. e.g., let users define how to parse their timestamp: yyyy-MM-ddHH:mm:ss.SSS etc..

      Requirement 3:
      Support NULL as well as empty char to describe the null data point. For example, the following 3 lines are the same:

      2020-08-18T10:22:31.603+08:00,root.sg.d1,1,null,null

      2020-08-18T10:22:31.603+08:00,root.sg.d1,1,,

      2020-08-18T10:22:31.603+08:00,root.sg.d1,1, ,

      Requirement 4:

      Support claiming the storage group name once rather than repeat the storage group name for each line:

      e.g., for format b, we can tell the tool the sg is `root.sg` and then each row looks like:

      2020-08-18T10:22:35.631+08:00,d1,1,2.0,null

      Another option is add a new column called storage_group for each row.

      For UT:
      1. all data type should be covered;
      2. incorrect csv format should be covered;

      Attachments

        Issue Links

          is a parent of

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1061 nodetool and importcsv issue in cluster env

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1232 import csv，the imported data is incompletely and some imported time column is wrong.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1234 ImportCSV tool cannot parse empty string values

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1316 A sensor point value is invalid and this row can be imported successfully，but the following rows in this file are not executed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-392 The generated dump0.csv can't be imported.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1313 import-csv can not indicate timestamp format，this causes millisecond is zero

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1315 export-csv.sh -tf yyyy-MM-dd\'T\'HH:mm:ss.SSSZ error： is not supported

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. IOTDB-1510 An error is reported when import data which time format is "yyyy/MM/dd HH:mm:ss"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-1305 Support Time type like 00:00:00 when importing csv files.

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-1446 csv import redesign

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-277 Lost docs about usage of import-csv.sh and export-csv.sh

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-80 Import/Export script Improvement

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-169 Compatibility of timestamp formats in 1)exportCsv, 2)client display and 3)sql

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-937 import generated CSV directly and don't need to create metadata

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. IOTDB-1010 Add CSV tool user guide on website

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. IOTDB-1013 Add docs of export-csv and import-csv

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #3820

          Web Link GitHub Pull Request #4045

          Activity

            People

              xuanronaldo Xuan Ronaldo
              hxd Xiangdong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: