Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4026

Avoid too many file append operations in HiveProducer of hive metrics reporter

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • v3.0.0-alpha2
    • None
    • None

    Description

      Currently  for each write in HiveProducer, there will be a hdfs append operation, which is heavy for HDFS.

      A improvement is to keep a FSDataOutputStream in  HiveProducer and write data to it continuously

       

       

      Attachments

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. KYLIN-4059 metrics write to hive error

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. KYLIN-4605 HiveProducer write metrics less when one host running some kylin servers

          • Major - Major loss of function.
          • Closed
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. KYLIN-4573 Add option to indicate whether to close file for every append for Hive Producer

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #671

          Activity

            People

              Unassigned Unassigned
              liushaohui Shaohui Liu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: