Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-1905

Fix bug where ORC self tune would under estimate native ORC writer memory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • gobblin-core
    • None

    Description

      During close, the native ORC writer would flush itself - emptying its buffers and leading to the memory estimate to be extremely minimal, then record its memory estimate in the saved state if it has not recorded an estimate before. So for datasets that write a small number of records to file it would greatly underestimate the memory within the native ORC writer.

      Attachments

        Activity

          People

            abti Abhishek Tiwari
            wlo William Lo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 0.5h
                0.5h