Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-808 Throttle/reject writes to avoid running out of memory
  3. KUDU-716

Single-disk tpch_real_world with SF=500 should complete successfully

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Private Beta
    • None
    • test

    Description

      We now have a "tpch_real_world" benchmark that loads lineitem for an arbitrary scale factor and runs queries against it. In trying to run this on a single-disk system, we are running into a number of issues that prevent it from completing successfully:

      • OOMs on the server side, due to:
        • poor default MRS size throttling behavior (per-MRS limits, no default setting)
        • live scanners retaining large amounts of memory from MRS (KUDU-18)
      • OOMs on the client side (bugs like KUDU-715)
      • timeouts during writes (eg due to client not respecting backpressure)
      • timeouts during scans (unclear why - perhaps the server went into swap?)

      We should make it a release criteria that tpch_real_world SF500 can complete even on underprovisioned hardware (single-HDD).

      Attachments

        1. data.tsv
          449 kB
          Todd Lipcon
        2. inserts-over-time.png
          18 kB
          Todd Lipcon
        3. inserts-vs-bytes-on-disk.png
          60 kB
          Todd Lipcon
        4. memory usage.png
          35 kB
          Todd Lipcon
        5. mrs-size.png
          68 kB
          Todd Lipcon
        6. mrs-vs-bytes-on-disk.png
          67 kB
          Todd Lipcon

        Issue Links

          relates to

          Sub-task - The sub-task of the issue KUDU-542 Make MRS throttling global

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. KUDU-615 MRS/DMS hold on to anchors longer than necessary

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. KUDU-720 Log GC never runs if MRS is under constant write pressure

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. KUDU-715 tpch_real_world crash with bad_alloc in client

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. KUDU-18 Handle "slow readers" not holding too much memory

          • Major - Major loss of function.
          • Open

          Activity

            People

              jdcryans Jean-Daniel Cryans
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: