Uploaded image for project: 'REEF (Retired)'
  1. REEF (Retired)
  2. REEF-69

More fine-grained JVM options control

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.12
    • None
    • None

    Description

      REEF Evaluators and Drivers take over the whole memory of their host container as part of the JVM heap. This has bitten us in the past when Netty's buffers pushed us over the memory limit, which (on YARN) results in killed containers.

      As a remedy, we added the JVM heap "slack" option, to leave some memory unallocated to the heap. This "solved" the issue at hand, but is hardly a principled solution.

      Also, this is fundamentally an application concern, not one that REEF can generically solve. Consider Tasks that spawn processes as an example: In that case, the majority of the container memory should be used for that process, not the JVM hosting the Task.

      Hence, we should add fine-grained control over the JVM startup options to the REEF API.

      Attachments

        Issue Links

          Activity

            People

              chobrian Brian Cho
              markus.weimer Markus Weimer
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: