Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-420

OffHeapMemory configuration only supports a single type of executor

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.2

    Description

      At the implementation of the `getMemoryConf` method in the `JobLauncher` class, the method is implemented with an assumption that there is only a single type of container. However, our resources specification file supports not only a single type but multiple types (e.g. transient/reserved or executors with heterogeneous memory specs). I think you could refer to the `requestContainer` method in the `RuntimeMaster` class. This should support the different executors with a uniform ratio or something similar.

      + I think the default value of off-heap memory of 0.2 is a bit too large, as for workloads that do not use the off heap optimization simply leaves 20% of its memory idle. This should be changed in the way that the off-heap memory optimization is used by default or the default number of 0.2 should be minimized.

      Attachments

        Issue Links

          Activity

            People

              hwarim Hwarim Hyun
              wonook Wonook
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m