Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-7979

Fix out of the box defaults with spillable memory configs

    XMLWordPrintableJSON

Details

    Description

      Looks like we are very conservative wrt memory configs used for spillable map based FSV. 

       
      For eg, we are only allocating 15Mb out of the box to file groups when using spillable map based FSV.
      public long getMaxMemoryForFileGroupMap() {
      long totalMemory = getLong(SPILLABLE_MEMORY);
      return totalMemory - getMaxMemoryForPendingCompaction() - getMaxMemoryForBootstrapBaseFile();
      }
       
      SPILLABLE_MEMORY = default is 100Mb.
      getMaxMemoryForPendingCompaction = 80% of 100MB.
      getMaxMemoryForBootstrapBaseFile = 5% of 100Mb.
      so, overall, out of the box we are allocating only 15Mb for getMaxMemoryForFileGroupMap.
      ref: https://github.com/apache/hudi/blob/bb0621edee97507cf2460e8cb57b5307510b917e/hudi-[…]/apache/hudi/common/table/view/FileSystemViewStorageConfig.java
      Wondering do we even need 80% for pending compaction tracker in our FSV. I am thinking to make it 15%. so that we can give more memory to actual file groups. We may not have lot of pending compactions for a given table. 

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              shivnarayan sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: