Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-7478

Fix max delta commits guard check w/ MDT

    XMLWordPrintableJSON

Details

    Description

      protected static void checkNumDeltaCommits(HoodieTableMetaClient metaClient, int maxNumDeltaCommitsWhenPending) {
      final HoodieActiveTimeline activeTimeline = metaClient.reloadActiveTimeline();
      Option<HoodieInstant> lastCompaction = activeTimeline.filterCompletedInstants()
      .filter(s -> s.getAction().equals(COMPACTION_ACTION)).lastInstant();
      int numDeltaCommits = lastCompaction.isPresent()
      ? activeTimeline.getDeltaCommitTimeline().findInstantsAfter(lastCompaction.get().getTimestamp()).countInstants()
      : activeTimeline.getDeltaCommitTimeline().countInstants();
      if (numDeltaCommits > maxNumDeltaCommitsWhenPending)

      { throw new HoodieMetadataException(String.format("Metadata table's deltacommits exceeded %d: " + "this is likely caused by a pending instant in the data table. Resolve the pending instant " + "or adjust `%s`, then restart the pipeline.", maxNumDeltaCommitsWhenPending, HoodieMetadataConfig.METADATA_MAX_NUM_DELTACOMMITS_WHEN_PENDING.key())); }

      }

      Here we account for action type "compaction. But compaction completed instant will have "commit" as action. So, we need to fix it. 

       

       

       

      Attachments

        Issue Links

          Activity

            People

              wombatukun Vova Kolmakov
              shivnarayan sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: