Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1591

Planner for high-level applications should validate the partition number of intermediate topics

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.14.0
    • 0.15.0
    • None

    Description

      We have seen cases where the intermediate topic created by partitionBy() operator was created with partition number calculated by the planner. However, when input topic partition expands with traffic, usually that would have trickling effect to the downstream topic partition and task assignments (i.e. 128 partitions from the input topic got shuffled into only 8 partitions in the intermediate topic usually does not distribute the work load widely enough). At least, the planner needs to validation and print a warning message if the intermediate topic partition is less than the max of input/output topic partition number.

      Attachments

        Activity

          People

            Unassigned Unassigned
            nickpan47 Yi Pan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: