Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11208

SampleContainerLogAggregationPolicy is inconsistent with expectations

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.2.2
    • None
    • None

    Description

      repro steps:

      Configure the following parameters in yarn-site.xml

        <property>
          <name>yarn.nodemanager.log-aggregation.policy.class</name>
       <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.SampleContainerLogAggregationPolicy</value>
        </property>
        <property>
          <name>yarn.nodemanager.log-aggregation.policy.parameters</name>
          <value>SR:0.3,MIN:20</value>
        </property>

       

      run pi:

      hadoop  org.apache.hadoop.examples.QuasiMonteCarlo 100 5

       

      three nodemanagers in my cluster

       

      Expected quantity:20+(100-20)*0.3+1=45 container logs uploaded

      Actual quantity: 21 container logs uploaded

       

      So why are there fewer logs uploaded?

       

      I print debug log in SampleContainerLogAggregationPolicy.

      shouldDoLogAggregation()

       

      logContext.getContainerId().hashCode():-1455756646

      logContext.getContainerId().hashCode() % (1/sampleRate):-0.041277885

       

      So in most cases, it will not be uploaded

       

      Attachments

        1. YARN-11208.patch
          2 kB
          chuanjie.duan

        Issue Links

          Activity

            People

              Unassigned Unassigned
              chuanjie.duan chuanjie.duan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: