Details
-
Improvement
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
3.2.2
-
None
-
None
Description
repro steps:
Configure the following parameters in yarn-site.xml
<property>
<name>yarn.nodemanager.log-aggregation.policy.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.SampleContainerLogAggregationPolicy</value>
</property>
<property>
<name>yarn.nodemanager.log-aggregation.policy.parameters</name>
<value>SR:0.3,MIN:20</value>
</property>
run pi:
hadoop org.apache.hadoop.examples.QuasiMonteCarlo 100 5
three nodemanagers in my cluster
Expected quantity:20+(100-20)*0.3+1=45 container logs uploaded
Actual quantity: 21 container logs uploaded
So why are there fewer logs uploaded?
I print debug log in SampleContainerLogAggregationPolicy.
shouldDoLogAggregation()
logContext.getContainerId().hashCode():-1455756646
logContext.getContainerId().hashCode() % (1/sampleRate):-0.041277885
So in most cases, it will not be uploaded
Attachments
Attachments
Issue Links
- relates to
-
YARN-221 NM should provide a way for AM to tell it not to aggregate logs.
- Resolved