Description
In some circumstances (high GC, high CPU usage, creating lots of
S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
behind processing tasks that are submitted to it; because tasks are
submitted on a regular schedule, the unbounded queue backing the
ExecutorService might grow to several gigs [2]. By using
scheduleWithFixedDelay instead, we ensure that under pressure this leak won't
happen. In order to mitigate the growth, a simple fix [3] is proposed, simply replacing scheduler.scheduleAtFixedRate by scheduler.scheduleWithFixedDelay.
[1] it is single threaded and shared across all instances of MutableQuantiles: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68
[2] see attached mutable-quantiles-leak.png.
[3] mutable-quantiles.patch
Attachments
Attachments
Issue Links
- is depended upon by
-
HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
- Resolved
- is related to
-
HADOOP-16278 With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die
- Resolved