Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0, 2.1.0, 2.2.0
-
None
Description
Inside this chunk of code
if (systemFreeMemoryMb < mediumMemoryThresholdMb) { if (memoryLimitExceededStart < 0) { memoryLimitExceededStart = Time.currentTimeMillis(); } else { long timeInViolation = Time.currentTimeMillis() - memoryLimitExceededStart; if (timeInViolation > mediumMemoryGracePeriodMs) { LOG.warn( "{} is using {} MB > memory limit {} MB for {} seconds", typeOfCheck, usageMb, memoryLimitMb, timeInViolation / 1000); return true; } } }
At very beginning, memoryLimitExceededStart in BasicContainer is initialized as 0. :
https://github.com/apache/storm/blob/2.2.x-branch/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L80
protected volatile long memoryLimitExceededStart;
So once it hits this scenario, the grace period doesn't really take any effect because the timeInViolation will be very large (equals to currentTime)
The logs from a test:
2020-06-08 20:39:18.277 o.a.s.d.s.BasicContainer SLOT_6707 [WARN] WORKER 9c16e81e-4936-4029-bcda-ceb5b74b8f42 is using 167 MB > memory limit 158 MB for 1591648758 seconds
