[FLINK-9969] Unreasonable memory requirements to complete examples/batch/WordCount - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0
Fix Version/s: 1.5.3, 1.6.0
Component/s: Runtime / Coordination
Labels:
- pull-request-available

Description

setup on AWS EMR:

5 worker nodes (m4.4xlarge nodes)

1 master node (m4.large)

following command fails with out of memory errors:

export HADOOP_CLASSPATH=`hadoop classpath`
./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 16000 examples/batch/WordCount.jar

Only increasing memory over 17.2GB example completes. At the same time after disabling flip6 following command succeeds:

export HADOOP_CLASSPATH=`hadoop classpath`
./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 1000 examples/batch/WordCount.jar

Attachments

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

yarn_logs
27/Jul/18 13:42
993 kB
Piotr Nowojski

Issue Links

links to

GitHub Pull Request #6479

Activity

People

Assignee:: Till Rohrmann

Reporter:: Piotr Nowojski

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 26/Jul/18 12:24

Updated:: 28/Feb/19 14:00

Resolved:: 03/Aug/18 14:39