Major Description
bin/flume-ng tried to load hbase/hadoop class path by this:
103 local HBASE_CLASSPATH="" 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \ 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \ 106 java.library.path 2>/dev/null)
It actually turned out to be:
$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar \ org.apache.flume.tools.GetJavaProperty java.library.path
However what I saw is:
-bash-3.2$ hbase -cp ../lib/flume-ng-core-1.2.0-incubating-SNAPSHOT.jar org.apache.flume.tools.GetJavaProperty java.library.path /usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/usr/lib/hbase/bin/../lib/native/Linux-amd64-64 Heap par new generation total 235968K, used 8391K [0x00000002fae00000, 0x000000030ae00000, 0x000000030ae00000) eden space 209792K, 4% used [0x00000002fae00000, 0x00000002fb631f30, 0x0000000307ae0000) from space 26176K, 0% used [0x0000000307ae0000, 0x0000000307ae0000, 0x0000000309470000) to space 26176K, 0% used [0x0000000309470000, 0x0000000309470000, 0x000000030ae00000) concurrent mark-sweep generation total 20709376K, used 0K [0x000000030ae00000, 0x00000007fae00000, 0x00000007fae00000) concurrent-mark-sweep perm gen total 21248K, used 2724K [0x00000007fae00000, 0x00000007fc2c0000, 0x0000000800000000)
The hbase gc info outputs to stdout and screwed up the flume-ng script.
The root cause is the combination of several factors:
1. turn on hbase gc log by:
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateS tamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
2. the gc log directory is protected by limiting the permission as 755, and owned by hbase user.
3. use another user, such as flume, to execute the script.
Since flume user doesn't have write permission to the hbase gc log directory, jvm will output the gc info to stdout, and the flume script will be screwed up.
A simple but tricky fix could be adding ``grep hbase'' in the scrip to filter out the gc info:
103 local HBASE_CLASSPATH="" 104 local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \ 105 ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \ 106 java.library.path | grep hbase 2>/dev/null)
Attachments
Attachments
Issue Links
- links to
