I get connection error when I try to run map-reduce.sh command with kafka.pull conf

Hi

Any idea about this error... i can see my spark/hadoop/kafka looks fine but when i run the command
bin/gobblin-mapreduce.sh start --conf /home/ubuntu/gobblin/gobblin-dist/conf/config/kafka.pull

Exception in thread main java.net.ConnectException: Call From <<host name/IP>> to localhost:8020 failed on connection exception: java.net.ConnectException:e: http://wiki.apache.org/hadoop/ConnectionRefused

Github Url : https://github.com/linkedin/gobblin/issues/394
Github Reporter : prasadnh
Github Created At : 2015-10-19T23:48:51Z
Github Updated At : 2016-03-04T17:24:06Z

Comments

---

zliu41 wrote on 2015-10-19T23:52:31Z : @prasadnh please double check if `localhost:8020` is the correct uri for your hdfs. You may use `hdfs getconf -confKey fs.default.name` to verify.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149379315

---

prasadnh wrote on 2015-10-20T07:46:09Z : thanks for the response...
so we cannot have different port for hdfs?but we are using 9000...

-Prasad

On Mon, Oct 19, 2015 at 4:52 PM, Ziyang Liu notifications@github.com
wrote:

> @prasadnh https://github.com/prasadnh please double check if
> localhost:8020 is the correct uri for your hdfs. You may use hdfs getconf
> -confKey fs.default.name to verify.
>
> —
> Reply to this email directly or view it on GitHub
> https://github.com/linkedin/gobblin/issues/394#issuecomment-149379315.

Thanks,
Prasad

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149465429

---

liyinan926 wrote on 2015-10-20T16:14:55Z : @prasadnh please change the setting of `fs.uri` in `conf/gobblin-mapreduce.properties` to the correct URI for the NameNode of your HDFS setup, e.g., `fs.uri=hdfs://localhost:9000`.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149619685

---

prasadnh wrote on 2015-10-20T23:33:35Z : Thanks... I put wrong port number in properties file...updated and tried again...

but I see same issue

Exception in thread main java.net.ConnectException: Call From <> to localhost:9000 failed on connection exception: java.net.ConnectException:e: http://wiki.apache.org/hadoop/ConnectionRefused

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149734800

---

prasadnh wrote on 2015-10-20T23:49:44Z : I think we cannot give localhost...instead we need to hostname with 9000...but i get different exception

Exception in thread main java.lang.IllegalArgumentException: java.net.UnknownHostException: hdfs

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149737371

---

zliu41 wrote on 2015-10-21T01:46:24Z : @prasadnh you can take a look at the stack trace. If the exception happens during the initialization of state store, you need to set `state.store.fs.uri=hdfs://localhost:9000`.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149754742

---

prasadnh wrote on 2015-10-21T01:59:42Z : This value already there in properties file
state.store.fs.uri=${fs.uri}

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-149756858

---

prasadnh wrote on 2015-10-22T00:48:29Z : what could be the reason command hangs forever here... don't see any logs or trace to findout whats happening

./gobblin-mapreduce.sh stop --conf /home/ubuntu/gobblin/gobblin-dist/conf/config/kafka.pull
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ubuntu/gobblin/gobblin-dist/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/bin/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150065429

---

liyinan926 wrote on 2015-10-22T05:29:38Z : `gobblin-mapreduce.sh` does not support the option `stop`.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150112017

---

prasadnh wrote on 2015-10-22T08:51:31Z : oops copied wrong command... same behavior with start

./gobblin-mapreduce.sh start --conf /home/ubuntu/gobblin/gobblin-dist/conf/config/kafka.pull
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ubuntu/gobblin/gobblin-dist/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/bin/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150153370

---

liyinan926 wrote on 2015-10-22T16:03:51Z : `gobblin-mapreduce.sh` does not support the option `start` either... Have you read https://github.com/linkedin/gobblin/wiki/Gobblin%20Deployment#Hadoop-MapReduce-Deployment?

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150272234

---

prasadnh wrote on 2015-10-22T18:03:29Z : sorry...not sure why I used this stop/start... i was running standalone before and just changed to mapreduce....
even no start/stop same issue... SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

./gobblin-mapreduce.sh --conf /home/ubuntu/gobblin/gobblin-dist/conf/config/kafka.pull
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/ubuntu/gobblin/gobblin-dist/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/bin/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150307625

---

prasadnh wrote on 2015-10-22T18:19:30Z : I see process running but i don't see any logs, not sure its really working.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150311552

---

liyinan926 wrote on 2015-10-22T18:25:42Z : Please remove `gobblin-dist/lib/slf4j-log4j12-1.7.5.jar` and give a try.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-150313657

---

liyinan926 wrote on 2015-10-29T19:16:22Z : @prasadnh any update on this issue?

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-152291732

---

prasadnh wrote on 2015-10-30T00:35:16Z : No update... No success so far. 
I am working on upgrading our servers... Will try one more time once it's up and running.

Sent from Outlook

On Thu, Oct 29, 2015 at 12:16 PM -0700, Yinan Li notifications@github.com wrote:

@prasadnh any update on this issue?

—
Reply to this email directly or view it on GitHub.

Github Url : https://github.com/linkedin/gobblin/issues/394#issuecomment-152367571