Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-3109

pyspark can't access mariadb via jdbc

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.7.3
    • None
    • None
    • host OS: centos7
      docker: ver17.09-ce
      container: apache/zeppelin:0.7.3

    Description

      I want to access mariadb using pyspark but error raised.
      my notebook as follows.

      ```bash
      %spark.dep
      z.load("/zeppelin/interpreter/jdbc/mariadb-java-client-2.2.0.jar")
      ->
      res0: org.apache.zeppelin.dep.Dependency = org.apache.zeppelin.dep.Dependency@6ed2e421
      ```

      ```bash
      %spark.pyspark
      import pyspark
      print(pyspark._version_)

      dataframe = sqlContext.read.format("jdbc").options(
      url ="jdbc:mariadb://mariadb:3306/db",
      driver="org.mariadb.jdbc.Driver",
      dbtable="table",
      user="user",
      password="pass"
      ).load().take(10)

      ->
      2.1.0

      Traceback (most recent call last):
      File "/tmp/zeppelin_pyspark-400087723623135418.py", line 367, in <module>
      raise Exception(traceback.format_exc())
      Exception: Traceback (most recent call last):
      File "/tmp/zeppelin_pyspark-400087723623135418.py", line 360, in <module>
      exec(code, _zcUserQueryNameSpace)
      File "<stdin>", line 8, in <module>
      File "/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/readwriter.py", line 155, in load
      return self._df(self._jreader.load())
      File "/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in _call_
      answer, self.gateway_client, self.target_id, self.name)
      File "/zeppelin/interpreter/spark/pyspark/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
      return f(*a, **kw)
      File "/zeppelin/interpreter/spark/pyspark/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
      format(target_id, ".", name), value)
      Py4JJavaError: An error occurred while calling o47.load.
      : java.lang.UnsatisfiedLinkError: Unable to load library 'c': /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
      at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:145)
      at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:188)
      at com.sun.jna.Library$Handler.<init>(Library.java:123)
      at com.sun.jna.Native.loadLibrary(Native.java:255)
      at com.sun.jna.Native.loadLibrary(Native.java:241)
      at org.mariadb.jdbc.internal.util.PidFactory$CLibrary.<clinit>(PidFactory.java:112)
      at org.mariadb.jdbc.internal.util.PidFactory.lambda$getInstance$0(PidFactory.java:78)
      at org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.writeConnectAttributes(SendHandshakeResponsePacket.java:211)
      at org.mariadb.jdbc.internal.com.send.SendHandshakeResponsePacket.send(SendHandshakeResponsePacket.java:184)
      at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:796)
      at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:775)
      at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:453)
      at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1084)
      at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:493)
      at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:156)
      at org.mariadb.jdbc.Driver.connect(Driver.java:90)
      at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
      at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:59)
      at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$1.apply(JdbcUtils.scala:50)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:58)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:113)
      at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:45)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:748)
      ```

      mariadb and zeppelin container are reachable. I confirmed it by jdbc interpreter.

      Attachments

        Activity

          People

            Unassigned Unassigned
            susi29gari shota suzuki
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: