Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22479

SaveIntoDataSourceCommand logs jdbc credentials

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.2.1, 2.3.0
    • SQL
    • None

    Description

      JDBC credentials are not redacted in plans including a 'SaveIntoDataSourceCommand'.

      Steps to reproduce:

      spark-shell --packages org.postgresql:postgresql:42.1.1
      
      import org.apache.spark.sql.execution.QueryExecution
      import org.apache.spark.sql.util.QueryExecutionListener
      val listener = new QueryExecutionListener {
        override def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit = {}
        override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = {
          System.out.println(qe.toString())
        }
      }
      spark.listenerManager.register(listener)
      spark.range(100).write.format("jdbc").option("url", "jdbc:postgresql:sparkdb").option("password", "pass").option("driver", "org.postgresql.Driver").option("dbtable", "test").save()
      

      The above will yield the following plan:

      == Parsed Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Analyzed Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Optimized Logical Plan ==
      SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
         +- Range (0, 100, step=1, splits=Some(8))
      
      == Physical Plan ==
      ExecutedCommand
         +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists
               +- Range (0, 100, step=1, splits=Some(8))
      

      Attachments

        Issue Links

          Activity

            People

              onursatici Onur Satici
              onursatici Onur Satici
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: