Description
JDBC credentials are not redacted in plans including a 'SaveIntoDataSourceCommand'.
Steps to reproduce:
spark-shell --packages org.postgresql:postgresql:42.1.1
import org.apache.spark.sql.execution.QueryExecution import org.apache.spark.sql.util.QueryExecutionListener val listener = new QueryExecutionListener { override def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit = {} override def onSuccess(funcName: String, qe: QueryExecution, duration: Long): Unit = { System.out.println(qe.toString()) } } spark.listenerManager.register(listener) spark.range(100).write.format("jdbc").option("url", "jdbc:postgresql:sparkdb").option("password", "pass").option("driver", "org.postgresql.Driver").option("dbtable", "test").save()
The above will yield the following plan:
== Parsed Logical Plan == SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists +- Range (0, 100, step=1, splits=Some(8)) == Analyzed Logical Plan == SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists +- Range (0, 100, step=1, splits=Some(8)) == Optimized Logical Plan == SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists +- Range (0, 100, step=1, splits=Some(8)) == Physical Plan == ExecutedCommand +- SaveIntoDataSourceCommand jdbc, Map(dbtable -> test10, driver -> org.postgresql.Driver, url -> jdbc:postgresql:sparkdb, password -> pass), ErrorIfExists +- Range (0, 100, step=1, splits=Some(8))
Attachments
Issue Links
- breaks
-
SPARK-23567 spark.redaction.regex should not include user by default, docs not updated
- Resolved
-
SPARK-23850 We should not redact username|user|url from UI by default
- Resolved
- relates to
-
SPARK-25726 Flaky test: SaveIntoDataSourceCommandSuite.`simpleString is redacted`
- Resolved
- links to