Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
Description
SYSTEMML-1194 replaced SQLContext with SparkSession in SystemML for Spark 2.1.0 since SQLContext is deprecated.
Restore the old Java SQLContext method signatures in case any users are using SystemML methods and are unable to use SparkSessions (SparkSessions are described in https://databricks.com/blog/2016/08/15/how-to-use-sparksession-in-apache-spark-2-0.html)
Classes where this applies:
old MLContext class (whole class is deprecated)
old MLMatrix class (whole class is deprecated)
old MLOutput class (whole class is deprecated)
FrameRDDConverterUtils (this is a non-API class)
RDDConverterUtils (this is a non-API class)
RDDConverterUtilsExt (this is a non-API class)
In non-API classes, these SQLContext methods should be marked as deprecated and removed in a future version of SystemML (1.0) since SparkSessions should generally be used with Spark 2. As mentioned in SQLContext documentation, "As of Spark 2.0, this is replaced by SparkSession. However, we are keeping the class here for backward compatibility."
Fixed by PR396.