Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Currently, we are able to pass in Spark DataFrames with Vector type columns as input to SystemML scripts to be converted to a SystemML matrix. We should add the Vector type to the frame conversion code too.
Example: Given a Spark DataFrame of schema <double, vector, string>, we should be able to convert that to a SystemML frame with a bunch of double columns and the final string column.
ok, thanks again for pointing out this missing functionality. We now support dataframe-frame conversions with mixed schemas that include vector columns. We make one simplifying assumption though: we only allow a single vector column in the schema (but at arbitrary positions and mixed with arbitrary scalar fields) because this allows us to handle schema information without looking at the data (for the vector sizes).