Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5874 How to improve the current ML pipeline API?
  3. SPARK-7176

Add validation functionality to individual Param

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.4.0
    • ML
    • None

    Description

      This issue was brought up by https://issues.apache.org/jira/browse/SPARK-7147: There are multiple locations in which a parameter can be specified (in setters and in calling fit/transform), and we should check the parameter value in each one. It would be nice to avoid duplicating the checking code.

      Proposed change:

      • Add field to class Param: val isValid: T => Boolean, with default value of always returning true
      • Add helper methods in new ParamCheck object for constructing common validation checks (especially to help Java users)
      • Ensure that this validation is always done, most likely by always settings parameters by constructing a ParamPair + validating inside the ParamPair constructor

      Attachments

        Issue Links

          Activity

            People

              josephkb Joseph K. Bradley
              josephkb Joseph K. Bradley
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: