Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19498

Discussion: Making MLlib APIs extensible for 3rd party libraries

    XMLWordPrintableJSON

Details

    • Brainstorming
    • Status: Resolved
    • Critical
    • Resolution: Incomplete
    • 2.2.0
    • None
    • ML

    Description

      Per the recent discussion on the dev list, this JIRA is for discussing how we can make MLlib DataFrame-based APIs more extensible, especially for the purpose of writing 3rd-party libraries with APIs extended from the MLlib APIs (for custom Transformers, Estimators, etc.).

      • For people who have written such libraries, what issues have you run into?
      • What APIs are not public or extensible enough? Do they require changes before being made more public?
      • Are APIs for non-Scala languages such as Java and Python friendly or extensive enough?

      The easy answer is to make everything public, but that would be terrible of course in the long-term. Let's discuss what is needed and how we can present stable, sufficient, and easy-to-use APIs for 3rd-party developers.

      Attachments

        Issue Links

          is duplicated by

          Wish - General wishlist item. SPARK-19717 Expanding Spark ML under Different Namespace

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-17048 ML model read for custom transformers in a pipeline does not work

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-8515 Improve ML attribute API

          • Major - Major loss of function.
          • Resolved

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' SPARK-5874 How to improve the current ML pipeline API?

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Brainstorming - A place to record back and forth on notions not yet formed enough to make a 'New Feature' or 'Task' SPARK-7146 Should ML sharedParams be a public API?

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-10817 ML abstraction umbrella

          • Major - Major loss of function.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-8984 Developer documentation for ML Pipelines

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              josephkb Joseph K. Bradley
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: