Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1486

Support multi-model training in MLlib

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Auto Closed
    • None
    • None
    • MLlib

    Description

      It is rare in practice to train just one model with a given set of parameters. Usually, this is done by training multiple models with different sets of parameters and then select the best based on their performance on the validation set. MLlib should provide native support for multi-model training/scoring. It requires decoupling of concepts like problem, formulation, algorithm, parameter set, and model, which are missing in MLlib now. MLI implements similar concepts, which we can borrow. There are different approaches for multi-model training:

      0) Keep one copy of the data, and train models one after another (or maybe in parallel, depending on the scheduler).

      1) Keep one copy of the data, and train multiple models at the same time (similar to `runs` in KMeans).

      2) Make multiple copies of the data (still stored distributively), and use more cores to distribute the work.

      3) Collect the data, make the entire dataset available on workers, and train one or more models on each worker.

      Users should be able to choose which execution mode they want to use. Note that 3) could cover many use cases in practice when the training data is not huge, e.g., <1GB.

      This task will be divided into sub-tasks and this JIRA is created to discuss the design and track the overall progress.

      Attachments

        Issue Links

          Activity

            People

              brkyvz Burak Yavuz
              mengxr Xiangrui Meng
              Votes:
              8 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: