Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4590

Early investigation of parameter server

    XMLWordPrintableJSON

Details

    • Brainstorming
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • ML, MLlib
    • None

    Description

      In the currently implementation of GLM solvers, we save intermediate models on the driver node and update it through broadcast and aggregation. Even with torrent broadcast and tree aggregation added in 1.1, it is hard to go beyond ~10 million features. This JIRA is for investigating the parameter server approach, including algorithm, infrastructure, and dependencies.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6567 Large linear model parallelism via a join and reduceByKey

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-6932 A Prototype of Parameter Server

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              rezazadeh Reza Zadeh
              mengxr Xiangrui Meng
              Votes:
              8 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: