Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5563

LDA with online variational inference

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.4.0
    • MLlib
    • None

    Description

      Latent Dirichlet Allocation (LDA) parameters can be inferred using online variational inference, as in Hoffman, Blei and Bach. “Online Learning for Latent Dirichlet Allocation.” NIPS, 2010. This algorithm should be very efficient and should be able to handle much larger datasets than batch algorithms for LDA.

      This algorithm will also be important for supporting Streaming versions of LDA.

      The implementation will ideally use the same API as the existing LDA but use a different underlying optimizer.

      This will require hooking in to the existing mllib.optimization frameworks.

      This will require some discussion about whether batch versions of online variational inference should be supported, as well as what variational approximation should be used now or in the future.

      Attachments

        Issue Links

          is required by

          Umbrella - An overarching type made of sub-tasks SPARK-5572 LDA improvement listing

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6259 Python API for LDA

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7421 Online LDA cleanups

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7475 adjust ldaExample for online LDA

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Documentation - Documentation or Website SPARK-7496 User guide update for Online LDA

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Pull request #4419 [Github] Pull Request #4419 (hhbyyh)

          Activity

            People

              yuhaoyan yuhao yang
              josephkb Joseph K. Bradley
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: