Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20767

The training continuation for saved LDA model

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 2.1.1
    • None
    • ML
    • None

    Description

      Current online implementation of the LDA model fit (OnlineLDAOptimizer) does not support the model update (ie. to account for the population/covariates drift) nor the continuation of model fitting in case of the insufficient number of iterations.

      Technical aspects:

      1. The implementation of LDA fitting does not currently allow the coefficients pre-setting (private setter), as noted by a comment in the source code of OnlineLDAOptimizer.setLambda: "This is only used for testing now. In the future, it can help support training stop/resume".

      2. The lambda matrix is always randomly initialized by the optimizer, which needs fixing for preset lambda matrix.

      The adaptation of the classes by the user is not possible due to protected setters & sealed / final classes.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              cezden Cezary Dendek
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: