Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19144

Add test for GaussianMixture with distributed decompositions

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • None
    • None
    • ML

    Description

      test("check distributed decomposition") {
          val k = 5
          val d = decompositionData.head.size
          assert(GaussianMixture.shouldDistributeGaussians(k, d))
      
          val gmm = new GaussianMixture().setK(k).setSeed(seed).fit(decompositionDataset)
          assert(gmm.getK === k)
      }
      

      In ML GaussianMixtureSuite, the above test only check that when we distribute the computation that it produces a model, but does not check that it produces a correct model.
      It should have, but depends on GaussianMixture can be set with initialModel (SPARK-15785), otherwise, this algorithm seems incapable of learning even this very contrived example. After that was resolved, we can add a correctness test in the above case.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              yanboliang Yanbo Liang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: