Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31851 Redesign PySpark documentation
  3. SPARK-32186

Development - Debugging

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • Documentation, PySpark
    • None

    Description

      1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspark/profiler.html, 

      >>> sc._conf.set("spark.python.profile", "true")
>>> rdd = sc.parallelize(range(100)).map(str)
>>> rdd.count()
100
>>> sc.show_profiles()

      2. Python Debugger, e.g.) https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html#remote-interpreter (?)

      3. Monitoring Python Workers: top, `ls`, etc. (?)

      4. PyCharm setup (?) (at https://issues.apache.org/jira/browse/SPARK-32189)

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-32179 Replace and redesign the documentation base

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #29639 [Github] Pull Request #29639 (HyukjinKwon)

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: