Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34095

Spark 3.0.1 is giving WARN while running job

Details

    • Bug
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • 3.0.1
    • None
    • Java API
    • None

    Description

      I am running Spark 3.0.1 with Java11. Getting the following WARN.

       

      WARNING: An illegal reflective access operation has occurred
      WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
      WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
      WARNING: All illegal access operations will be denied in a future release

      Attachments

        Issue Links

          Activity

            smurarka Sachit Murarka added a comment -

            sowen holden -> Can you please suggest something on this?

            smurarka Sachit Murarka added a comment - sowen holden  -> Can you please suggest something on this?

             

            smurarka I am getting a similar warning to this, which I am unable to hide in Jupyter Lab running openjdk-11-jre-headless/now 11.0.11+9-0ubuntu2~20.04.

            I've set the log4j.logger.org.apache.spark.api.python.PythonGatewayServer setting to ERROR to try to hide this, however this has not worked for me (despite setting the Spark context logger to ERROR by default).

            This is somewhat annoying for users trying to launch a Spark session in a notebook

            Here is the output:

            Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48) 
            [GCC 9.3.0] on linux
            Type "help", "copyright", "credits" or "license" for more information.
            WARNING: An illegal reflective access operation has occurred
            WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
            WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
            WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
            WARNING: All illegal access operations will be denied in a future release
            Welcome to
                  ____              __
                 / __/__  ___ _____/ /__
                _\ \/ _ \/ _ `/ __/  '_/
               /__ / .__/\_,_/_/ /_/\_\   version 3.1.2
                  /_/Using Python version 3.9.6 (default, Jul 11 2021 03:39:48)
            Spark context Web UI available at http://4d0ddf19f818:4040
            Spark context available as 'sc' (master = local[*], app id = local-1629293803625).
            SparkSession available as 'spark'.
            

             

             

            berglh Berg Lloyd-Haig added a comment -   smurarka I am getting a similar warning to this, which I am unable to hide in Jupyter Lab running openjdk-11-jre-headless/now 11.0.11+9-0ubuntu2~20.04. I've set the log4j.logger.org.apache.spark.api.python.PythonGatewayServer setting to ERROR to try to hide this, however this has not worked for me (despite setting the Spark context logger to ERROR by default). This is somewhat annoying for users trying to launch a Spark session in a notebook Here is the output: Python 3.9.6 | packaged by conda-forge | ( default , Jul 11 2021, 03:39:48) [GCC 9.3.0] on linux Type "help" , "copyright" , "credits" or "license" for more information. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar) to constructor java.nio.DirectByteBuffer( long , int ) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 3.1.2 /_/Using Python version 3.9.6 ( default , Jul 11 2021 03:39:48) Spark context Web UI available at http: //4d0ddf19f818:4040 Spark context available as 'sc' (master = local[*], app id = local-1629293803625). SparkSession available as 'spark' .    

            People

              Unassigned Unassigned
              smurarka Sachit Murarka
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated: