Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-49055

Investigate OrcEncryptionSuite UnsatisfiedLinkError on AppleSilicon MacOS environment

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • 4.0.0
    • None
    • SQL
    • None

    Description

      git reset --hard 49b4c3bc9c09325de941dfaf41e4fd3a4a4c345f // [SPARK-45393][BUILD] Upgrade Hadoop to 3.4.0
      build/sbt clean "sql/testOnly org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite"
      ...
      [info] OrcEncryptionSuite:
      12:42:55.441 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      12:42:57.950 WARN org.apache.hadoop.crypto.OpensslCipher: Failed to load OpenSSL Cipher.
      java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl()'
      	at org.apache.hadoop.util.NativeCodeLoader.buildSupportsOpenssl(Native Method)
      	at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:86)
      	at org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.<init>(OpensslAesCtrCryptoCodec.java:36)
      	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
      	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
      [info] - Write and read an encrypted file (2 seconds, 486 milliseconds)
      [info] - Write and read an encrypted table (402 milliseconds)
      [info] - SPARK-35325: Write and read encrypted nested columns (299 milliseconds)
      [info] - SPARK-35992: Write and read fully-encrypted columns with default masking (623 milliseconds)
      12:42:59.856 WARN org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite: 
      
      ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.orc.OrcEncryptionSuite, threads: rpc-boss-3-1 (daemon=true), Thread-17 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true), Thread-18 (daemon=true), ForkJoinPool.commonPool-worker-3 (daemon=true) =====
      [info] Run completed in 5 seconds, 291 milliseconds.
      [info] Total number of tests run: 4
      [info] Suites: completed 1, aborted 0
      [info] Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
      [info] All tests passed.
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              LuciferYang Yang Jie
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: