Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-15094

Warning about using private constructor of java.nio.DirectByteBuffer in Java 11

Details

    Description

      The unsafe off-heap in FLINK-13985 was implemented by hacking into a private constructor of java.nio.DirectByteBuffer. This causes undesirable warnings in Java 11:

      WARNING: An illegal reflective access operation has occurred
      WARNING: Illegal reflective access by org.apache.flink.core.memory.MemoryUtils (file:/C:/Development/repos/flink/flink-core/target/classes/) to constructor java.nio.DirectByteBuffer(long,int)
      WARNING: Please consider reporting this to the maintainers of org.apache.flink.core.memory.MemoryUtils
      WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
      WARNING: All illegal access operations will be denied in a future release

      We should look into this, confirm that it is harmless and try to silence it to avoid confusion for users.

      Attachments

        Issue Links

          Activity

            githubbot ASF GitHub Bot logged work - 25/Feb/20 09:31
            • Time Spent:
              10m
               
              azagrebin commented on pull request #11160: [FLINK-15094] Use Unsafe to instantiate and construct DirectByteBuffer
              URL: https://github.com/apache/flink/pull/11160
               
               
                 
               
              ----------------------------------------------------------------
              This is an automated message from the Apache Git Service.
              To respond to the message, please log on to GitHub and use the
              URL above to go to the specific comment.
               
              For queries about this service, please contact Infrastructure at:
              users@infra.apache.org
            githubbot ASF GitHub Bot logged work - 20/Feb/20 14:11
            • Time Spent:
              10m
               
              azagrebin commented on pull request #11160: [FLINK-15094] Use Unsafe to instantiate and construct DirectByteBuffer
              URL: https://github.com/apache/flink/pull/11160
               
               
                 ## What is the purpose of the change
                 
                 If we use reflection to create a DirectByteBuffer to wrap unsafe native memory allocations, it causes illegal access warnings in Java9+.
                 
                 This PR changes this to use Unsafe to instantiate a DirectByteBuffer. The address and capacity fields are set by direct unsafe memory operations. Other fields are set by calling ByteBuffer#clear at the end.
                 Unsafe operations skips the illegal access verification and do not result in warnings. The same approach is used to get the address field.
                 
                 This solution still relies on Unsafe which is about to be removed in future Java releases. If it is removed and we still do not want to contribute to direct memory by allocating native managed memory, we will have to find an alternative solution, like e.g. writing a custom native allocator and use JNI API to instantiate the wrapping DirectByteBuffer (NewDirectByteBuffer in C).
                 
                 ## Verifying this change
                 
                 unit tests
                 
                 
               
              ----------------------------------------------------------------
              This is an automated message from the Apache Git Service.
              To respond to the message, please log on to GitHub and use the
              URL above to go to the specific comment.
               
              For queries about this service, please contact Infrastructure at:
              users@infra.apache.org

            People

              azagrebin Andrey Zagrebin
              azagrebin Andrey Zagrebin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m