Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-1139

Tthread pool size in tests are not enough in a 32 core machine if gateway port mapping enabled

    XMLWordPrintableJSON

Details

    • Test
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • Tests
    • None

    Description

      Currently Jetty calculates acceptor and selector count as a fraction of processor count (core/8 and core/2 respectively) and will maximize them as 4. That means 4 acceptor and 4 select in a 32+ core machine for each ServerConnector. In Server.doStart jetty will check if maxThreadPoolSize is large enough to store a request thread and all acceptors and selectors thread for each connector together.
      Currently in tests thread pool max size fixed as 16. In GatewayServer.start if gateway port mapping is enabled it will initialize two connectors in jetty which on a 32+ core machine will create 4+4 acceptor and 4+4 selector. With the one extra request thread it will exceed the 16 thread pool limit and org.apache.hadoop.gateway.GatewayPortMappingFuncTest will throw the following exception:

      java.lang.IllegalStateException: Insufficient threads: max=16 < needed(acceptors=8 + selectors=8 + request=1)
      	at org.apache.hadoop.gateway.GatewayPortMappingFuncTest.setup(GatewayPortMappingFuncTest.java:96)
      

      This is only a unit test problem since the default value for thread pool size in GatewayConfigImpl is 254.

      Probably it would be a better method to calculate thread pool count from core count or max thread count in tests or just use the default value from GatewayConfigImpl.

      Attachments

        1. KNOX-1139.patch
          1.0 kB
          Attila Csoma

        Activity

          People

            csomaati Attila Csoma
            csomaati Attila Csoma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: