Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-1139

Tthread pool size in tests are not enough in a 32 core machine if gateway port mapping enabled

Details

    • Test
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • Tests
    • None

    Description

      Currently Jetty calculates acceptor and selector count as a fraction of processor count (core/8 and core/2 respectively) and will maximize them as 4. That means 4 acceptor and 4 select in a 32+ core machine for each ServerConnector. In Server.doStart jetty will check if maxThreadPoolSize is large enough to store a request thread and all acceptors and selectors thread for each connector together.
      Currently in tests thread pool max size fixed as 16. In GatewayServer.start if gateway port mapping is enabled it will initialize two connectors in jetty which on a 32+ core machine will create 4+4 acceptor and 4+4 selector. With the one extra request thread it will exceed the 16 thread pool limit and org.apache.hadoop.gateway.GatewayPortMappingFuncTest will throw the following exception:

      java.lang.IllegalStateException: Insufficient threads: max=16 < needed(acceptors=8 + selectors=8 + request=1)
      	at org.apache.hadoop.gateway.GatewayPortMappingFuncTest.setup(GatewayPortMappingFuncTest.java:96)
      

      This is only a unit test problem since the default value for thread pool size in GatewayConfigImpl is 254.

      Probably it would be a better method to calculate thread pool count from core count or max thread count in tests or just use the default value from GatewayConfigImpl.

      Attachments

        1. KNOX-1139.patch
          1.0 kB
          Attila Csoma

        Activity

          csomaati Attila Csoma added a comment -

          A possible solution using the same default value from GatewayConfigImpl

          csomaati Attila Csoma added a comment - A possible solution using the same default value from GatewayConfigImpl

          Commit cb982448bab78619dd09c5a812ec60f521773e15 in knox's branch refs/heads/v0.14.0 from moresandeep
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=cb98244 ]

          KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)

          jira-bot ASF subversion and git services added a comment - Commit cb982448bab78619dd09c5a812ec60f521773e15 in knox's branch refs/heads/v0.14.0 from moresandeep [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=cb98244 ] KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)

          Commit 19362b9dd99586e64ba4cadc6e29a41d40e41f72 in knox's branch refs/heads/master from moresandeep
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=19362b9 ]

          KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)

          jira-bot ASF subversion and git services added a comment - Commit 19362b9dd99586e64ba4cadc6e29a41d40e41f72 in knox's branch refs/heads/master from moresandeep [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=19362b9 ] KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)
          smore Sandeep More added a comment -

          csomaati thanks for pointing it out and providing the patch !
          I have checked it in 0.14.0 as well as master.

          smore Sandeep More added a comment - csomaati thanks for pointing it out and providing the patch ! I have checked it in 0.14.0 as well as master.
          csomaati Attila Csoma added a comment -

          moresandeep I just realized, that there are two GatewayTestConfig (in gateway-test and in gateway-test-release-utils) both of them affected with this issue but the patch I provided only fixed it in one file. Should I reopen this with the new patch or create a new ticket with the same issue and fix the other GatewayTestConfig there?

          csomaati Attila Csoma added a comment - moresandeep I just realized, that there are two GatewayTestConfig (in gateway-test and in gateway-test-release-utils) both of them affected with this issue but the patch I provided only fixed it in one file. Should I reopen this with the new patch or create a new ticket with the same issue and fix the other GatewayTestConfig there?
          smore Sandeep More added a comment -

          Hello csomaati the issue seems to be for the GatewayPortMappingFuncTest, which uses the patched GatewayTestConfig class so we should be fine here !

          smore Sandeep More added a comment - Hello csomaati the issue seems to be for the GatewayPortMappingFuncTest, which uses the patched GatewayTestConfig class so we should be fine here !
          csomaati Attila Csoma added a comment -

          Thanks moresandeep. If I'm not mistaken the GatewayPortMappingFuncTest uses the GatewayTestConfig but from gateway-test and the patch modified the gateway-test-release-utils' GatewayTestConfig.

          csomaati Attila Csoma added a comment - Thanks moresandeep . If I'm not mistaken the GatewayPortMappingFuncTest uses the GatewayTestConfig but from gateway-test and the patch modified the gateway-test-release-utils' GatewayTestConfig.
          smore Sandeep More added a comment -

          Looks like we only have one version of 'GatewayTestConfig' (atleast for master and 0.14.0) thanks to some much needed refactoring for our UnitTests

          smore Sandeep More added a comment - Looks like we only have one version of 'GatewayTestConfig' (atleast for master and 0.14.0) thanks to some much needed refactoring for our UnitTests
          csomaati Attila Csoma added a comment -

          I see, thank you moresandeep for the clarification. I messed up something locally and was not aware of it

          csomaati Attila Csoma added a comment - I see, thank you moresandeep for the clarification. I messed up something locally and was not aware of it
          smore Sandeep More added a comment -

          NP thanks for double checking and making sure no bugs slip through !!

          smore Sandeep More added a comment - NP thanks for double checking and making sure no bugs slip through !!

          Commit 19362b9dd99586e64ba4cadc6e29a41d40e41f72 in knox's branch refs/heads/KNOX-998-Package_Restructuring from moresandeep
          [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=19362b9 ]

          KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)

          jira-bot ASF subversion and git services added a comment - Commit 19362b9dd99586e64ba4cadc6e29a41d40e41f72 in knox's branch refs/heads/ KNOX-998 -Package_Restructuring from moresandeep [ https://git-wip-us.apache.org/repos/asf?p=knox.git;h=19362b9 ] KNOX-1139 - Fix threapool size for UnitTests (Attila Csoma via Sandeep More)

          People

            csomaati Attila Csoma
            csomaati Attila Csoma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: