Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-4157

ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.10.1, 0.9.3
    • None

    Description

      In the dependency tree, there are 2 occurrences of compile scope direct netty dependencies, however, they're not used at all. I compiled locally successfully without them. E.g. when investigating blackduck alerts (complaining about netty deps for current 3.10.5.Final), it would be cleaner to start from a dependency tree where Tez doesn't depend on netty directly in order to eliminate its responsibility (and move the focus to underlying hadoop for instance).

      Tez depends on netty3 almost only in ShuffleHandler and some related classes. We can eliminate netty3 by upgrading it, but this effort might involve some testing due to fundamental changes from netty3->netty4 + we don't have a reference yet, as hadoop's ShuffleHandler is still on netty3.

      As per the netty documentation, we can also expect some performance improvement (e.g. Pooled buffers).

      Background:
      netty4 migration guideline: https://netty.io/wiki/new-and-noteworthy-in-4.0.html
      articles of possible performance improvement:
      https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html
      https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/

      Attachments

        1. dep.log
          201 kB
          László Bodor
        2. TEZ-4157.01.patch
          0.9 kB
          László Bodor
        3. TEZ-4157.02.patch
          73 kB
          László Bodor
        4. TEZ-4157.03.patch
          74 kB
          László Bodor
        5. TEZ-4157.04.patch
          75 kB
          László Bodor
        6. TEZ-4157.05.patch
          75 kB
          László Bodor
        7. TEZ-4157.06.patch
          75 kB
          László Bodor

        Issue Links

          Activity

            People

              abstractdog László Bodor
              abstractdog László Bodor
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h