Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4440

FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • fairscheduler
    • None
    • Reviewed

    Description

      It seems there is a bug on FSAppAttempt#getAllowedLocalityLevelByTime method

      // default level is NODE_LOCAL
          if (! allowedLocalityLevel.containsKey(priority)) {
            allowedLocalityLevel.put(priority, NodeType.NODE_LOCAL);
            return NodeType.NODE_LOCAL;
          }
      

      If you first invoke this method, it doesn't init time in lastScheduledContainer and this will lead to execute these code for next invokation:

          // check waiting time
          long waitTime = currentTimeMs;
          if (lastScheduledContainer.containsKey(priority)) {
            waitTime -= lastScheduledContainer.get(priority);
          } else {
            waitTime -= getStartTime();
          }
      

      the waitTime will subtract to FsApp startTime, and this will be easily more than the delay time and allowedLocality degrade. Because FsApp startTime will be start earlier than currentTimeMs. So we should add the initial time of priority to prevent comparing with FsApp startTime and allowedLocalityLevel degrade. And this problem will have more negative influence for small-jobs. The YARN-4399 also discuss some problem in aspect of locality.

      Attachments

        1. YARN-4440.003.patch
          5 kB
          Yiqun Lin
        2. YARN-4440.002.patch
          5 kB
          Yiqun Lin
        3. YARN-4440.001.patch
          5 kB
          Yiqun Lin

        Issue Links

          Activity

            People

              linyiqun Yiqun Lin
              linyiqun Yiqun Lin
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: