Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-770

Timebomb for Fetcher

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.1
    • None
    • None
    • Patch Available

    Description

      This patch provides the Fetcher with a timebomb mechanism. By default the timebomb is not activated; it can be set using the parameter fetcher.timebomb.mins. The number of minutes is relative to the start of the Fetch job. When the number of minutes is reached, the QueueFeeder skips all remaining entries then all active queues are purged. This allows to keep the Fetch step under comtrol and works well in combination with NUTCH-769

      Attachments

        1. NUTCH-770-v3.patch
          6 kB
          Julien Nioche
        2. NUTCH-770-v2.patch
          47 kB
          Julien Nioche
        3. log-770
          0.9 kB
          MilleBii
        4. NUTCH-770.patch
          5 kB
          Julien Nioche

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-769 Fetcher to skip queues for URLS getting repeated exceptions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-769 Fetcher to skip queues for URLS getting repeated exceptions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Activity

            People

              ab Andrzej Bialecki
              jnioche Julien Nioche
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: