Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1777

Fetcher not getting all the entries in input

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Auto Closed
    • 2.2.1
    • 2.5
    • fetcher
    • None

    Description

      See comments in NUTCH-1714 :

      The Generator marks 50K entries with GENERATE_MARK but the Fetcher shows only 49,461 as Map Input Records (and the same number as Reduce input records) => looks like we are not getting all the records we should be getting. I dumped the content of the table pre-fetching and it contains the right number of entries i.e. 50K

      This was noticed after applying NUTCH-1714 and NUTCH-1674 but could also have been the case before that.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-1674 Use batchId filter to enable scan (GORA-119) for Fetch,Parse,Update,Index

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. NUTCH-1714 Nutch 2.x upgrade to Gora 0.4

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              jnioche Julien Nioche
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: