Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-481

Support independent output committers for multiple outputs

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.12.0
    • Core
    • None

    Description

      I faced this issue while trying to write to Kite and HDFS in the same pipeline. A similar issue was logged for Kite[1][2].
      I was attempting to write a PCollection to Kite and a different PTable to HDFS as a text file. The write to Kite succeeded, however the write to HDFS only produced a _SUCCESS file with no text file.
      Commenting out the write to Kite seems to solve the issue and I can see the text file being written.

      [1] - https://issues.cloudera.org/browse/CDK-756
      [2] - http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3CCAF-WD4QCUe0Toh3qewpDNnom3u786PVJLgH7T6Go_AbcTpLTaw@mail.gmail.com%3E

      Attachments

        1. CRUNCH-481.patch
          19 kB
          Thomas White
        2. CRUNCH-481.patch
          17 kB
          Thomas White
        3. CRUNCH-481.patch
          14 kB
          Thomas White
        4. CRUNCH-481.patch
          18 kB
          Josh Wills
        5. CRUNCH-481c.patch
          17 kB
          Josh Wills
        6. CRUNCH-481-hadoop-2-compat.patch
          19 kB
          Ryan Brush

        Activity

          People

            jwills Josh Wills
            aniketk Aniket Kulkarni
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: