Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2316

Incorrect results for FILTER *** BY ( *** OR ***) with FilterLogicExpressionSimplifier optimizer turned on

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 0.8.0, 0.8.1, 0.9.0, 0.9.1
    • 0.9.2, 0.10.0, 0.8.2
    • None
    • None
    • Hide
      FilterLogicExpressionSimplifier optimization rule is disabled by default. To enable the optimization rule, set the property pig.exec.filterLogicExpressionSimplifier to true.
      Show
      FilterLogicExpressionSimplifier optimization rule is disabled by default. To enable the optimization rule, set the property pig.exec.filterLogicExpressionSimplifier to true.

    Description

      An example for this bug:

      cat weird.txt
      1,a
      2,b
      3,c

      When running pig with the following statements:

      A = LOAD 'weird.txt' using PigStorage(',') AS (col1:int,col2);
      B = FILTER A BY ((col1==1) OR (col1 != 1));
      DUMP B;

      I expect to get the result of all three rows back, but I receive only two rows.

      (2,b)
      (3,c)

      When we start pig with optimizer turning off.

      pig -optimizer_off All

      With optimizer turning off, we get the expected results and I get three rows for the same statements.

      (1,a)
      (2,b)
      (3,c)

      --------------------------------------------------------

      This bug was test on:

      pig-0.9.1,
      pig-0.9.0,
      pig-0.8.1,
      pig-0.8.0

      All produced same incorrect results.

      --------------------------------------------------------

      When looked at the logical plan for this example, we found FilterlogicExpressionSimplifier optimizer produced incorrect logical plan. So we guess the bug is caused by FilterlogicExpressionSimplifier optimizer.

      Attachments

        1. pig-2316-08-v3.txt
          9 kB
          Thejas Nair
        2. pig-2316-09-v3.txt
          8 kB
          Thejas Nair
        3. pig-2316-trunk-v3.txt
          8 kB
          Thejas Nair
        4. pig-2316-trunk-v1.txt
          4 kB
          Koji Noguchi

        Issue Links

          Activity

            People

              knoguchi Koji Noguchi
              wenfengbx Huanyu Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: