Unit test failures on a test build/port of Kudu on ppc64le architecture

Hello,

We are in the process of porting Apache:Kudu  V 1.15.0 to IBM ppc64le (PowerPC Linux Little Endian) platform. We have a test build and ran the unit tests using “ctest” mentioned in the build instructions. The results are as follows:

On ppc64le:

93% tests passed, 31 tests failed out of 456

On x86-64

93% tests passed, 34 tests failed out of 456

The results are similar, except for a few failing testcases failing on the individual architecture.

Our familiarity with the design and code base of Kudu is very recent. We appreciate help and guidance from those in the community who are familiar with these testcases in getting these testcases resolved. The summary of results these is attached at the end of this note.

To initiate discussion,  I shall pick one testcase which failed on the ppc64le only: compaction-test. This test consists of 23 sub-tests. On x86-64 all 23 pass. On ppc64le 22 of them pass and only the following fails:

TEST_F(TestCompaction, TestFlushMRSWithRolling)

The  TestFlushMRSWithRolling  has three assertions. On ppc64le the first passes and the next two fail.

rows.clear();

  rowsets[0]->DebugDump(&rows);

  EXPECT_EQ(R"(RowIdxInBlock: 0; Base: (string key="hello 00000000", int32 val=0, )"

                "int32 nullable_val=0); Undo Mutations: [@1(DELETE)]; Redo Mutations: [];",

            rows[0]);

  rows.clear();

  rowsets[1]->DebugDump(&rows);

  EXPECT_EQ(R"(RowIdxInBlock: 0; Base: (string key="hello 00017150", int32 val=1715, )"

            "int32 nullable_val=NULL); Undo Mutations: [@1716(DELETE)]; Redo Mutations: [];",

            rows[0]);

  EXPECT_EQ(R"(RowIdxInBlock: 1; Base: (string key="hello 00017160", int32 val=1716, )"

            "int32 nullable_val=1716); Undo Mutations: [@1717(DELETE)]; Redo Mutations: [];",

            rows[1]);

By tracing the data, it seems that the number of records per rowset  is off (less) by 5 when compared with the x86-64 implementation. For example, the rowset[0] has 1750 records (17 blocks of 100 each + 18^th^block at 15) on x86-64; on ppc64le rowset[0] has 1700 records (17 block blocks of each 100 + 18^th^ block). Consequently, the composition of the first block of  the rowset[1] on x86-64 is as follows:

CompactionInputRowToString(input_row)

RowIdxInBlock: 0; Base: (string key="hello 00017150", int32 val=1715, int32 nullable_val=NULL); Undo Mutations: [@1716(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 1; Base: (string key="hello 00017160", int32 val=1716, int32 nullable_val=1716); Undo Mutations: [@1717(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 2; Base: (string key="hello 00017170", int32 val=1717, int32 nullable_val=NULL); Undo Mutations: [@1718(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 3; Base: (string key="hello 00017180", int32 val=1718, int32 nullable_val=1718); Undo Mutations: [@1719(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 4; Base: (string key="hello 00017190", int32 val=1719, int32 nullable_val=NULL); Undo Mutations: [@1720(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 5; Base: (string key="hello 00017200", int32 val=1720, int32 nullable_val=1720); Undo Mutations: [@1721(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 6; Base: (string key="hello 00017210", int32 val=1721, int32 nullable_val=NULL); Undo Mutations: [@1722(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

On ppc64le it is:

Block 0

CompactionInputRowToString(input_row)

RowIdxInBlock: 0; Base: (string key="hello 00017100", int32 val=1710, int32 nullable_val=1710); Undo Mutations: [@1711(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 1; Base: (string key="hello 00017110", int32 val=1711, int32 nullable_val=NULL); Undo Mutations: [@1712(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 2; Base: (string key="hello 00017120", int32 val=1712, int32 nullable_val=1712); Undo Mutations: [@1713(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 3; Base: (string key="hello 00017130", int32 val=1713, int32 nullable_val=NULL); Undo Mutations: [@1714(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 4; Base: (string key="hello 00017140", int32 val=1714, int32 nullable_val=1714); Undo Mutations: [@1715(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

RowIdxInBlock: 5; Base: (string key="hello 00017150", int32 val=1715, int32 nullable_val=NULL); Undo Mutations: [@1716(DELETE)]; Redo Mutations: [];

CompactionInputRowToString(input_row)

This means on ppc64le the second assertion on RowIdxInBlock: 5 passes. We trying to locate in the code where the mismatch of records entry is happening between the implementations.

Appreciate your help to resolve this and the other unit tests as we fully implement Kudu on ppc64le.

Summary of Unit test case Failures on ppc64le and x86-64

Thanks, and regards

Hari