Executing lots of schema alters concurrently can lead to dropped alters

A recently added dtest has been flapping on cassci and has exposed an issue with running lots of schema alterations concurrently. The failures occur on healthy clusters but seem to occur at higher rates when 1 node is down during the alters.

The test executes the following – 440 total commands:

• Create 20 new tables
• Drop 7 columns one at time across 20 tables
• Add 7 columns one at time across 20 tables
• Add one column index on each of the 7 columns on 20 tables

Outcome is random. Majority of the failures are dropped columns still being present, but new columns and indexes have been observed to be incorrect. The logs are don’t have exceptions and the columns/indexes that are incorrect don’t seem to follow a pattern. Running a nodetool describecluster on each node shows the same schema id on all nodes.

Attached is a python script extracted from the dtest. Running against a local 3 node cluster will reproduce the issue (with enough runs – fails ~20% on my machine).

Also attached is the node logs from a run with when a dropped column (alter_me_7 table, column s1) is still present. Checking the system_schema tables for this case shows the s1 column in both the columns and drop_columns tables.

This has been flapping on cassci on versions 2+ and doesn’t seem to be related to changes in 3.0. More testing needs to be done though.

//cc enigmacurry