Query waiting indefinitely for table metadata to arrive

Impala queries may hang indefinitely while waiting for the metadata of a deleted table to arrive through a statestore topic update. You will see many messages like this in the log of the impalad coordinating the hung query:

Missing tables were not received in 120000ms. Load request will be retried. <list of tables>

If one of the tables mentioned in those log messates has been deleted, then you may be hitting this issue.

This code in Frontend#getMissingTbls() clearly shows the bug:

  private Set<TableName> getMissingTbls(Set<TableName> tableNames) {
    Set<TableName> missingTbls = new HashSet<TableName>();
    for (TableName tblName: tableNames) {
      Db db = getCatalog().getDb(tblName.getDb());
      if (db == null) continue; <--- wrong! database has been dropped and may never arrive
      Table tbl = db.getTable(tblName.getTbl());
      if (tbl == null) continue; <--- wrong! table has been dropped and may never arrive
      if (!tbl.isLoaded()) missingTbls.add(tblName);
    }
    return missingTbls;
  }

Getting into this hung state requires an elaborate series of events, for example:

• impalad A requests table T to be loaded and gets into the wait loop
• impalad B issues a "DROP TABLE T"
• catalogd loads the metadata for table T
• statestored requests topic update from catalogd; update includes T
• statestored sends update to impalad B
• impalad B completes the "DROP TABLE T" operation
• statestored requests topic update from catalogd; update includes deletion of T
• statestored sends update to impalad A which includes the deletion of table T
• impalad A is still in the wait loop; the metadata for T will never arrive because T has been dropped

Notice how impalad A may "skip" the first update for T which includes the metadata for T. This typically only happens on very busy clusters where the statestore has trouble sending all catalog snapshots to all subscribers in a timely fashion (i.e. some subscribers skip some snapshots).

Workaround

• Re-create tables with the same name as the deleted ones (schema and format do not matter, only the dabatase and table name must match)
• Might need to run "invalidate metadata <table>" on them
• Once the hung queries finished (failed or succeeded), the re-created tables can be dropped again