Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
4.0.0-alpha-2
-
None
Description
At the moment, column statistics tables in the metastore schema look like this (it's similar for PART_COL_STATS):
CREATE TABLE "APP"."TAB_COL_STATS"(
"CAT_NAME" VARCHAR(256) NOT NULL,
"DB_NAME" VARCHAR(128) NOT NULL,
"TABLE_NAME" VARCHAR(256) NOT NULL,
"COLUMN_NAME" VARCHAR(767) NOT NULL,
"COLUMN_TYPE" VARCHAR(128) NOT NULL,
"LONG_LOW_VALUE" BIGINT,
"LONG_HIGH_VALUE" BIGINT,
"DOUBLE_LOW_VALUE" DOUBLE,
"DOUBLE_HIGH_VALUE" DOUBLE,
"BIG_DECIMAL_LOW_VALUE" VARCHAR(4000),
"BIG_DECIMAL_HIGH_VALUE" VARCHAR(4000),
"NUM_DISTINCTS" BIGINT,
"NUM_NULLS" BIGINT NOT NULL,
"AVG_COL_LEN" DOUBLE,
"MAX_COL_LEN" BIGINT,
"NUM_TRUES" BIGINT,
"NUM_FALSES" BIGINT,
"LAST_ANALYZED" BIGINT,
"CS_ID" BIGINT NOT NULL,
"TBL_ID" BIGINT NOT NULL,
"BIT_VECTOR" BLOB,
"ENGINE" VARCHAR(128) NOT NULL
);
The idea is to have a single blob named STATISTICS to replace them, as follows:
CREATE TABLE "APP"."TAB_COL_STATS"(
"CAT_NAME" VARCHAR(256) NOT NULL,
"DB_NAME" VARCHAR(128) NOT NULL,
"TABLE_NAME" VARCHAR(256) NOT NULL,
"COLUMN_NAME" VARCHAR(767) NOT NULL,
"COLUMN_TYPE" VARCHAR(128) NOT NULL,
"STATISTICS" BLOB,
"LAST_ANALYZED" BIGINT,
"CS_ID" BIGINT NOT NULL,
"TBL_ID" BIGINT NOT NULL,
"ENGINE" VARCHAR(128) NOT NULL
);
The STATISTICS column could be the serialization of a Json-encoded string, which will be consumed in a "schema-on-read" fashion.
At first at least the removed column statistics will be encoded in the STATISTICS column, but since each "consumer" will read the portion of the schema it is interested into, multiple engines (see the ENGINE column) can read and write statistics as they deem fit.
Another advantage is that, if we plan to add more statistics in the future, we won't need to change the thrift interface for the metastore again.
Attachments
Issue Links
- relates to
-
-
- Closed
-