Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
In general, there are three core points of distributed flow refactoring:
1. Using only one interface instead of two in order to process distributed configuration and corresponding events. Currently, both ConfigurationRegistry and MetastorageManager are used. Besides the obvious point that it leads to some mess it also means that metastorage keys should be exposed from configuration registry.
In order to get rid of MetastorageManager as a direct interface that provides access to configuration it's required to
- Support internal/private properties in configuration framework
- Add ability to retrieve distributed up-to-date value with configuration registry. Something like ConfigurationProperty.distirbuteValue() is expected. As a result, it will allow changing
//TODO This is an egregious violation of encapsulation. Current approach has to be revisited. private Set<String> tableNamesConfigured() { IgniteBiTuple<ByteArray, ByteArray> range = toRange(new ByteArray(PUBLIC_PREFIX)); Set<String> tableNames = new HashSet<>(); try (Cursor<Entry> cursor = metaStorageMgr.range(range.get1(), range.get2())) { while (cursor.hasNext()) { Entry entry = cursor.next(); List<String> keySplit = ConfigurationUtil.split(entry.key().toString()); if (keySplit.size() == 5 && NamedListNode.NAME.equals(keySplit.get(4))) tableNames.add(ByteUtils.fromBytes(entry.value()).toString()); } } catch (Exception e) { LOG.error("Can't get table names.", e); } return tableNames; }
to something similar to
private Set<String> tableNamesConfigured() {
return new HashSet<>(clusterCfgMgr.configurationRegistry().getConfiguration(TablesConfiguration.KEY).
tables().distirbuteValue().namedListKeys());
}
2. Using 1 step flow instead of 2 step one:
Currently, creation of a table configuration triggers corresponding listeners that calculate assignments and schema descriptor and persists them to meta storage. After implementing internal configuration extensions it'll be possible to combine given steps in most cases, so as a result something like the following might be used:
clusterCfgMgr
.configurationRegistry()
.getConfiguration(TablesConfiguration.KEY)
.tables()
.change(
change -> change.create(
name,
(ch) -> {
tableInitChange.accept(ch);
((ExtendedTableChange)ch).changeAssignments(
ByteUtils.toBytes(affMgr.calculateAssignments(ch.partitions(), ch.replicas())));
((SchemaExtendedTableChange)ch).changeSchemaDescriptor(
ByteUtils.toBytes(schemaMgr.prepareSchemaDescriptor(tableInitChange)));
}
)
)
.get();
Open issue here is table id generation. In case of using rev and update counter from meta storage we still need two phase creation process with second step that will persist table id to the configuration.
3. Using appropriate listeners instead of wide ones: instead of listening tableUpdates and manual routing to particular schema update it's possible to listen to every column type change, column rename, etc directly.
Attachments
Issue Links
- relates to
-
-
- Resolved
-