Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Testing the federated instructions with misaligned matrices (either by different hosts or ranges) and fixing the appearing problems.
I found two issues with the AggregateBinaryFEDInstruction and the MMFEDInstruction:
- There is an alignment check without verifying that the federated data partitions are located on the same host (only checking for alignment of the federated ranges)
- We do not create the output matrix for the respective federated spark instruction which results in an exception that the variable is not found (MMFEDInstruction)
The BinaryMatrixMatrixFEDInstruction throws an Execution when getting two misaligned matrices, instead of pulling one federated matrix to the driver and executing it like in the other cases with only one federated matrix.
The AppendFEDInstruction doesn't support aligned federated matrices. Append spark instructions are only rewritten to federated instructions if the second input is not federated.
Attachments
Issue Links
- links to
Commit 97a8a4a91324bfa4085e6b33310a02215a26a6af in systemds's branch refs/heads/main from ywcb00
[ https://gitbox.apache.org/repos/asf?p=systemds.git;h=97a8a4a ]
SYSTEMDS-3206Federated Execution w/ Misaligned MatricesThis commit modify the Federated Execution of:
to allow for and test Misaligned matrix inputs, indicating federated
sites that are not aligned across operations. Also contained is removal
of some code across the federated instructions. In Detail:
NOTE: otherwise, MatrixObject.readBlobFromHDFS()
-> CacheableData.acquireReadAndRelease() -> CacheableData.acquireRead()
-> CacheableData.acquireReadIntern() -> MatrixObject.readBlobFromHDFS()
is forming an endless loop
but the hdfs file and RDDhandle does not exist
set the matrix characteristics according to the output dimensions
Closes #1446