Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
None
Description
There has been some talk about Giraph's scaling limitations due to keeping the whole graph and messages in RAM.
We need to investigate methods to fall back to disk when running out of memory, while gracefully degrading performance.
This issue is for graph storage. Messages should probably be a separate issue, although the interplay between the two is crucial.
We should also discuss what are our primary goals here: completing a job (albeit slowly) instead of failing when the graph is too big, while still encouraging memory optimizations and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode, making it almost a standard way of operating.
Attachments
Attachments
Issue Links
- is related to
-
GIRAPH-45 Improve the way to keep outgoing messages
- Resolved
-
GIRAPH-247 Introduce edge based partitioning for InputSplits
- Resolved