Meerkat build log
Meerkat is a p2p distributed application to maintain data synchronisation between all nodes using GRPC. Initially part of a hackathon idea proposed (available in this post)
Commits
- 3f8b029 -> Migrating to GRPC and testing out basic node pinging.
- 2291e07 -> Implement a basic pool joining protocol.
- e9da45e -> Data transmission with failure in edge cases. Using 2 go routines to handle different aspects of the program.
- 7a51506 -> Creating mutex locks to fix data race issues persistent in the program.
- 6acc241 -> Basic graceful exit for simple fault tolerance during signal interrupts
- b8ffb9f -> Creating Disk Snapshots to identify events of
DELETION,ADDITION(updates and addition are broadcasted as the same type of event for now).
Issues
- Issue #7: Streaming file contents as chunks
- Current method passes all bytes in one go through a stream. This is an issue with larger files
- Issue #6: fix: file deletion broadcasting
- file deletions
- folder operations are shadowed
- Issue #5: _ Handle unexpected disconnects from nodes_
- Handle interrupts
- Handle fatal erros
- Issue #4: Dealing with merging 2 network pools
- Focus on this issue is to work with network partitions.
- Current system only deals with how a node can enter an existing network pool.
- Issue #3: Move towards a gossip protocol
- Broadcast of changes happes to all nodes
- Move to gossip to reduce network overload, the goal is that data propagates to all nodes eventually.
- Issue #2: Handshakes and Disconnects not going through
- Duplicate addition of nodes in the client list
- Safe node leaving boradcast doesn't propagate across the nodes
- Issue #1: Attempt to connect to client twice
Research for future improvements
- Raft Consensus Algorithm Paper
- Distributed Systems lecture by Martin Kleppmann