Using zookeeper in distributed systems

Apache Zookeeper is a crucial component used for coordination in distributed systems. It can be deployed as a highly available centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Deployment:
The usual deployment is to have a ensemble (quorum) of odd number of zookeeper servers, depending on the number of clients it is servicing. The rationale is that, in case of failure of a few servers, the zookeeper service can still continue to function if there is still a majority.

For example, a 3 node ensemble will tolerate failure of 1 node because the other 2 form a majority. Compare this with a 4 node ensemble, which can still tolerate failure of just 1 node.

Based on a "leader election algorithm", one of the nodes is designated leader. All requests from clients go to this elected leader, which synchronizes the data with the remaining nodes

Data model:
The data model of zookeeper resembles the tree structure of unix filesystem. Each path refers to a zookeeper node (znode). Znodes are allowed to have data as well as metadata associated with them. Further, clients can create ephemeral znodes, which are deleted when the client terminates its session.

Watches:
The power of zookeeper comes with the concept of watches on znodes. A watch will be triggered when a znode changes. We will see watches in action in the example below.

Example:
Here is an algorithm to implement global locking. Clients wishing to obtain a lock do the following:

1. Call create( ) with a pathname of "_locknode_/guid-lock-" and the sequence and ephemeral flags set. The guid is needed in case the create() result is missed. See the note below.
2. Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect).
3. If the pathname created in step 1 has the lowest sequence number suffix, the client has the lock and the client exits the protocol.
4. The client calls exists( ) with the watch flag set on the path in the lock directory with the next lowest sequence number.
5. if exists( ) returns false, go to step 2. Otherwise, wait for a notification for the pathname from the previous step before going to step 2.

The unlock protocol is very simple: clients wishing to release a lock simply delete the node they created in step 1.


References:
1. http://zookeeper.apache.org