Class QueueSharder<U,T extends QueueBase<U>>

All Implemented Interfaces:
Closeable, AutoCloseable

public class QueueSharder<U,T extends QueueBase<U>> extends Object implements Closeable

A utility for shard a distributed queue.

Due to limitations in ZooKeeper's transport layer, a single queue will break if it has more than 10K-ish items in it. This class provides a facade over multiple distributed queues. It monitors the queues and if any one of them goes over a threshold, a new queue is added. Puts are distributed amongst the queues.

NOTE: item ordering is maintained within each managed queue but cannot be maintained across queues. i.e. items might get consumed out of order if they are in different managed queues.

  • Constructor Details

    • QueueSharder

      public QueueSharder(CuratorFramework client, QueueAllocator<U,T> queueAllocator, String queuePath, String leaderPath, QueueSharderPolicies policies)
      client - client
      queueAllocator - allocator for new queues
      queuePath - path for the queues
      leaderPath - path for the leader that monitors queue sizes (must be different than queuePath)
      policies - sharding policies
  • Method Details

    • start

      public void start() throws Exception
      The sharder must be started
      Exception - errors
    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
    • getQueue

      public T getQueue()
      Return one of the managed queues - the selection method cannot be relied on. It should be considered a random managed queue.
      a queue
    • getShardQty

      public int getShardQty()
      Return the current number of mananged queues
    • getQueuePaths

      public Collection<String> getQueuePaths()
      Return the current set of shard paths