Current large volume Batch Processing is often limited to one machine, they are run sequentially, have a single point of failure and provide no dashboard to monitor and scale on-demand. Process logging is non-existent or clunky.
Opturo’s Network Processing product provides a robust platform with a decentralized architecture, an Administration Console and one or many Operational Clusters. Following are the components:
- An Operational Cluster consists of one “Manager/Broker” and multiple “Worker” daemon processes. Worker daemon within a Cluster can run on one or more machines on the network.
- A Manager/Broker within a Cluster maintains a list of “Queues” and distributes “Jobs” to Workers.
- Each Worker/Consumer daemon subscribes to Jobs on one or more Queues and processes them.
- A Batch Process with one or more “Iterations” or “Jobs”, is initiated by a “Producer” which sends a message to the Manager with list of Jobs/Iterations and a Queue name.
- A Job/Message or Iteration is one loop of any process, such as calculating analytics on 20 accounts (20 Iterations) or processing 100 files (100 Iterations).
Using this architecture, the processing of any Batch Process with one or more Jobs/Iterations can be distributed across multiple machines on a network. This drastically cut run times and improves reliability.
Administration Console allows real-time monitoring current Batch Processes. Provides the ability to scale Workers on-demand to handle larger volumes.
