public class


extends Object
   ↳ org.apache.helix.taskexecution.TaskExecutionDemo

Class Overview

Demo for execution of task Dag using primitives provided by Helix. This demo sets up a Dag of tasks for providing analytics for impression and click events. Each node on the dag has an id and declares the desired parallelism and IDs of the nodes it depends on. When we submit this dag for execution using submitDag() TaskCluster, we create a Helix resource for each node with number of partitions based on desired parallelism. We use a "OnlineOffline" state model. The demo starts NUM_WORKERS workers. Each worker is given a TaskFactory and TaskResultStore. Each worker is assigned a bunch of task partitions by Helix and gets state transition messages for the task partitions it is assigned. When the worker gets a state transition message for a task partition, it first checks if all upstream dependencies are satisfied by making sure that corresponding partitions have transitioned to "Online" state. It then creates a Task object using TaskFactory based on resource name (since task IDs have been mapped to resource names) The demo has the following steps

  1. Start zookeeper
  2. Setup task cluster TaskCluster
  3. Start Helix controller
  4. Populate dummy impression and click data
  5. Start workers
  6. Submit dag


Public Constructors
Public Methods
static void main(String[] args)
Inherited Methods
From class java.lang.Object

Public Constructors

public TaskExecutionDemo ()

Public Methods

public static void main (String[] args)