Zebroo Sets

⏳ WIP

Data is passed as Zebroo Sets between workers. There are several ways to then access the Zebroo Set data within a worker.

Record

Available for workers which receive a list of data as an input from preceding pipeline steps. record references the currently processed data object.

record.display_name = f"{record.first_name} - {record.last_name}"
record #return record

Data

Available for workers which receive a list of data as an input from preceding pipeline steps. data references the complete set of input data passed to the worker.

all_display_names = []

for rec in data:  # Iterate through all records
    all_display_names.append(f"{rec.first_name} - {rec.last_name}")

all_display_names  # Return all_display_names

Tip: if data is a list with a single element, here’s a shortcut for accessing the first list element: data.attribute1 can be used in place of of data[0].attribute1.

Index

Available for workers which receive a list of data as an input from preceding pipeline steps. index references the index number of the currently processed record.

Indexing starts at 0.

Type: Integer

record.position = index
record #return record

HTTP Response

Available for workers that make HTTP/S requests. response references the response content of the request triggered by the worker.

value = response.value

# When receiving headers is enabled on the worker
headers = response.headers

...

project.com/records/start_index={response.last_index + 1}&pagination_size={response.page_size}