API Reference

Futures module

The following methods are part of the futures module. They can be accessed like so:

from scoop import futures

results = futures.map(func, data)
futureObject = futures.submit(func, arg)
...

More informations are available in the Usage document.

scoop.futures.as_completed(fs, timeout=None)

An iterator over the given futures that yields each as it completes.

Parameters:

• fs – The sequence of Futures (possibly created by another instance) to wait upon.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.

Returns:

An iterator that yields the given Futures as they complete (finished or cancelled).

scoop.futures.map(func, *iterables, **kargs)

Equivalent to map(func, *iterables, ...) but func is executed asynchronously and several calls to func may be made concurrently. The returned iterator raises a TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to map() [To be done in future version of SCOOP]. If timeout is not specified or None then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator.

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• kargs – A dictionary of additional keyword arguments that will be passed to the callable object.

Returns:

A generator of map results, each corresponding to one map iteration.

scoop.futures.mapReduce(mapFunc, reductionOp, *iterables, **kargs)

Exectues the map() function and then applies a reduction function to its result. The reduction function will cumulatively merge the results of the map function in order to get a final single value.

Parameters:

• mapFunc – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• reductionOp – Any picklable callable object (function or class object with __call__ method); this object will be called to reduce the Futures results. The callable must support two parameters and return a single value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• kargs – A dictionary of additional keyword arguments that will be passed to the mapFunc callable object.

Returns:

A single value.

scoop.futures.mapScan(mapFunc, reductionOp, *iterables, **kargs)

Exectues the map() function and then applies a reduction function to its result while keeping intermediate reduction values.

Parameters:

• mapFunc – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• reductionOp – Any picklable callable object (function or class object with __call__ method); this object will be called to reduce the Futures results. The callable must support two parameters and return a single value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• kargs – A dictionary of additional keyword arguments that will be passed to the mapFunc callable object.

Returns:

Every return value of the reduction function applied to every mapped data sequentially ordered.

scoop.futures.shutdown(wait=True)

This function is here for compatibility with futures (PEP 3148).

Parameters:wait – Unapplied parameter.

scoop.futures.submit(func, *args, **kargs)

Submit an independent parallel Future that will either run locally or remotely as func(*args, **kargs).

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Future. The callable must return a value.
• args – A tuple of positional arguments that will be passed to the callable object.
• kargs – A dictionary of additional keyword arguments that will be passed to the callable object.

Returns:

A future object for retrieving the Future result.

On return, the Future can be pending execution locally but may also be transfered remotely depending on load or on remote distributed workers. You may carry on with any further computations while the Future completes. Result retrieval is made via the result() function on the Future.

scoop.futures.wait(fs, timeout=None, return_when='ALL_COMPLETED')

Wait for the futures in the given sequence to complete.

Parameters:

• fs – The sequence of Futures (possibly created by another instance) to wait upon.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• return_when –

  Indicates when this function should return. The options are:

  FIRST_COMPLETED	Return when any future finishes or is cancelled.
  FIRST_EXCEPTION	Return when any future finishes by raising an exception. If no future raises an exception then it is equivalent to ALL_COMPLETED.
  ALL_COMPLETED	Return when all futures finish or are cancelled.

Returns:

A named 2-tuple of sets. The first set, named ‘done’, contains the futures that completed (is finished or cancelled) before the wait completed. The second set, named ‘not_done’, contains uncompleted futures.

Future class

The submit() function returns a Future object. This instance possess the following methods.

class scoop._types.Future(parentId, callable, *args, **kargs)

This class encapsulates an independent future that can be executed in parallel. A future can spawn other parallel futures which themselves can recursively spawn other futures.

add_done_callback(callable, inCallbackType='standard', inCallbackGroup=None)

Attach a callable to the future that will be called when the future is cancelled or finishes running. Callable will be called with the future as its only argument.

Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.

If the future has already completed or been cancelled then callable will be called immediately.

cancel()

If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.

cancelled()

True if the call was successfully cancelled, False otherwise.

done()

True if the call was successfully cancelled or finished running, False otherwise.

exception(timeout=None)

Return the exception raised by the call. If the call hasn’t yet completed then this method will wait up to timeout seconds [To be done in future version of SCOOP]. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call completed without raising then None is returned.

Returns:The exception raised by the call.

result(timeout=None)

Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to ‘’timeout’’ seconds [To be done in future version of SCOOP]. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call raised an exception then this method will raise the same exception.

Returns:The value returned by the call.

running()

True if the call is currently being executed and cannot be cancelled.

Shared module

This module provides the setConst() and getConst() functions allowing arbitrary object sharing between futures.

scoop.shared.getConst(name, timeout=0.1)

Get a constant that was shared beforehand.

Parameters:

• name – The name of the shared variable to retrieve.
• timeout – The maximum time to wait in seconds for the propagation of the variable.

Returns:

The shared object.

Usage: value = getConst(‘name’)

scoop.shared.setConst(**kwargs)

Set a constant that will be shared to every workers.

Parameters:**kwargs – One or more combination(s) key=value. Key being the variable name and value the object to share. Returns:None.

Usage: setConst(name=value)

SCOOP Constants

The following variables are available to a program that was launched using SCOOP.

Note

Please note that using these is considered as advanced usage. You should not use these for other purposes than debugging.

Constants	Description
scoop.IS_ORIGIN	Boolean value. True if current instance is the root worker.
scoop.WORKER_NAME	String value. Name of the current instance.
scoop.BROKER_NAME	String value. Name of the broker to which this instance is attached.
scoop.BROKER_ADDRESS	String value. Address of the socket communicating work information.
scoop.META_ADDRESS	String value. Address of the socket communicating meta information.
scoop.SIZE	Integer value. Size of the current worker pool.
scoop.DEBUG	Boolean value. True if debug mode is enabled, false otherwise.
scoop.worker	2-tuple. Unique identifier of the current instance in the pool.