API Reference

Futures module

The following methods are part of the futures module. They can be accessed like so:

from scoop import futures

results = futures.map(func, data)
futureObject = futures.submit(func, arg)
...

More informations are available in the Usage document.

scoop.futures.as_completed(fs, timeout=None)

Iterates over the given futures that yields each as it completes. This call is blocking.

Parameters:

• fs – The sequence of Futures to wait upon.
• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

Returns:

An iterator that yields the given Futures as they complete (finished or cancelled).

scoop.futures.map(func, *iterables)

Equivalent to map(func, *iterables, ...) but func is executed asynchronously and several calls to func may be made concurrently. This non-blocking call returns an iterator which raises a TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to map(). If timeout is not specified or None then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator.

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

Returns:

A generator of map results, each corresponding to one map iteration.

scoop.futures.map_as_completed(func, *iterables)

Equivalent to map, but the results are returned as soon as they are made available.

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

Returns:

A generator of map results, each corresponding to one map iteration.

scoop.futures.shutdown(wait=True)

This function is here for compatibility with futures (PEP 3148) and doesn’t have any behavior.

Parameters:wait – Unapplied parameter.

scoop.futures.wait(fs, timeout=-1, return_when='ALL_COMPLETED')

Wait for the futures in the given sequence to complete. Using this function may prevent a worker from executing.

Parameters:

• fs – The sequence of Futures to wait upon.
• timeout – The maximum number of seconds to wait. If negative or not specified, then there is no limit on the wait time.
• return_when –

  Indicates when this function should return. The options are:

  FIRST_COMPLETED	Return when any future finishes or is cancelled.
  FIRST_EXCEPTION	Return when any future finishes by raising an exception. If no future raises an exception then it is equivalent to ALL_COMPLETED.
  ALL_COMPLETED	Return when all futures finish or are cancelled.

Returns:

A named 2-tuple of sets. The first set, named ‘done’, contains the futures that completed (is finished or cancelled) before the wait completed. The second set, named ‘not_done’, contains uncompleted futures.

Future class

The submit() function returns a Future object. This instance possesses the following methods.

class scoop._types.Future(parentId, callable, *args, **kargs)

This class encapsulates an independent future that can be executed in parallel. A future can spawn other parallel futures which themselves can recursively spawn other futures.

add_done_callback(callable_, inCallbackType='standard', inCallbackGroup=None)

Attach a callable to the future that will be called when the future is cancelled or finishes running. Callable will be called with the future as its only argument.

Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.

If the future has already completed or been cancelled then callable will be called immediately.

cancel()

If the call is currently being executed or sent for remote execution, then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.

cancelled()

Returns True if the call was successfully cancelled, False otherwise.

done()

Returns True if the call was successfully cancelled or finished running, False otherwise. This function updates the executionQueue so it receives all the awaiting message.

exception(timeout=None)

Return the exception raised by the call. If the call hasn’t yet completed then this method will wait up to timeout seconds. More information in the Usage page. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call completed without raising then None is returned.

Returns:The exception raised by the call.

result(timeout=None)

Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to ‘’timeout’’ seconds. More information in the Usage page. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call raised an exception then this method will raise the same exception.

Returns:The value returned by the callable object.

running()

Returns True if the call is currently being executed and cannot be cancelled.

Shared module

This module provides the setConst() and getConst() functions allowing arbitrary object sharing between futures. These objects can only be defined once and cannot be modified once shared, hence the name constant.

class scoop.shared.SharedElementEncapsulation(element)

Encapsulates a reference to an element available in the shared module.

This is used by Futures (map on lambda, for instance).

scoop.shared.getConst(name, timeout=0.1)

Get a shared constant.

Parameters:

• name – The name of the shared variable to retrieve.
• timeout – The maximum time to wait in seconds for the propagation of the constant.

Returns:

The shared object.

Usage: value = getConst(‘name’)

SCOOP Constants and objects

The following objects are available to a program that was launched using SCOOP.

Note

Please note that using these is considered as advanced usage. You should not rely on these for other purposes than debugging.

Constants	Description
scoop.IS_ORIGIN	Boolean value. True if current instance is the root worker.
scoop.BROKER	broker.broker.BrokerInfo namedtuple. Address, ports and hostname of the broker.
scoop.DEBUG	Boolean value. True if debug mode is enabled, false otherwise.
scoop.IS_RUNNING	Boolean value. True if SCOOP is currently running, false otherwise.
scoop.worker	2-tuple. Unique identifier of the current instance in the pool.
scoop.logger	Logger object. Provides log formatting and redirection facilities. See the official documentation for more information on its usage.