API Reference

Futures module

The following methods are part of the futures module. They can be accessed like so:

from scoop import futures

results = futures.map(func, data)
futureObject = futures.submit(func, arg)
...

More informations are available in the Usage document.

scoop.futures.as_completed(fs, timeout=None)

An iterator over the given futures that yields each as it completes.

Parameters:

• fs – The sequence of Futures (possibly created by another instance) to wait upon.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.

Returns:

An iterator that yields the given Futures as they complete (finished or cancelled).

scoop.futures.map(func, *iterables, **kargs)

Equivalent to map(func, *iterables, ...) but func is executed asynchronously and several calls to func may be made concurrently. The returned iterator raises a TimeoutError if __next__() is called and the result isn’t available after timeout seconds from the original call to map() [To be done in future version of SCOOP]. If timeout is not specified or None then there is no limit to the wait time. If a call raises an exception then that exception will be raised when its value is retrieved from the iterator.

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Futures. The callable must return a value.
• iterables – Iterable objects; each will be zipped to form an iterable of arguments tuples that will be passed to the callable object as a separate Future.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• kargs – A dictionary of additional keyword arguments that will be passed to the callable object.

Returns:

A generator of map results, each corresponding to one map iteration.

scoop.futures.shutdown(wait=True)

This function is here for compatibility with futures (PEP 3148).

Parameters:wait – Unapplied parameter.

scoop.futures.submit(func, *args, **kargs)

Submit an independent parallel scoop._types.Future that will either run locally or remotely as func(*args, **kargs).

Parameters:

• func – Any picklable callable object (function or class object with __call__ method); this object will be called to execute the Future. The callable must return a value.
• args – A tuple of positional arguments that will be passed to the callable object.
• kargs – A dictionary of additional keyword arguments that will be passed to the callable object.

Returns:

A future object for retrieving the Future result.

On return, the Future is pending execution locally, but may also be transfered remotely depending on load or on remote distributed workers. You may carry on with any further computations while the Future completes. Result retrieval is made via the result() function on the Future.

scoop.futures.wait(fs, timeout=None, return_when='ALL_COMPLETED')

Wait for the futures in the given sequence to complete.

Parameters:

• fs – The sequence of Futures (possibly created by another instance) to wait upon.
• timeout – The maximum number of seconds to wait [To be done in future version of SCOOP]. If None, then there is no limit on the wait time.
• return_when –

  Indicates when this function should return. The options are:

  FIRST_COMPLETED	Return when any future finishes or is cancelled.
  FIRST_EXCEPTION	Return when any future finishes by raising an exception. If no future raises an exception then it is equivalent to ALL_COMPLETED.
  ALL_COMPLETED	Return when all futures finish or are cancelled.

Returns:

A named 2-tuple of sets. The first set, named ‘done’, contains the futures that completed (is finished or cancelled) before the wait completed. The second set, named ‘not_done’, contains uncompleted futures.

Future class

The submit() function returns a Future object. This instance possess the following methods.

class scoop._types.Future(parentId, callable, *args, **kargs)

This class encapsulates an independent future that can be executed in parallel. A future can spawn other parallel futures which themselves can recursively spawn other futures.

add_done_callback(callable)

Attach a callable to the future that will be called when the future is cancelled or finishes running. Callable will be called with the future as its only argument.

Added callables are called in the order that they were added and are always called in a thread belonging to the process that added them. If the callable raises an Exception then it will be logged and ignored. If the callable raises another BaseException then behavior is not defined.

If the future has already completed or been cancelled then callable will be called immediately.

cancel()

If the call is currently being executed then it cannot be cancelled and the method will return False, otherwise the call will be cancelled and the method will return True.

cancelled()

True if the call was successfully cancelled, False otherwise.

done()

True if the call was successfully cancelled or finished running, False otherwise.

exception(timeout=None)

Return the exception raised by the call. If the call hasn’t yet completed then this method will wait up to timeout seconds [To be done in future version of SCOOP]. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call completed without raising then None is returned.

Returns:The exception raised by the call.

result(timeout=None)

Return the value returned by the call. If the call hasn’t yet completed then this method will wait up to ‘’timeout’’ seconds [To be done in future version of SCOOP]. If the call hasn’t completed in timeout seconds then a TimeoutError will be raised. If timeout is not specified or None then there is no limit to the wait time.

If the future is cancelled before completing then CancelledError will be raised.

If the call raised an exception then this method will raise the same exception.

Returns:The value returned by the call.

running()

True if the call is currently being executed and cannot be cancelled.