Examples

You can find the examples detailed on this page and more in the examples/ directory of SCOOP.

Please check the API Reference for any implentation detail of the proposed functions.

Introduction to the map() function

A core concept of task-based parallelism as presented in SCOOP is the map. An introductory example to map working is presented in examples/mapDoc.py.

1
2
3
4
5
6
7
8
9
from __future__ import print_function
from scoop import futures

def helloWorld(value):
    return "Hello World from Future #{0}".format(value)

if __name__ == "__main__":
    returnValues = list(futures.map(helloWorld, range(16)))
    print("\n".join(returnValues))

Line 1 allows Python 2 users to have a print function compatible with Python 3.

On line 2, SCOOP is imported.

On line 4-5, the function that will be mapped is declared.

The condition on line 7 is a safety barrier that prevents the main program to be executed on every workers. It ensures that the map is issued only by one worker, the root.

The map() function is located on line 8. It launches the helloWorld function 16 times, each time with a different argument value selected from the range(16) argument. This method is compatible with the standard Python map() function and thus can be seamlessly interchanged without modifying its arguments.

The example then prints the return values of every calls on line 9.

You can launch this program using python -m scoop. The output should look like this:

~/scoop/examples$ python -m scoop -n 8 mapDoc.py
Hello World from Future #0
Hello World from Future #1
Hello World from Future #2
[...]

Note

Results of a map are always ordered even if their computation was made asynchronously on multiple computers.

Note

You can toy around with the previous example by changing the second parameter of the map() function. Is it working with string arrays, pure strings or other variable types?

Computation of \pi

A Monte-Carlo method to calculate \pi using SCOOP to parallelize its computation is found in examples/piCalc.py. You should familiarize yourself with Monte-Carlo methods before going forth with this example.

Monte Carlo computation of Pi.

Image from Wikipedia made by CaitlinJo that shows the Monte Carlo computation of \pi.

First, we need to import the needed functions as such:

1
2
3
from math import hypot
from random import random
from scoop import futures

The Monte-Carlo method is then defined. It spawns two pseudo-random numbers that are fed to the hypot function which calculates the hypotenuse of its parameters. This step computes the Pythagorean equation (\sqrt{x^2+y^2}) of the given parameters to find the distance from the origin (0,0) to the randomly placed point (which X and Y values were generated from the two pseudo-random values). Then, the result is compared to one to evaluate if this point is inside or outside the unit disk. If it is inside (have a distance from the origin lesser than one), a value of one is produced (red dots in the figure), otherwise the value is zero (blue dots in the figure). The experiment is repeated tries number of times with new random values.

The function returns the number times a pseudo-randomly generated point fell inside the unit disk for a given number of tries.

1
2
def test(tries):
    return sum(hypot(random(), random()) < 1 for _ in range(tries))

One way to obtain a more precise result with a Monte-Carlo method is to perform the method multiple times. The following function executes repeatedly the previous function to gain more precision. These calls are handled by SCOOP using it’s map() function. The results, that is the number of times a random distribution over a 1x1 square hits the unit disk over a given number of tries, are then summed and divided by the total of tries. Since we only covered the upper right quadrant of the unit disk because both parameters are positive in a cartesian map, the result must be multiplied by 4 to get the relation between area and circumference, namely \pi.

1
2
3
def calcPi(nbFutures, tries):
    expr = futures.map(test, [tries] * nbFutures)
    return 4. * sum(expr) / float(nbFutures * tries)

As previously stated, you must wrap your code with a test for the __main__ name.

1
2
if __name__ == "__main__":
    print("pi = {}".format(calcPi(3000, 5000)))

You can now run your code using the command python -m scoop.

Sharing Constant

One usage of shared constants is to halt a computation when a worker has found a solution such as a brute forcing example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
from itertools import product, tee
import string
from scoop import futures, shared

# Create the hash to brute force
HASH_TO_FIND = hash("SCO")


def generateHashes(iterator):
    """Compute hashes of given iterator elements"""
    for combination in iterator:
        # Stop as soon as a worker finds the solution
        if shared.getConst('Done', timeout=0):
            return False

        # Compute the current combination hash
        currentString = "".join(combination).strip()
        if hash(currentString) == HASH_TO_FIND:
            # Share to every other worker that the solution has been found
            shared.setConst(Done=True)
            return currentString
    
    # Report that computing has not ended
    return False


if __name__ == "__main__":
    # Generate possible characters
    possibleCharacters = []
    possibleCharacters.extend(list(string.ascii_uppercase))
    possibleCharacters.extend(' ')

    # Generate the solution space.
    stringIterator = product(possibleCharacters, repeat=3)

    # Partition the solution space into iterators 
    # Keep in mind that the tee operator evaluates the whole solution space
    # making it pretty memory inefficient.
    SplittedIterator = tee(stringIterator, 1000)

    # Parallelize the solution space evaluation
    results = futures.map(generateHashes, SplittedIterator)

    # Loop until a solution is found
    for result in results:
        if result:
            break

    print(result)

Overall example

The examples/fullTree.py example holds a wrap-up of available SCOOP functionnalities. It notably shows that SCOOP is capable of handling twisted and complex hierarchical requirements.

Getting acquainted with the previous examples is fairly enough to use SCOOP, no need to dive into this complicated example.

Manual

Back to Welcome

Previous topic

Usage

Next topic

API Reference

This Page