ytree.utilities.parallel.parallel_nodes¶

ytree.utilities.parallel.parallel_nodes(trees, group='forest', save_every=None, filename=None, njobs=None, dynamic=None)[source]¶

Iterate over all nodes in a list of trees in parallel.

Both trees and/or nodes within a tree are divided up between available process groups using multi-level parallelism. Analysis field values can then be assigned to all nodes (halos). Trees will be saved either at the end of the loop over all trees or after a number of trees given by the save_every keyword are completed.

This uses the yt parallel_objects function, which is parallelized with MPI underneath and so is suitable for parallelism across compute nodes.

Parameters:

trees (list of TreeNode objects) – The trees to be iterated over in parallel.
group (optional, str ("forest", "tree", or "prog")) – Determines the nodes to be iterated over in the tree: “forest” for all nodes in the forest, “tree” for all nodes in the tree, or “prog” for all nodes in the line of main progenitors. Default: “forest”
save_every (optional, int or False) – Number of trees to be completed before results are saved. This is used to save intermediate results in case scripts need to be restarted. If None, save will only occur after iterating over all trees. If False, no saving will be done. Default: None
filename (optional, string) – The name of the new arbor to be saved. If None, the naming convention will follow the filename keyword of the save_arbor function. Default: None
njobs (optional, tuple of ints) – The number of process groups for parallel iteration over trees and nodes within each tree. The first value sets behavior for iteration over trees and the second for iteration over nodes in a tree. For example, set to (1, 0) to parallelize only over nodes in a tree and (0, 1) to parallelize only over trees. For multi-level parallelism set the first value to a number less than the total number of processors and the second to 0. For example, if running with 8 processors, set njobs to (2, 0) to iterate over each tree with a group of 4 processors. Within each tree, each of the 4 processors in the group will work on a single node. If set to None, njobs will be set to (0, 1) if there are most trees than processors (tree parallel) and (1, 0) otherwise (node parallel). Default: None
dynamic (optional, tuples of bools) – Toggles task queue on/off for parallelism over trees (first value) and nodes within a tree (second). Set to a value False to divide iterations evenly among process groups. Set to True to allocate iterations with a task queue. If True, the number of processors available will be one fewer than the total as one will act as the task queue server. Yes, this can be set to (True, True). Try it. Default: (False, False)

Examples

>>> import ytree
>>> a = ytree.load("arbor/arbor.h5")
>>> a.add_analysis_field("test_field", default=-1, units="Msun")
>>> trees = list(a[:])
>>> for node in ytree.parallel_nodes(trees):
...     node["test_field"] = 2 * node["mass"] # some analysis