ytree.utilities.parallel.parallel_trees

ytree.utilities.parallel.parallel_trees(trees, save_every=None, filename=None, njobs=0, dynamic=False)[source]

Iterate over a list of trees in parallel.

Trees are divided up between the available processor groups. Analysis field values can then be assigned to halos within the tree. The trees will be saved either at the end of the loop or after a number of trees given by the save_every keyword are completed.

This uses the yt parallel_objects function, which is parallelized with MPI underneath and so is suitable for parallelism across compute nodes.

Parameters:
  • trees (list of TreeNode objects) – The trees to be iterated over in parallel.
  • save_every (optional, int or False) – Number of trees to be completed before results are saved. This is used to save intermediate results in case scripts need to be restarted. If None, save will only occur after iterating over all trees. If False, no saving will be done. Default: None
  • filename (optional, string) – The name of the new arbor to be saved. If None, the naming convention will follow the filename keyword of the save_arbor function. Default: None
  • njobs (optional, int) – The number of process groups for parallel iteration. Set to 0 to make the same number of process groups as available processors. Hence, each tree will be allocated to a single processor. Set to a number less than the total number of processors to create groups with multiple processors, which will allow for further parallelization within a tree. For example, running with 8 processors and setting njobs to 4 will result in 4 groups of 2 processors each. Default: 0
  • dynamic (optional, bool) – Set to False to divide iterations evenly among process groups. Set to True to allocate iterations with a task queue. If True, the number of processors available will be one fewer than the total as one will act as the task queue server. Default: False

Examples

>>> import ytree
>>> a = ytree.load("arbor/arbor.h5")
>>> a.add_analysis_field("test_field", default=-1, units="Msun")
>>> trees = list(a[:])
>>> for tree in ytree.parallel_trees(trees):
...     for node in tree["forest"]:
...         node["test_field"] = 2 * node["mass"] # some analysis