API¶
Nodes¶
Suite¶
- class pyflow.Suite(name, host=None, *args, **kwargs)¶
- __init__(name, host=None, *args, **kwargs)¶
Represents a collection of interrelated ecFlow tasks.
- Parameters
name (str) – Name of the suite to create.
host (Host) – The host to execute the suite on. If None, default ecFlow behaviour will be used.
json (dict) – Parsed JSON for creation of the children node(s).
workdir (str) – The working directory for the tasks, can be fixed or an ecFlow variable.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the suite is a shadow node created to satisfy an Extern, and should not be generated.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
repeat (RepeatDay) – An attribute that allows a node to be repeated infinitely.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the suite.
Example:
with pyflow.Suite('s', host=pyflow.LocalHost(), defstatus=pyflow.state.suspended, FOO='BAR') as s: pass
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- deploy_suite(target=<class 'pyflow.deployment.FileSystem'>, **options)¶
Deploys suite and its components.
- Parameters
target (Deployment) – Deployment target for the suite.
**options (dict) – Accept extra keyword arguments as deployment options.
- Returns
Deployment target object.
- Return type
Deployment
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow suite object.
- Returns
ecFlow suite object.
- Return type
ecflow.Suite
- find_node(subpath)¶
Returns node under provided relative path.
- Parameters
subpath (str) – Relative path of the node to search for.
- Returns
Found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- relative_path(node)¶
Returns relative path of the suite.
- Parameters
node (Node) – Unused.
- Returns
Relative path of the suite.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of modules.
- Returns
List of modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The anchor object.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property files_path¶
The files path of the node.
- Type
str
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property include_path¶
The include path of the node.
- Type
str
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
Task¶
- class pyflow.Task(name, autolimit=True, submit_arguments=None, exit_hook=None, clean_workdir=False, **kwargs)¶
- __init__(name, autolimit=True, submit_arguments=None, exit_hook=None, clean_workdir=False, **kwargs)¶
Describes what should be carried out as one executable unit within an ecFlow suite.
- Parameters
autolimit (bool) – Whether to automatically add the task to the executing hosts limit, if it has one.
submit_arguments (dict) – Parameters to encode into the script to make the scheduler happy.
exit_hook (str,list) – a script containing some commands to be called at exit time.
clean_workdir (bool) – Whether to ensure that the working directory is empty.+
script (str,list) – The script command or the list of script commands associated with the task.
json (dict) – Parsed JSON for creation of the children node(s).
host (Host) – The host to execute the task on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the task is a shadow node created to satisfy an Extern, and should not be generated.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
repeat (RepeatDay) – An attribute that allows a node to be repeated infinitely.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the task.
Example:
with pyflow.Task('t', script='echo "Hello, world!"', FOO='bar') as t: pass
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow task object.
- Returns
ecFlow task object.
- Return type
ecflow.Task
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_script()¶
Generates the complete script for the task.
- Returns
Complete script for the task.
- Return type
str
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of task modules.
- Returns
List of task modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the host or task modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property deploy_extension¶
The script file extension to be used during deployment of the task.
- Type
str
- property deploy_path¶
The deployment path of the current task, may be None.
- Type
str
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submit_arguments¶
The dictionary of submit arguments.
- Type
dict
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
Family¶
- class pyflow.Family(name, json=None, modules=None, purge_modules=False, extern=False, **kwargs)¶
- __init__(name, json=None, modules=None, purge_modules=False, extern=False, **kwargs)¶
Provides both visual and logical grouping of related families and tasks.
- Parameters
name (str) – The name of the family to create.
json (dict) – Parsed JSON for creating the children node(s).
host (Host) – The host to execute the family on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the family is a shadow node created to satisfy an Extern, and should not be generated.
workdir (string) – Working directory for tasks.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
repeat (RepeatDay) – An attribute that allows a node to be repeated infinitely.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the family.
Example:
with pyflow.Family('f', labels={'foo': 'bar'}) as f: pass
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow family object.
- Returns
ecFlow family object.
- Return type
ecflow.Family
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of modules.
- Returns
List of modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property manual_path¶
The deployment path of the current task, may be None.
- Type
str
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
AnchorFamily¶
- class pyflow.AnchorFamily(name, json=None, modules=None, purge_modules=False, extern=False, **kwargs)¶
- __init__(name, json=None, modules=None, purge_modules=False, extern=False, **kwargs)¶
Provides grouping of tasks that require encapsulation.
- Parameters
name (str) – Name of the anchor family to create.
json (dict) – Parsed JSON for creation of the children node(s).
host (Host) – The host to execute the anchor family on.
modules (tuple) – The list of modules to load.
purge_modules (bool) – Causes the generated script to include the code to purge all loaded modules at script runtime.
extern (bool) – Whether the anchor family is a shadow node created to satisfy an Extern, and should not be generated.
autocancel (Autocancel) – An attribute for automatic removal of the node which has completed.
completes (Complete) – An attribute for setting a condition for setting the node as complete depending on other tasks or families.
cron (Cron) – An attribute for setting a cron dependency of the node for the current day.
date (Date) – An attribute for setting a date dependency of the node.
day (Day) – An attribute for setting a day of the week dependency of the node.
defstatus (Defstatus) – An attribute for declaring the default status of the node.
families (Family) – An attribute for adding a child family on the node.
follow (Follow) – An attribute for setting a condition for running the node behind another repeated node which has completed.
inlimits (InLimit) – An attribute for grouping of tasks to which a limit should be applied.
labels (Label) – An attribute for a string value that can be set from a script.
limits (Limit) – An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
meters (Meter) – An attribute for a range of integer values that can be set from a script.
repeat (RepeatDay) – An attribute that allows a node to be repeated infinitely.
tasks (Task) – An attribute for adding a child task on the node.
time (Time) – An attribute for setting a time dependency of the node.
today (Today) – An attribute for setting a cron dependency of the node for the current day.
triggers (Trigger) – An attribute for setting a condition for running the node depending on other tasks or families.
variables (Variable) – An attribute for setting an ecFlow variable.
zombies (Zombies) – An attribute that defines how a zombie should be handled in an automated fashion.
events (Event) – An attribute for declaring an action that a task can trigger while it is running.
**kwargs (str) – Accept extra keyword arguments as variables to be set on the anchor family.
Example:
with pyflow.AnchorFamily('af', labels={'foo': 'bar'}) as af: pass
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow family object.
- Returns
ecFlow family object.
- Return type
ecflow.Family
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of modules.
- Returns
List of modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The anchor object.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property files_path¶
The files path of the node.
- Type
str
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property include_path¶
The include path of the node.
- Type
str
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property manual_path¶
The deployment path of the current task, may be None.
- Type
str
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
Attributes¶
Variable¶
- class pyflow.Variable(name, value)¶
An attribute for setting an ecFlow variable.
- Parameters
name (str) – The name of the variable.
value (str) – The value of the variable.
Example:
Variable('FOO', 'foo_value')
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.Edit(**kwargs)¶
An attribute for setting multiple ecFlow variables.
- Parameters
**kwargs (dict) – Accept keyword arguments as variables to be set.
Example:
pyflow.Edit(FOO='foo_value', BAR='bar_value')
Trigger¶
- class pyflow.Trigger(value, *args)¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Parameters
value (expression) – Expression to evaluate for running the node.
*args (tuple) – Accept extra positional arguments for expressions provided as a JSON value.
Example:
pyflow.Trigger(t1 & t2)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Event¶
- class pyflow.Event(name)¶
An attribute for declaring an action that a node can trigger while it is running.
- Parameters
name (str) – The name of the event.
Example:
pyflow.Event('a')
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Complete¶
- class pyflow.Complete(value)¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
Note
Complete expression evaluation takes precedence over the trigger.
- Parameters
value (expression) – Expression to evaluate for setting the node as complete.
Example:
pyflow.Complete(t1 & t2)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Late¶
- class pyflow.Late(value)¶
An attribute for a flag for notifying if task does not run as expected.
- Parameters
value (str) – Expression to evaluate for setting the late flag.
Example:
pyflow.Late('-c +00:01') # set late flag if task takes longer than a minute
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Label¶
- class pyflow.attributes.Label(name, value)¶
An attribute for a string value that can be set from a script.
- Parameters
name (str) – The name of the label.
value (str) – The initial value of the label.
Example:
pyflow.attributes.Label('foo', 'bar')
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Meter¶
- class pyflow.Meter(name, min, max=None, threshold=None)¶
An attribute for a range of integer values that can be set from a script.
- Parameters
name (str) – The name of the meter.
min (int,tuple,list) – Minimum value of the meter. Alternatively, a tuple or list containing minimum, maximum and threshold value of the meter.
max (int) – Maximum value of the meter.
threshold (int) – Threshold value of the meter.
Example:
pyflow.Meter('progress', 1, 100, 90)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Defstatus¶
- class pyflow.Defstatus(value)¶
An attribute for declaring the default status of the node.
- Parameters
value (State) – Default state to set for the node.
Example:
pyflow.Defstatus(pyflow.state.suspended)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Time Dependencies¶
- class pyflow.Time(value)¶
An attribute for setting a time dependency of the node.
- Parameters
value (str) – Either a cron-like expression (m h d M D), cron-like time series expression (start(hh:mm) end (hh:mm) increment(hh:mm)) or an absolute or relative time stamp (hh:mm).
Example:
pyflow.Time("23:00") # at next 23:00 pyflow.Time("0 10-20 * * *") # every hour between 10 am and 8 pm
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.Cron(value, days_of_week=None, last_week_days_of_the_month=None, days_of_month=None, last_day_of_the_month=None, months=None)¶
An attribute for setting a cron dependency of the node.
- Parameters
value (str) – Either a cron-like expression (m h d M D), cron-like time series expression (start(hh:mm) end (hh:mm) increment(hh:mm)) or an absolute or relative time stamp (hh:mm).
days_of_week (list) – The list of the days of the week when the task should run, with 0 being Sunday and 6 Saturday.
last_week_days_of_the_month (list) – The list of the last days of the week of the month when the task should run, with 0 being Sunday and 6 Saturday.
days_of_month (list) – The list of the days of the month when the task should run.
last_day_of_the_month (bool) – Whether the task should run at the last day of the month.
months (list) – The list of the months when the task should run.
Example:
pyflow.Cron("0 23 * * *") # every day at 11 pm pyflow.Cron("0 8-12 * * *") # every hour between 8 and 12 am pyflow.Cron("0 11 * * SUN,TUE") # every Sunday and Tuesday at 11 am pyflow.Cron("0 2 1,15 * *") # every 1st and 15th of each month at 2 am pyflow.Cron("0 14 1 1 *") # every first of January at 2 pm pyflow.Cron("23:00", last_week_days_of_the_month=[5]) # every *last* Friday of month at 11 pm pyflow.Cron("23:00", days_of_month=[1], last_day_of_the_month=True) # every first and last of month at 11 pm
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.Crons(value)¶
An attribute for setting a cron time series dependency of the node.
- Parameters
value (str) – A cron-like time series expression (start(hh:mm) end(hh:mm) increment(hh:mm)) for the node dependency.
Example:
pyflow.Crons("00:00 23:59 00:05") # every 5 minutes during the day
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.Date(value, value2=None, value3=None)¶
An attribute for setting a date dependency of the node.
Note
All string values support wildcards (*).
- Parameters
value (str,datetime) – Either the day of the month (d), the complete date (d.M.Y) or the datetime object of the date dependency.
value2 (str) – Month of the date dependency (M).
value3 (str) – Year of the date dependency (Y).
Example:
pyflow.Date("31.12.2012") # the 31st of December 2012 pyflow.Date("01.*.*") # every first of the month pyflow.Date("*.10.*") # every day in October pyflow.Date("1.*.2008") # every first of the month, but only in 2008
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.attributes.Day(value)¶
An attribute for setting a day of the week dependency of the node.
- Parameters
value (str) – Day of the week of the dependency.
Example:
pyflow.attributes.Day('monday') # every monday
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.attributes.Today(value)¶
An attribute for setting a cron dependency of the node for the current day.
- Parameters
value (str) – A cron-like expression (m h d M D) for the node dependency, limited to current day.
Example:
pyflow.attributes.Today("0 12 * * *") # today at 12 pm
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Repeat¶
- class pyflow.RepeatDate(name, start, end, increment=1)¶
An attribute that allows a node to be repeated by a date value.
- Parameters
name (str) – The name of the repeat attribute.
start (datetime) – The start date of the repeat attribute.
end (datetime) – The end date of the repeat attribute.
increment (int) – The increment used to update the date.
Example:
pyflow.RepeatDate('REPEAT_DATE', datetime.date(year=2019, month=1, day=1), datetime.date(year=2019, month=12, day=31))
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property day¶
The day of the repeat date.
- Type
int
- property day_of_week¶
The day of the week of the repeat date.
- Type
int
- property fullname¶
The relative path of the attribute.
- Type
str
- property julian¶
The Julian date of the repeat date.
- Type
int
- property month¶
The month of the repeat date.
- Type
int
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- property year¶
The year of the repeat date.
- Type
int
- class pyflow.RepeatInteger(name, start, end, increment=1)¶
An attribute that allows a node to be repeated by an integer range.
- Parameters
name (str) – The name of the repeat attribute.
start (int) – The start integer value of the repeat attribute.
end (datetime) – The end integer value of the repeat attribute.
increment (int) – The step amount used to update the integer.
Example:
pyflow.RepeatInteger("REPEAT_INTEGER", 1, 5, 1)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.RepeatEnumerated(name, value)¶
An attribute that allows a node to be repeated by an enumerated list.
- Parameters
name (str) – The name of the repeat attribute.
list (tuple) – The list of enumerations for the repeat attribute.
Example:
pyflow.RepeatEnumerated("REPEAT_STRING", ["a", "b", "c", "d", "e"])
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- property values¶
The list of enumerated values.
- Type
list
- class pyflow.attributes.RepeatString(name, value)¶
An attribute that allows a node to be repeated by a string value.
- Parameters
name (str) – The name of the repeat attribute.
list (tuple) – The list of string values for the repeat attribute.
Example:
pyflow.RepeatString("COLOR", ["red", "green", "blue"])
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- property values¶
The list of repeat string values.
- Type
list
- class pyflow.attributes.RepeatDay(value)¶
An attribute that allows a node to be repeated infinitely.
- Parameters
value (int) – The repeat step.
Example:
pyflow.attributes.RepeatDay(1)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Limit¶
- class pyflow.Limit(name, value)¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Parameters
name (str) – The name of the limit.
value (int) – The maximum number of tasks.
Example:
pyflow.Limit('l', 3)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.InLimit(value)¶
An attribute for grouping of nodes to which a limit should be applied.
- Parameters
value (str,Limit_) – The name of the limit or a limit object.
Example:
l = pyflow.Limit('l', 3) pyflow.InLimit(l)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
- class pyflow.Inlimit(value)¶
An attribute for grouping of tasks to which a limit should be applied.
- Parameters
value (str,Limit_) – The name of the limit or a limit object.
Example:
l = pyflow.Limit('l', 3) pyflow.Inlimit(l)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Manual¶
- class pyflow.attributes.Manual(value)¶
An attribute for setting help text of the node.
- Parameters
value (str) – The help text or list of help texts to include in the node.
Example:
pyflow.attributes.Manual('This is a multi-line manual\nwhich can contain instructions')
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Autocancel¶
- class pyflow.attributes.Autocancel(value, value2=None)¶
An attribute for automatic removal of the node which has completed.
- Parameters
value (bool,int,str,list,tuple) – The time slot arguments for autocancel attribute. If True the node will be removed as soon as it has completed. If a list or tuple, items will be used for hour and minute when the node will be removed.
value2 (str) – The optional minute argument in case first argument contained only hour.
Example:
pyflow.attributes.Autocancel(True) # delete node immediately after completion pyflow.attributes.Autocancel('+01:30') # delete node 1 hour and 30 minutes after completion pyflow.attributes.Autocancel((1, 30)) # delete node at 1:30 am after completion pyflow.attributes.Autocancel(1, 30) # delete node at 1:30 am after completion pyflow.attributes.Autocancel(3) # delete node 3 days after completion
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Follow¶
- class pyflow.attributes.Follow(value)¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Parameters
value (RepeatDate) – The repeat date attribute of the followed node.
Example:
pyflow.attributes.Follow(pyflow.RepeatDate('REPEAT_DATE', datetime.date(year=2019, month=1, day=1), datetime.date(year=2019, month=12, day=31)))
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
Zombies¶
- class pyflow.attributes.Zombies(value)¶
An attribute that defines how a zombie should be handled in an automated fashion.
- Parameters
value (str) – A custom way a zombie should be handled when encountered.
Example:
pyflow.attributes.Zombies(None)
- relative_path(other)¶
Returns relative path of the attribute.
- Parameters
other (str) – Relative path of the parent node.
- Returns
Relative path of the attribute.
- Return type
str
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property fullname¶
The relative path of the attribute.
- Type
str
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property value¶
The value of the node.
- Type
str
External¶
- class pyflow.Extern(path)¶
Maps an external family, i.e. a family that is not built from the same repository.
- Parameters
path (str) – Path of the external family.
- Returns
An object that corresponds to an external family.
- Return type
Example:
pyflow.Extern('/f/g/h/i')
- class pyflow.ExternNode(path, tail_cls=<class 'pyflow.nodes.Family'>)¶
Maps an external node, i.e. a node that is not built from the same repository.
- Parameters
path (str) – Path of the external node.
tail_cls (class) – Object class of the external node.
- Returns
An object that corresponds to an external node.
- Return type
Node
Example:
pyflow.ExternNode('/a/b/c/d')
- class pyflow.ExternTask(path)¶
Maps an external task, i.e. a task that is not built from the same repository.
- Parameters
path (str) – Path of the external task.
- Returns
An object that corresponds to an external task.
- Return type
Example:
pyflow.ExternTask('/a/b/c/d')
- class pyflow.ExternFamily(path)¶
Maps an external family, i.e. a family that is not built from the same repository.
- Parameters
path (str) – Path of the external family.
- Returns
An object that corresponds to an external family.
- Return type
Example:
pyflow.ExternFamily('/f/g/h/i')
- class pyflow.ExternEvent(path)¶
Maps an external event, i.e. a event that is not built from the same repository.
- Parameters
path (str) – Path of the external event.
- Returns
An object that corresponds to an external event.
- Return type
Example:
pyflow.ExternEvent('/e/f/g/h:ev')
- class pyflow.ExternMeter(path)¶
Maps an external meter, i.e. a meter that is not built from the same repository.
- Parameters
path (str) – Path of the external meter.
- Returns
An object that corresponds to an external event.
- Return type
Example:
pyflow.ExternMeter('/g/h/i/j:mt')
- class pyflow.ExternYMD(path)¶
Maps an external repeat date, i.e. a repeat date that is not built from the same repository.
- Parameters
path (str) – Path of the external repeat date.
- Returns
An object that corresponds to an external repeat date.
- Return type
Example:
pyflow.ExternYMD('/a/b/c/d:YMD')
Deployment¶
- class pyflow.Notebook(suite, **options)¶
A dummy deployment target for Jupyter Notebooks, skips creation of directories and files and dumps fresh ecFlow definitions as cell output.
- Parameters
suite (Suite) – The suite object to deploy.
path (str) – The path to Git repository.
Example:
s = pyflow.Suite('suite') s.deploy_suite(target=pyflow.Notebook)
- copy(source, target)¶
Deploys the task script to target path.
- Parameters
source (str,bytes,list) – The task script to deploy.
target (str) – The deployment path.
- deploy_headers()¶
Installs all required header files.
- deploy_manual(deploy_path, full_script)¶
Deploys the manual to target path.
- Parameters
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the manual.
- deploy_task(deploy_path, full_script, required_includes)¶
Deploys the task to target path.
- Parameters
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the task.
required_includes (list) – The list of required header files.
- files_install_path()¶
str: Returns the files install path.
- save(source, target)¶
Deploys the task script to target path.
- Parameters
source (str) – The path to task script.
target (str) – The deployment path.
- class pyflow.DeployGitRepo(suite, path=None)¶
A deployment target for Git repositories, clears target repository and dumps fresh ecFlow definitions.
- Parameters
suite (Suite) – The suite object to deploy.
path (str) – The path to Git repository.
Example:
s = pf.Suite('suite') pyflow.DeployGitRepo(s, path='/path/to/git')
- check(target)¶
Check if the target should be deployed. Returns False if target has already been deployed, True otherwise.
- Parameters
target (str) – The target path for deployment.
- Returns
True if the target path is valid for deployment, False otherwise.
- Return type
bool
- copy(source, target)¶
Deploys the task script to target path. This method contains functionality needed for all deployments. Should be called in super() by all derived classes.
- Parameters
source (str) – The path to task script.
target (str) – The deployment path.
- deploy_headers()¶
Installs all required header files.
- deploy_manual(deploy_path, full_script)¶
Deploys the manual to target path.
- Parameters
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the manual.
- deploy_task(deploy_path, full_script, required_includes)¶
Deploys the task to target path.
- Parameters
deploy_path (str) – The deployment path.
full_script (str,list) – The full script of the task.
required_includes (list) – The list of required header files.
- files_install_path()¶
str: Returns the files install path.
- patch_path(path)¶
Patches the path so it includes the complete ecFlow path.
- Parameters
path (str) – The path to patch.
- Returns
The patched path.
- Return type
str
- save(source, target)¶
Deploys the task script to target path. This method contains functionality needed for all deployments. Should be called in super() by all derived classes.
- Parameters
source (str,bytes,list) – The task script to deploy.
target (str) – The deployment path.
Hosts¶
- class pyflow.Host(name, hostname=None, scratch_directory=None, log_directory=None, resources_directory=None, limit=None, extra_paths=None, extra_variables=None, environment_variables=None, module_source=None, modules=None, purge_modules=False, label_host=True, user='docs', ecflow_path=None, server_ecfvars=False)¶
An abstract base class for host-related functionality.
- Parameters
name (str) – The name of the host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths, or for login details. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
class MyHost(Host): pass
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- host_preamble(exit_hook=None)¶
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host.
- Type
dict
- property host_postamble¶
The host-specific cleanup script, always empty.
- Type
list
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
- class pyflow.NullHost(**kwargs)¶
A dummy host object invisible to ecFlow, but still throws exceptions if pyflow attempts to create tasks inside it.
- Parameters
hostname (str) – The hostname of the host, otherwise null will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
with pyflow.Suite('s', host=pf.NullHost()): pass
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Skips setting an exec_host label on nodes where this host is freshly set.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- host_preamble(exit_hook=None)¶
The host-specific implementation of preamble script, always raises an error.
- Raises
AttributeError – Constructing tasks under NullHost is invalid.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host, always empty.
- Type
dict
- property host_postamble¶
The host-specific implementation of cleanup script, always raises an error.
- Raises
AttributeError – Constructing tasks under NullHost is invalid.
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
- class pyflow.LocalHost(name='localhost', **kwargs)¶
A host object that executes scripts directly on the ecFlow server.
- Parameters
name (str) – The name of the host, localhost by default.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, the limit is 20 tasks.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
user (str) – The user running the script. May be used to determine paths. Defaults to current user.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
pyflow.LocalHost(purge_modules=True, modules=['mod1/123', '-mod2/321', 'mod3/33'])
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- copy_file_to(source_file, target_file)¶
Returns the script for copying a file to host.
- Parameters
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns
The script for copying a file to host.
- Return type
str
- host_preamble(exit_hook=None)¶
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- run_simple_command(cmd)¶
Returns the command to run a simple command on this host.
- Parameters
cmd (str) – A simple command to run.
- Returns
The command to run a simple command.
- Return type
str
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host.
- Type
dict
- property host_postamble¶
The host-specific cleanup script, always empty.
- Type
list
- property job_cmd¶
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type
str
- property kill_cmd¶
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type
str
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
- class pyflow.SSHHost(name, user=None, indirect_host=None, indirect_user=None, **kwargs)¶
A host object that executes scripts on the ecFlow server via SSH protocol.
- Parameters
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, the limit is 20 tasks.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
pyflow.SSHHost('dhs9999', user='max', scratch_directory='/data/a_mounted_filesystem/tmp')
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- copy_file_to(source_file, target_file)¶
Returns the script for copying a file to host.
- Parameters
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns
The script for copying a file to host.
- Return type
str
- host_preamble(exit_hook=None)¶
list: The host-specific implementation of preamble script, always empty.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- run_simple_command(cmd)¶
Returns the command to run a simple command on this host.
- Parameters
cmd (str) – A simple command to run.
- Returns
The command to run a simple command.
- Return type
str
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host.
- Type
dict
- property host_postamble¶
The host-specific cleanup script, always empty.
- Type
list
- property job_cmd¶
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type
str
- property kill_cmd¶
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type
str
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
- class pyflow.SLURMHost(name, **kwargs)¶
A host object that executes scripts on the ecFlow server via Slurm job scheduling system.
- Parameters
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
with pyflow.Suite('s', host=pyflow.SLURMHost('slurm_a')): pass
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- copy_file_to(source_file, target_file)¶
Returns the script for copying a file to host.
- Parameters
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns
The script for copying a file to host.
- Return type
str
- host_preamble(exit_hook=None)¶
list: The host-specific implementation of preamble script.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- run_simple_command(cmd)¶
Returns the command to run a simple command on this host.
- Parameters
cmd (str) – A simple command to run.
- Returns
The command to run a simple command.
- Return type
str
- script_submit_arguments(submit_arguments)¶
Returns list of script submit arguments.
- Parameters
submit_arguments (dict) – A dictionary of script submit arguments.
- Returns
The list of script submit arguments.
- Return type
list
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host.
- Type
dict
- property host_postamble¶
The host-specific cleanup script.
- Type
list
- property job_cmd¶
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type
str
- property kill_cmd¶
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type
str
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
- class pyflow.PBSHost(name, **kwargs)¶
A host object that executes scripts on the ecFlow server via batch server.
- Parameters
name (str) – The name of the host.
user (str) – The user to use for SSH commands to the host. Defaults to current user.
indirect_host (str) – The name of the host to use indirectly. May be in user@server format.
indirect_user (str) – The user to use for SSH commands on the indirect host.
hostname (str) – The hostname of the host, otherwise name will be used.
scratch_directory (str) – The path in which tasks will be run, unless otherwise specified.
log_directory (str) – The directory to use for script output. Normally ECF_HOME, but may need to be changed on systems with scheduling systems to make the output visible to the ecFlow server.
resources_directory (str) – The directory to use for suite resources. By default, scratch_directory is used.
limit (int) – The number of tasks that can run on this node simultaneously. By default, there is no limit.
extra_paths (list) – The list of paths that are added to PATH on the host.
extra_variables (dict) – The dictionary of additional ecFlow variables that are set on the host.
environment_variables (dict) – The dictionary of additional environment variables that are included in scripts.
module_source (str) – The shell script to source to initialise the module system.
modules (list) – The list of environment modules to load via module load command.
purge_models (bool) – Whether to run the module purge command, before loading any environment modules.
label_host (bool) – Whether to create an exec_host label on nodes where this host is freshly set.
ecflow_path (str) – The directory containing the ecflow_client executable.
server_ecfvars (bool) – If true, don’t define ECF_JOB_CMD, ECF_KILL_CMD, ECF_STATUS_CMD and ECF_OUT variables and use defaults from server
Example:
with pyflow.Suite('s', host=pyflow.PBSHost('host_a')): pass
- add_to_limits(task)¶
Adds a task to be contained within a hosts assigned limit.
- Parameters
task (Task) – The task to contain within the assigned limit.
- build_label()¶
Sets an exec_host label on nodes where this host is freshly set, if configured.
- build_limits(replace=False)¶
Sets the number of tasks that can run on this host simultaneously, if configured.
- Parameters
replace (bool) – Whether to replace the currently computed limit.
- Raises
DuplicateNodeError –
- copy_file_to(source_file, target_file)¶
Returns the script for copying a file to host.
- Parameters
source_file (str) – The source file to copy from.
target_file (str) – The target file to copy to.
- Returns
The script for copying a file to host.
- Return type
str
- host_preamble(exit_hook=None)¶
list: The host-specific implementation of preamble script.
- job_preamble(exit_hook=None)¶
list: The host-specific preamble for jobs.
- preamble(exit_hook=None)¶
list: The host-specific preamble script for jobs.
- preamble_error_function(ecflowpath, exit_hook=None)¶
Returns the host-specific error function for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
additional_commands (tuple) – The list of additional commands to include in the function.
- Returns
The error function script.
- Return type
str
- preamble_init(ecflowpath)¶
Returns the host-specific preamble initialisation section for jobs.
- Parameters
ecflowpath (str) – The path to ecFlow.
- Returns
The preamble initialisation script.
- Return type
str
- run_simple_command(cmd)¶
Returns the command to run a simple command on this host.
- Parameters
cmd (str) – A simple command to run.
- Returns
The command to run a simple command.
- Return type
str
- script_submit_arguments(submit_arguments)¶
Returns list of script submit arguments.
- Parameters
submit_arguments (dict) – A dictionary of script submit arguments.
- Returns
The list of script submit arguments.
- Return type
list
- property check_cmd¶
The ecflow check command.
- Type
str
- property ecflow_variables¶
The variables that must be set on relevant nodes to run on this host.
- Type
dict
- property host_postamble¶
The host-specific cleanup script.
- Type
list
- property job_cmd¶
The ecFlow submission command, sets the ECF_JOB_CMD variable.
- Type
str
- property kill_cmd¶
The ecflow kill command, sets the ECF_KILL_CMD variable.
- Type
str
- property limit¶
The number of tasks that can run on this host simultaneously.
- Type
int
- property status_cmd¶
The ecflow status command.
- Type
str
Scripts¶
- class pyflow.Script(value=None)¶
A group of commands that define the main work that is to be carried out for a task.
- Parameters
value (str,list) – The script command or the list of script commands.
Example:
with pyflow.Task('t', script=pyflow.Script('echo "Hello, world!"')): pass
- add_required_exportables(*args)¶
Defines a set of required exportable variables.
- Parameters
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)¶
Defines a required exportable attribute.
- Parameters
exportable (str) – The name of the exportable variables.
- static generate_list_scripts(values)¶
Generates a script of all passed scripts.
- Parameters
values (list) – The list of Script objects.
- Returns
The list of script commands.
- Return type
list
- generate_stub()¶
Returns complete script by combining the fragments.
- Returns
The list of script commands.
- Return type
list
- required_exportables()¶
Returns the set of required exportable variables.
- Returns
The set of required exportable variables.
- Return type
set
- property value¶
The string value with all script commands.
- Type
str
- class pyflow.PythonScript(value=None, python=3)¶
A script written in Python language.
- Parameters
value (str,list) – The Python command or the list of Python commands.
python (int) – The Python major version.
Example:
pyflow.PythonScript('w1 = "Hello"\nw2 = "world"\nprint(f"{w1}, {w2}!")', 3)
- add_required_exportables(*args)¶
Defines a set of required exportable variables.
- Parameters
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)¶
Defines a required exportable attribute.
- Parameters
exportable (str) – The name of the exportable variables.
- static generate_list_scripts(values)¶
Generates a script of all passed scripts.
- Parameters
values (list) – The list of Script objects.
- Returns
The list of script commands.
- Return type
list
- generate_stub()¶
Returns complete script by combining the fragments.
- Returns
The list of script commands.
- Return type
list
- required_exportables()¶
Returns the set of required exportable variables.
- Returns
The set of required exportable variables.
- Return type
set
- property value¶
The string value with all script commands.
- Type
str
- class pyflow.FileScript(filename)¶
A script with a provided filename, which is read at the time of suite generation.
- Parameters
filename (str) – The filename of the script to read from.
Example:
pyflow.FileScript('/path/to/script')
- add_required_exportables(*args)¶
Defines a set of required exportable variables.
- Parameters
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)¶
Defines a required exportable attribute.
- Parameters
exportable (str) – The name of the exportable variables.
- generate()¶
Returns the script commands from the provided file.
- Returns
The list of script commands.
- Return type
list
- static generate_list_scripts(values)¶
Generates a script of all passed scripts.
- Parameters
values (list) – The list of Script objects.
- Returns
The list of script commands.
- Return type
list
- generate_stub()¶
Returns complete script by combining the fragments.
- Returns
The list of script commands.
- Return type
list
- required_exportables()¶
Returns the set of required exportable variables.
- Returns
The set of required exportable variables.
- Return type
set
- property value¶
The string value with all script commands.
- Type
str
- class pyflow.TemplateScript(*args, **kwargs)¶
A script template with Jinja syntax support.
- Parameters
value (str,list) – The script command or the list of script commands.
**kwargs (dict) – Accept keyword arguments as values for template variables.
Example:
pyflow.TemplateScript('echo "{{ w1 }}, {{ w2 }}!"', w1='Hello', w2='world')
- add_parameters(**kwargs)¶
Defines extra parameters for template variables.
- Parameters
**kwargs (dict) – Accept keyword arguments as values for template variables.
- add_required_exportables(*args)¶
Defines a set of required exportable variables.
- Parameters
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)¶
Defines a required exportable attribute.
- Parameters
exportable (str) – The name of the exportable variables.
- static generate_list_scripts(values)¶
Generates a script of all passed scripts.
- Parameters
values (list) – The list of Script objects.
- Returns
The list of script commands.
- Return type
list
- generate_stub()¶
Returns complete script by combining the fragments.
- Returns
The list of script commands.
- Return type
list
- required_exportables()¶
Returns the set of required exportable variables.
- Returns
The set of required exportable variables.
- Return type
set
- property value¶
The string value with all script commands.
- Type
str
- class pyflow.TemplateFileScript(*args, **kwargs)¶
A script template with a provided filename, which is read at the time of suite generation.
- Parameters
filename (str) – The filename of the script to read from.
**kwargs (dict) – Accept keyword arguments as values for template variables.
Example:
pyflow.TemplateFileScript('/path/to/script', foo='bar')
- add_parameters(**kwargs)¶
Defines extra parameters for template variables.
- Parameters
**kwargs (dict) – Accept keyword arguments as values for template variables.
- add_required_exportables(*args)¶
Defines a set of required exportable variables.
- Parameters
*args (tuple) – Accept positional arguments as names of exportable variables.
- define_environment_variable(k, v, export=True)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
export (bool) – Whether to export the environment variable, must be True.
- environment_variable(k, v)¶
Defines an environment variable for the script.
- Parameters
k (str) – The name of the environment variable.
v (str) – The value of the environment variable.
- force_exported(exportable)¶
Defines a required exportable attribute.
- Parameters
exportable (str) – The name of the exportable variables.
- generate()¶
Returns the script commands from the provided file.
- Returns
The list of script commands.
- Return type
list
- static generate_list_scripts(values)¶
Generates a script of all passed scripts.
- Parameters
values (list) – The list of Script objects.
- Returns
The list of script commands.
- Return type
list
- generate_stub()¶
Returns complete script by combining the fragments.
- Returns
The list of script commands.
- Return type
list
- required_exportables()¶
Returns the set of required exportable variables.
- Returns
The set of required exportable variables.
- Return type
set
- property value¶
The string value with all script commands.
- Type
str
Resources¶
- class pyflow.DataResource(name, hosts, source_data)¶
Provides a data resource to be deployed at suite generation time.
- Parameters
name (str) – The name of the resource.
hosts (Host,list) – The host or list of hosts to deploy the resource to.
source_data – The resource data.
Example:
pyflow.DataResource('data1', [pyflow.LocalHost(resources_directory='/path/to/data')], "this is some data".encode('utf-8'))
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- build_script()¶
Returns the installer script for the data resource.
- Returns
The list of installer commands.
- Return type
list
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow task object.
- Returns
ecFlow task object.
- Return type
ecflow.Task
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_script()¶
Generates the complete script for the task.
- Returns
Complete script for the task.
- Return type
str
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- install_file_stub(target)¶
Installs any data associated with the resource object that is going to be deployed from the ecFlow server.
- Parameters
target (Deployment) – The target deployment where the resource data should be installed.
- location()¶
Returns the path of the resource.
- Returns
The path of the resource.
- Return type
str
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- md5()¶
Returns MD5 checksum of the resource data.
- Returns
The MD5 checksum of the resource data.
- Return type
str
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- save_data(target, filename)¶
Deploys the data for the resource at suite generation time.
- Parameters
target (Deployment) – The deployment target.
filename (str) – The filename for the resource data.
- task_modules()¶
Returns list of task modules.
- Returns
List of task modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the host or task modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property deploy_extension¶
The script file extension to be used during deployment of the task.
- Type
str
- property deploy_path¶
The deployment path of the current task, may be None.
- Type
str
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submit_arguments¶
The dictionary of submit arguments.
- Type
dict
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
- class pyflow.FileResource(name, hosts, source_file)¶
Provides a file resource to be deployed at suite generation time.
- Parameters
name (str) – The name of the resource.
hosts (list,Host_) – The list of hosts to deploy the resource to.
source_file (str) – The filename of the resource.
Example:
pyflow.FileResource('data2', [pyflow.LocalHost(resources_directory='/path/to/data')], 'path/to/data.dat')
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- build_script()¶
Returns the installer script for the data resource.
- Returns
The list of installer commands.
- Return type
list
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- data()¶
Returns the resource data from the provided file.
- Returns
The resource data.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow task object.
- Returns
ecFlow task object.
- Return type
ecflow.Task
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_script()¶
Generates the complete script for the task.
- Returns
Complete script for the task.
- Return type
str
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- install_file_stub(target)¶
Installs any data associated with the resource object that is going to be deployed from the ecFlow server.
- Parameters
target (Deployment) – The target deployment where the resource data should be installed.
- location()¶
Returns the path of the resource.
- Returns
The path of the resource.
- Return type
str
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- md5()¶
Returns MD5 checksum of the resource data.
- Returns
The MD5 checksum of the resource data.
- Return type
str
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- save_data(target, filename)¶
Deploys the data for the resource at suite generation time.
- Parameters
target (Deployment) – The deployment target.
filename (str) – The filename for the resource data.
- task_modules()¶
Returns list of task modules.
- Returns
List of task modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the host or task modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property deploy_extension¶
The script file extension to be used during deployment of the task.
- Type
str
- property deploy_path¶
The deployment path of the current task, may be None.
- Type
str
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submit_arguments¶
The dictionary of submit arguments.
- Type
dict
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
- class pyflow.WebResource(name, hosts, url, md5=None)¶
Provides a web resource to be deployed at suite generation time.
- Parameters
name (str) – The name of the resource.
hosts (list,Host_) – The list of hosts to deploy the resource to.
url (str) – The URL of the resource.
md5 (str) – The MD5 checksum of the resource.
Example:
pyflow.WebResource('data3', [pyflow.LocalHost(resources_directory='/path/to/data')], 'https://example.com/data', md5='0123456789abcdef')
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- build_script()¶
Returns the installer script for the data resource.
- Returns
The list of installer commands.
- Return type
list
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow task object.
- Returns
ecFlow task object.
- Return type
ecflow.Task
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_script()¶
Generates the complete script for the task.
- Returns
Complete script for the task.
- Return type
str
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- get_resource(filename)¶
Returns script commands to retrieve the resource to deploy at runtime.
- Parameters
filename (str) – The filename for the resource data.
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- install_file_stub(target)¶
Installs any data associated with the resource object that is going to be deployed from the ecFlow server.
- Parameters
target (Deployment) – The target deployment where the resource data should be installed.
- location()¶
Returns the path of the resource.
- Returns
The path of the resource.
- Return type
str
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- md5()¶
Returns MD5 checksum of the resource data. If not provided at construction time, resource will be downloaded and hashed.
- Returns
The MD5 checksum of the resource data.
- Return type
str
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of task modules.
- Returns
List of task modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the host or task modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property deploy_extension¶
The script file extension to be used during deployment of the task.
- Type
str
- property deploy_path¶
The deployment path of the current task, may be None.
- Type
str
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submit_arguments¶
The dictionary of submit arguments.
- Type
dict
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
- class pyflow.Resources(name=None, host=None)¶
Provides both visual and logical grouping of related resources.
Note
Just a wrapper (for now) on Family, used for consistency and elegance.
- Parameters
name (str) – The name of the resource group to create.
host (Host) – The host to execute the resource group on.
Example:
with pyflow.Resources(host=pf.LocalHost()): pass
- add_node(node)¶
Adds a child to current node.
- Parameters
node (Node) – The child node to add.
- Returns
Added child node.
- Return type
Node
- append_node(node)¶
Appends a child to current node.
- Parameters
node (Node) – The child node to append.
- Returns
The parent node.
- Return type
Node
- check_definition()¶
Checks ecFlow definitions of the node.
- Raises
*RuntimeError* – ecFlow definitions failed checks.
- clear_type(cls)¶
Removes child nodes of provided type.
- Parameters
cls (class) – The node type class name.
- draw_graph(view=True)¶
Draws the DOT graph.
- Parameters
view (bool) – Unused.
- Returns
DOT graph.
- Return type
Dot
- draw_tree()¶
Draws node tree as a DOT graph.
- Returns
The node DOT graph.
- Return type
Dot
- ecflow_definition()¶
Returns node definition.
- Returns
The node definition.
- Return type
ecflow.Defs
- ecflow_object()¶
Returns the corresponding ecFlow family object.
- Returns
ecFlow family object.
- Return type
ecflow.Family
- find_node(subpath)¶
Returns node under provided path.
- Parameters
subpath (str) – Path of the node to search for.
- Returns
The found node object.
- Return type
Node
- generate_node()¶
Generates node definition.
- Returns
Generated ecFlow node object.
- Return type
Node
- generate_stub(scripts)¶
Returns complete script by combining the fragments.
- Parameters
scripts (tuple) – List of script fragments.
- Returns
Complete script.
- Return type
str
- has_variable(name)¶
Signals if the current node has a variable defined.
- Parameters
name (str) – Name of the variable to search for.
- Returns
Whether the current node has the variable defined or not.
- Return type
bool
- lookup_variable(name)¶
Looks up value of the variable in current or parent node.
- Parameters
name (str) – Name of the variable to look up for.
- Returns
Variable value, if found.
- Return type
str
- lookup_variable_value(name, default=None)¶
Looks up value of the variable in current or parent node, with fallback on the provided default value.
- Parameters
name (str) – Name of the variable to look up for.
default (str) – Default valueName of the variable to look up for.
- Returns
Variable value, if found. Otherwise provided default value.
- Return type
str
- make_expression()¶
Generates node expression.
- Returns
Whether the node is complete or not.
- Return type
bool
- relative_path(node)¶
Returns relative path of the node.
- Returns
Relative path of the node.
- Return type
str
- remove_node(node)¶
Removes specific child node.
- Parameters
node (Node) – The child node to remove.
- replace_on_server(host, port=None)¶
Replaces node on the target host.
- Parameters
host (Host) – Target host.
port (str) – Port number of the target host.
- task_modules()¶
Returns list of modules.
- Returns
List of modules.
- Return type
list
- task_purge_modules()¶
Causes the generated script to include the code to purge all loaded modules at script runtime.
- Returns
Whether the modules have been purged or not.
- Return type
bool
- property aborted¶
The node aborted status.
- Type
bool
- property active¶
The node active status.
- Type
bool
- property all_exportables¶
The dictionary of all exportable attributes in the current or parent node.
- Type
dict
- property all_variables¶
The dictionary of all variables in the current or parent node.
- Type
dict
- property anchor¶
The current anchor (either Suite or AnchorFamily object) containing the node.
- Type
Anchor
- property autocancel¶
An attribute for automatic removal of the node which has completed.
- Type
- property children¶
The list of all direct child nodes.
- Type
list
- property complete¶
The node complete status.
- Type
bool
- property completes¶
An attribute for setting a condition for setting the node as complete depending on other tasks or families.
- Type
- property events¶
An attribute for declaring an action that a node can trigger while it is running.
- Type
- property executable_children¶
The list of all tasks and families directly contained within a Family.
- Type
list
- property follow¶
An attribute for setting a condition for running the node behind another repeated node which has completed.
- Type
- property fullname¶
The full path of the node from the root.
- Type
str
- property headers¶
The current and parent node headers, including head and tail.
- Type
list
- property host¶
Returns the currently active host object. If not found in current node, search in parents.
- Returns
Currently active host object.
- Return type
- property inlimits¶
An attribute for grouping of nodes to which a limit should be applied.
- Type
- property limits¶
An attribute for a simple load management by limiting the number of tasks submitted by a specific ecFlow server.
- Type
- property manual¶
The manual of the node, i.e. help text.
- Type
str,list
- property manual_path¶
The deployment path of the current task, may be None.
- Type
str
- property meters¶
An attribute for a range of integer values that can be set from a script.
- Type
- property modules¶
The list of environment modules for the node.
- Type
list
- property name¶
The visible name of the node.
- Type
str
- property parent¶
The parent node object.
- Type
Node
- property path_list¶
The list of node paths.
- Type
list
- property queued¶
The node queued status.
- Type
bool
- property submitted¶
The node submitted status.
- Type
bool
- property to_html¶
The representation of the node in HTML.
- Type
str
- property today¶
An attribute for setting a cron dependency of the node for the current day.
- Type
- property triggers¶
An attribute for setting a condition for running the node depending on other tasks or families.
- Type
- property unknown¶
The node unknown status.
- Type
bool
- property value¶
The value of the node.
- Type
str
- property workdir¶
Returns the currently working directory for tasks. If not found in current node, search in parents.
- Returns
Currently active working directory for tasks.
- Return type
string
Miscellaneous¶
State¶
- pyflow.state.aborted = ecflow.ecflow.DState.aborted¶
Defines an aborted state.
- pyflow.state.active = ecflow.ecflow.DState.active¶
Defines an active state.
- pyflow.state.complete = ecflow.ecflow.DState.complete¶
Defines a complete state.
- pyflow.state.queued = ecflow.ecflow.DState.queued¶
Defines a queued state.
- pyflow.state.submitted = ecflow.ecflow.DState.submitted¶
Defines a submitted state.
- pyflow.state.suspended = ecflow.ecflow.DState.suspended¶
Defines a suspended state.
- pyflow.state.unknown = ecflow.ecflow.DState.unknown¶
Defines an unknown state.
Helper Functions¶
- pyflow.ecflow_name(s)¶
Converts a string value to one that can be used in names of ecFlow objects, by removing or converting illegal characters.
- Parameters
s (str) – A string value to convert.
- Returns
The string value safe for use in names of ecFlow objects.
- Return type
str
Example:
pyflow.ecflow_name('hyphenated-name')
- pyflow.all_complete(nodes)¶
Returns a trigger expression for all of the supplied nodes being complete.
- Parameters
nodes (list) – The list of input nodes.
- Returns
Trigger expression for all of the supplied nodes being complete.
- Return type
expression
Example:
trigger = pyflow.all_complete(pyflow.Task('task-{}'.format(i)) for i in range(10))
- pyflow.sequence(nodes)¶
Sets triggers so that the input tasks/families will run in sequence, in the order that they are given.
The input nodes are modified in-place. Existing triggers will be preserved by &-ing them with the new trigger.
- Parameters
nodes (list) – The list of input nodes.
Example:
pyflow.sequence(pyflow.Task('{}_{}'.format(name, i), i) for i in range(counters))
Deferred¶
- class pyflow.Deferred(func, *args)¶
Defines an expression which will be evaluated only at suite generation time.
- Parameters
func (function) – The callback function to generate the expression.
*args (tuple) – Accept extra positional arguments for the callback function.
Example:
def my_callback(var, val): pass pyflow.Deferred(my_callback, 'foo', 'bar')
- generate_expression(parent=None)¶
Generates the expression.
- Parameters
parent (Node) – The parent node to generate the expression for.
- Returns
Generated expression.
- Return type
expression