APPLICATIONS:
- Running large, complex computational workflows, pipelines (e.g., Broad Institute’s Cromwell)
- Data analysis workflows in High Performance Computing (HPC), large cluster, and cloud environments
BENEFITS:
- Improves reusability and robustness of workflows/pipelines
- Allows for easier maintenance and modification
- Enables distribution of workflows across different computing facilities
BACKGROUND:
Large, complex multi-compute-node workflows are not easily maintained, modified, or run on multiple computing resources. This is true for the Berkeley Lab’s Joint Genome Institute (JGI) as well as other organizations that run large and complex calculations. The JGI Analysis Workflow Service (JAWS), aims to improve the reusability and robustness of bioinformatic and other workflows in evolving and diverse HPC, large cluster, and cloud environments.
TECHNOLOGY OVERVIEW:
The Joint Genome Institute (JGI) at Berkeley Lab has developed JAWS as a framework to run complex computational workflows at one or more compute sites (e.g. grid-clusters, cloud resources). JAWS moves data and code to user-specified compute resources, executes computation, and returns results. JAWS leverages Globus, developed at the University of Chicago which implements GridFTP, for high-throughput transfer of files (e.g. workflows’ input data, output results) between the client server and computation sites (i.e. compute clusters). JAWS utilizesCromwell, developed at the Broad Institute, to execute workflows written in either Workflow Description Language (WDL) or Common Workflow Language (CWL). Additionally, JAWS supports containers (compatible with Docker, Shifter, and Singularity) to ensure use of well-defined compute environments and workflow task codebases. To run the compute tasks, JAWS provides the JGI Task Manager (JTM), a compute task framing tool that manages workers on compute nodes and executes tasks on them.
FOR MORE INFORMATION:
https://www.osti.gov/biblio/1797434-joint-genome-institute-analysis-workflow-service-jaws-v2
DEVELOPERS:
- Brian Foster
- Jeff L Froula
- Edward Kirton
- Angela Kollmer
- Mario Melara
- Georg Rath
- Kelly Rowland
- Seung Jin Sul
- Stephan Trong
STATUS: Copyrighted
OPPORTUNITIES: Available/Open Source