Yabi - A Workflow Management System and Workflow Engine for Data Reduction Pipeline

Wednesday, 10 October, 2018 - 12:13

THE SITUATION

To target a specific experiment, researchers developed different workflows management systems (WMS), to model and run complex recurrent tasks. The problem is that many of this workflow only work for that specific experiment, being unlikely to be reused on a different one.

THE NEED

Several WMS are available, targeted to a specific experiment or scientific community. However, tests and comparisons with other similar solutions are necessary to understand how a workflow can be exported from an environment to another, to guarantee interoperability and reusability. This will allow an easy export and import of workflows between distinct WMS.

THE SOLUTION

Yabi is an application stack that provide users with an intuitive, easy to use, abstraction of compute and data environments, being deployed across a diverse set of scientific disciplines and high performance computing environments.

HOW DOES IT WORK?

Thanks to an easy web-based access, Yabi describes and manages jobs with an internal MySQL database. Even if Common Workflow Language (CWL) is not natively supported in Yabi, tools, jobs, workflows, etc. can be exported in JSON files that can be easily translated in CWL and vice versa. In addition, automated transfer of workflows from other workflow management systems, like Galaxy, to Yabi is also possible.

THE BENEFIT FOR ASTRONOMY?

If astronomical data centers adopt this kind of solutions, the final user does not have to worry about software installation, hardware configuration, or retrieving huge data from complex archive, but can just focus on scientific analysis.

Contact

Andrea Bignamini: andrea [dot] bignaminiinaf [dot] it
Visit the Yabi page on the ASTERICS - OBELICS Software Repository
Yabi source code on GitHub: https://github.com/muccg/yabi
Note: Yabi is released under GNU GPL v3 for non-commercial use only.

Paper: Hunter AA, Macgregor AB, Szabo TO, Wellington CA and Bellgard MI, Yabi: An online research environment for Grid, High Performance and Cloud computing, Source Code for Biology and Medicine 2012, 7:1 doi:10.1186/1751-0473-7-1 Published: 15 February 2012 Source Code for Biology and Medicine (http://www.scfbm.org/content/7/1/1/abstract)