Sharing workflow between labs: practices
This page try to list main steps when several galaxy instances whish to exchange workflows.
The first two sections deals with wrappers standardization and toolshed integration.
Instances which give workflows are concerned with these sections.
The last two sections deals with wrappers installation from toolshed and different kind of tests you can run after tools installation.
Instances which receive workflows are concerned with these sections.
Prepare your wrappers
Note: you must be administrator of your instance to realize this section.
To begin the process, prepare the wrappers and the associated files of the workflow according to items listed below:
- No absolute path in a xml wrapper file or in a shell script.
Replace absolute path with current directory path: "./".
- If you use a shell script, add the following lines:
directory/myexecfile -i $input_tree $*;
where "myexecfile" is the name of the executable file your are wrapping.
- Create a directory for the tool and put here all necessary files:
XML wrapper, executable file , shell script and input data files for a basic test.
- Reload the tool via the admin interface: Admin/server/reload a tool's configuration.
Install workflow's tools on a given toolshed
Note: you must be administrator of the given toolshed instance and if the galaxy instance to achieve this section.
- Create a tar ball archive of the tool.
Go to the directory previously created and: tar -cvf myarchive.tar.gz ./*
- Connect to the galaxy tool shed server (as administrator).
- Select "Create a new repository" (you can also create a new category if you whish to organize different workflows)
- Select "Upload files to repository" and load the tar ball previously created.
Select yes for the "Uncompress file" option.
Now, install the workflow on the toolshed
Note: again, you must be administrator of the toolshed.
- On your galaxy instance, create the workflow you wish to exchange.
- In the workflow list, select "Download or export", then "Download to file". A local copy on the workflow is generated.
- On the toolshed server create a repository for your workflow.
- In "Repository actions", choose "upload file", and upload the local copy of your workflow.
You can check the upload result: go to "manage repository" a link to a svg representation of your workflow is avalaible.
Install the workflow on an other instance
Note: you must be administrator of this "other" instance.
You will now import the worklow from the "given" toolshed repository. This toolshed must be declared in tool_shed_conf.xml of the "other" instance: <tool_shed name="A Given ToolShed " url="the url of the toolshed” />.
The label a "Given ToolShed" is the label that identify this toolshed on the "other" instance.
The workflow installation will follow two main steps:
- First, you will have to install the repository previously created:
Select "Admin", "Search and browse toolsheds", select the corresponding repository and the option "preview and install".
- After this step only the repository will be installed on the instance.
To install the worklow you will have to import it. Go to the newly repository installed and select the workflow previously packaged. Select "Import and workflow to Galaxy".
After this step the workflow should be exploitable in the "other" galaxy instance.
Note: We noticed import workflows issues for version older the Feb 2013.
Validate new workflow on the "other" instance
In this section we want to validate the workflow newly installed on the instance. You can notice that this protocol could be followed when you want to test the existing workflows and tools on your instance (after a galaxy new release update for example).
- Functional tests
Even before being integrated into galaxy a tool could have a corresponding functional test. This "black box" testing is useful to check regression when a program is modified. In our galaxy context this test is not useful, but if the test is available, input and output data will be useful for galaxy functional tests (see next section).
- Functional tests in a Galaxy instance
The galaxy platform provide a functional test framework: http://wiki.galaxyproject.org/Admin/Running%20Tests. First feedbacks from the French community indicate that tests are easy to implement.
If you have a lot of tools on your instance and decide to write functional tests for each, this job will be not straightforward: You will have to get a lot of input and output, and running all the tests could take a lot of time.
In the context of tools exchange between platforms, input and output data and functional must be present in the archive uploaded in the toolshed.
- A complete workflow for validation
As we saw below, testing a work tools by tools could not be straightforward (several input/output data to get, time issues). A less robust but more easy way is to design a workflow for validation. This workflow could be well suited for a given bioinformatic analysis, but if you have independent tools and you want to test each, it is possible to group all the tests in a single workflow (see different testing workflow on URGI instance: http://urgi.versailles.inra.fr/galaxy2/workflow/list_published). If you adopt practices like continuous integration or continuous delivery, you may be interested with the automation of these workflows. For this purpose the galaxy API is well suited. In the context of workflow sharing between labs, validation workflow and corresponding input and output data should be integrated in the toolshed.