MGEScan on Galaxy Workflow¶

Overview¶

This tutorial demonstrates a quick start of using MGEScan on Galaxy workflow with a sample dataset, D. melanogaster genome. A public server at Indiana University (http://silo.cs.indiana.edu:38080) provides sample datasets and MGEScan tools to try MGEScan on Galaxy without installation hassle.

Run MGEScan-LTR and MGEScan-nonLTR for D. melanogaster¶

In this tutorial, we will try to run both MGEScan-LTR and MGEScan-nonLTR with D. melanogaster genome dataset. You can find the dataset at the Shared Data menu on top and MGEScan tools on the left frame.

Access to Galaxy/MGEScan¶

Open Galaxy/MGEScan at your web browser:

• http://silo.cs.indiana.edu:38080

Login or Register (Optional)¶

You can save your work if you have account on Galaxy workflow. The user-based history in Galaxy/MGEScan stores your data and launched tasks. The guest user account is able to run the MGEScan tools without the login but results or history data won’t be saved if the web browser session is closed.

Register¶

Email address is required to sign up.

• http://silo.cs.indiana.edu:38080/user/create

Login¶

If you already have an account, you can use your user id and password at the User > Login page.

• http://silo.cs.indiana.edu:38080/user/login

Example: Drosophila melanogaster¶

In the Data Library, enable the checkbox for d.melanogaster and click “Select datasets for import into selected histories” from the down arrow at the end.

You will find 8 fasta files are available. We need to import all of them, make them all checked and click “Import library datasets” in the middle of the page.

Once you imported the D. melanogaster datasets into your history, you are ready to run MGEScan tools on Galaxy. Go to the main page, and checkout imported datasets (8 files) on the right frame of the page.

Note

You can select where datasets to be imported.

Create a single link to multiple inputs¶

In the example of d. melanogaster, we have 8 fasta files as its sequences. To analyze them all at the same time, we create a single link to the files prior to running MGEScan tool on Galaxy. One archive file to many files (e.g. file.tar) will be used as an input of MGEScan tool on Galaxy. Note that Galaxy workflow does not support multiple arbitrary inputs but this symlink tool allows you to have dynamic inputs as a Galaxy input dataset.

• FInd “Tools > Create a symlink to multiple datasets” on the left frame.

We will add 8 fasta files each by clicking “Add new Dataset” from “8: Drosophila_melanogaster.BDGP6.dna.chromosome.dmel_mitochondrion_genome.fa” to “1: Drosophila_melanogaster.BDGP6.dna.chromosome.2L.fa” like so:

Make sure you have added all the files without duplication. The added order is not important though. File(s) will be placed in a same directory without order.

MGEScan Tool¶

MGEScan runs both LTR and nonLTR with a selected input genome sequence. Find “MGEScan > MGEScan” tool on the left frame and confirm that the symlink dataset we created in the previous step is loaded in “From” select form like so:

Enable MPI¶

To accelerate processing time, select “Yes” at “Enable MPI” select form and specify “Number of MPI Processes”. If you have a multi-core system, use up to the number of cores. silo.cs.indiana.edu has 24 cores but we will use 4 in this tutorial to avoid being a noisy neighbor.

Our options are:

• From: Create a symlink to multiple datasets on data 2 and data 8, and others
• MGEScan: Both
• Enable MPI: Yes
• Number of MPI Processes: 4

And click “Execute”.

Computation Time¶

Our test case took 3 hours for analyzing LTR and nonLTR of D. melanogaster:

• nonLTR: 19 minutes
• LTR: 3 hours
• Total: 3 hours

Results¶

Upon the MGEScan tools completion, the output files are accessible via Galaxy in gff3 format, a plain text, or an archived (e.g. tar.gz) file. You will notice that the color of your tools has been changed to green like so:

You can download the output files to your local storage, or get access to Genome Browser with provided links.

Visualization: UCSC or Ensembl Genome Browser¶

Your genomic data in a Generic Feature Format Version 3 (gff3) can be displayed by a well known visualization tool such as UCSC or Ensembl Genome Browser on Galaxy with custom annotations of MGEScan for LTR and nonLTR. Find the link provided for gff3 to view interactive graphical display of genome sequence data.

UCSC Genome Browser (Example View)¶

Ensembl (Example View)¶

Additional Options¶

There are other options to view results on a web interface or local.

• View data: Content of the result file

• Download: Download the file

Description of tools¶

Each tool in Galaxy has its description to explain how to use.

MGEScan (Both)¶

This workflow contains 10 steps to run both LTR and nonLTR programs in parallel. Find “MGEScan (Both)” at Workflow menu on top.

MGEScan-LTR¶

This workflow contains 3 steps to run the LTR program.

• Step 1: Split scaffolds
• Step 2: RepeatMasker (optional)
• Step 3: Finding ltr
• Step 4: gff converter

MGEScan-nonLTR¶

This workflow contains 6 steps to run the nonLTR program.

• Create a symlink to multiple datasets
• Step 1: forward strand
• Step 2: Reversing Complement
• Step 3: backward strand
• Step 4: Validating Q Value
• Step 5: gff converter

Workflow Canvas¶

In Galaxy > Workflow > Edit, you can modify or update the MGEScan workflow on Galaxy Workflow Canvas.

Registered Workflow in Local¶

Once you completed composing/updating workflow, you can save your work on local. You can download and store workflow file on your storage.

Registered Workflow in Public Server (usegalaxy.org)¶

Through Galaxy Public Workflow Website, your workflow can be shared with other scientists and researchers. MGEScan workflow has been registed on https://usegalaxy.org/workflow/list_published.

Overview of MGEScan Workflow (Draft)¶

The published MGEScan workflow consists of LTR and non-LTR programs in parallel. LTR has four components including splitting scaffolds, pre-processing by repeatmasker, finding LTRs, and converting results in gff3 format.

Quick Start

Installation¶

If you have installed MGEScan on Galaxy, MGEScan CLI tools are available on your system.

Usage¶

Try mgescan -h on your terminal:

(mgescan)$ mgescan -h
MGEScan: identifying ltr and non-ltr in genome sequences

Usage:
        mgescan both <genome_dir> [--output=<data_dir>] [--mpi=<num>]
        mgescan ltr <genome_dir> [--output=<data_dir>] [--mpi=<num>]
        mgescan nonltr <genome_dir> [--output=<data_dir>] [--mpi=<num>]
        mgescan (-h | --help)
        mgescan --version

Options:
        -h --help   Show this screen.
        --version   Show version.
        --output=<data_dir> Directory results will be saved

MGEScan Programs¶

mgescan CLI tool provides options to run ltr, nonltr or both programs.

(mgescan)$ mgescan both $HOME/dmelanogaster --output=$HOME/mgescan_result_dmelanogaster

ltr: starting
nonltr: starting
nonltr: finishing (elapsed time: 306.881129026)
ltr: finishing (elapsed time: 1306.881129026)

$ ls -al dmelanogaster
total 167564
drwx------  2 mgescan mgescan     4096 Jan 28 23:23 .
drwx------ 13 mgescan mgescan     4096 Apr  7 18:45 ..
-rw-------  1 mgescan mgescan 23395126 Dec 18  2014 2L.fa
-rw-------  1 mgescan mgescan 21499210 Dec 18  2014 2R.fa
-rw-------  1 mgescan mgescan 24952673 Dec 18  2014 3L.fa
-rw-------  1 mgescan mgescan 28370194 Dec 18  2014 3R.fa
-rw-------  1 mgescan mgescan  1374441 Dec 18  2014 4.fa
-rw-------  1 mgescan mgescan 22796595 Dec 18  2014 X.fa
-rw-------  1 mgescan mgescan  2796595 Dec 18  2014 Y.fa

Preparation¶

There are required software to be installed prior to run MGEScan. You need to install system packages with sudo command (admin root privilege is required). virtualenv is used for Python package installation.

• root privilege to install packages with sudo

Quick Installation¶

One-liner command provides a quick installation of required software and configuration.

curl -L https://raw.githubusercontent.com/MGEScan/mgescan/master/one-liner/ubuntu | bash

Start a Galaxy/MGEscan web server with a default port 38080.

source ~/.mgescanrc
cd $GALAXY_HOME
nohup sh run.sh &

Normal Installation¶

Software for Python¶

If virtualenv, git, and python-dev are available on your system, you can skip this step.

Ubuntu

sudo apt-get update
sudo apt-get install python-virtualenv python-dev git -y

Fedora

sudo yum update
sudo yum install python-virtualenv python-devel git -y

Environment Variables¶

MGEScan will be installed on a default directory $HOME/mgescan3. You can change it if you prefer other location to install MGEScan.

export MGESCAN_HOME=$HOME/mgescan3
export MGESCAN_SRC=$MGESCAN_HOME/src
export GALAXY_HOME=$MGESCAN_HOME/galaxy
export TRF_HOME=$MGESCAN_HOME/trf
export RM_HOME=$MGESCAN_HOME/RepeatMasker
export MGESCAN_VENV=$MGESCAN_HOME/virtualenv/mgescan

Create a MGESCan start file .mgescanrc

cat <<EOF > $HOME/.mgescanrc
export MGESCAN_HOME=\$HOME/mgescan3
export MGESCAN_SRC=\$MGESCAN_HOME/src
export GALAXY_HOME=\$MGESCAN_HOME/galaxy
export TRF_HOME=\$MGESCAN_HOME/trf
export RM_HOME=\$MGESCAN_HOME/RepeatMasker
export MGESCAN_VENV=\$MGESCAN_HOME/virtualenv/mgescan
EOF

Then include it to your startup file (i.e. .bash_profile).

echo "source ~/.mgescanrc" >> $HOME/.bash_profile

Create a main directory.

source ~/.mgescanrc
mkdir $MGESCAN_HOME

Software for MGEScan¶

Galaxy Workflow, HMMER (3.1b1), EMBOSS Suite and TRF are required. RepeatMasker is optional.

cd $MGESCAN_HOME
git clone https://github.com/galaxyproject/galaxy/

sudo apt-get install hmmer emboss -y

sudo yum install gcc -y
wget ftp://selab.janelia.org/pub/software/hmmer3/3.1b2/hmmer-3.1b2-linux-intel-x86_64.tar.gz
tar xvzf hmmer-3.1b2-linux-intel-x86_64.tar.gz
cd  hmmer-3.1b2-linux-intel-x86_64
./configure
make
make check
make install

wget ftp://emboss.open-bio.org/pub/EMBOSS/emboss-latest.tar.gz
tar xvzf emboss-latest.tar.gz
cd EMBOSS-*
./configure
make
make check
make install

sudo apt-get install openmpi-bin libopenmpi-dev -y

mkdir -p $MGESCAN_VENV
virtualenv $MGESCAN_VENV
source $MGESCAN_VENV/bin/activate
echo "source $MGESCAN_VENV/bin/activate" >> ~/.bash_profile

mkdir -p $TRF_HOME
wget http://tandem.bu.edu/trf/downloads/trf407b.linux64 -P $TRF_HOME

mkdir $RM_HOME
wget http://www.repeatmasker.org/RepeatMasker-open-4-0-5.tar.gz
tar xvzf RepeatMasker-open-4-0-5.tar.gz
mv RepeatMasker/* $RM_HOME
ln -s $RM_HOME/RepeatMasker $MGESCAN_VENV/bin/

MGEScan Installation¶

MGEScan can be installed from Github repository (source code):

cd $MGESCAN_HOME
git clone https://github.com/MGEScan/mgescan.git
ln -s mgescan src
cd $MGESCAN_SRC
python setup.py install

Configuration¶

source $MGESCAN_VENV/bin/activate

cp -pr $MGESCAN_SRC/galaxy-modified/* $GALAXY_HOME

ln -s $TRF_HOME/trf407b.linux64 $MGESCAN_VENV/bin/trf
chmod 700 $MGESCAN_VENV/bin/trf

cd $RM_HOME
$RM_HOME/configure

sudo yum install perl-Data-Dumper perl-Text-Soundex -y
cd $RM_HOME
$RM_HOME/configure

RepeatMasker Configuration Program

This program assists with the configuration of the
RepeatMasker program.  The next set of screens will ask
you to enter information pertaining to your system
configuration.  At the end of the program your RepeatMasker
installation will be ready to use.

 <PRESS ENTER TO CONTINUE>

export GALAXY_ADMIN=mgescan_admin@mgescan.com

sed -i "s/#admin_users = None/admin_users = $GALAXY_ADMIN/" $GALAXY_HOME/universe_wsgi.ini

Start Galaxy¶

Simple run.sh script starts a Galaxy web server. First run of the script takes some time to initialize database.

cd $GALAXY_HOME
nohup sh run.sh &

Deploying MGEScan on Galaxy¶

First step is getting an Amazon account to launch virtual instances on Amazon IaaS platform EC2.

AWS EC2 Account¶

If you already have an account of Amazon AWS EC2, open AWS Management Console to launch our MGEScan image on EC2. Otherwise, create an AWS Account.

• http://aws.amazon.com

MGEScan Machine Image¶

In AWS Management Console, open EC2 Dashboard > Launch Instance. To choose an Amazon Machine Image (AMI) of MGEScan, select Community AMIs on the left tab, and search by name or id, e.g. mgescan or ami-10672b7a. (US East Region Only)

MGEScan EC2 Image Information¶

• Region: US East
• Image Name: MGEScan
• ID: ami-10672b7a
• Server type: 64bit
• Description: MGEscan on Galaxy for identifying LTR and nonLTR
• Root device type: ebs
• Virtualization type: hvm

Choose an Instance Type for MGEScan Instance¶

Once you choose MGEScan image as a base image, you need to select the size of instance. t2.micro uses 1 vCPUs and 1 GB memory which is in free tier. Ohter options are available to have large instance e.g. 40 vCPUs. Click Review and Launch icon at bottom of the page.

Tip

t2.micro: (Variable ECUs, 1 vCPUs, 2.5 GHz, Intel Xeon Family, 1 GiB memory, EBS only)

Security Group for Web¶

MGEscan / Galaxy uses 38080 default web port. We need to add a rule to have this port opened on the new instance.

There are a few steps you have to follow.

• Find “Security Groups” section and click “Edit security groups”. “Create a new

security group” is selected as a default with a 22 SSH port opened to anywhere.

• We will add 38080 tcp port. Click “Add Rule” and type 38080 in the “Port Range” input box.
• Don’t forget to update “Source” to “Anywhere” from “Custom IP”.
• Once you’re done, click “Reivew and Launch”.
• Click “Launch” again.
• Choose a SSH keypair from existing or new one.
• Click “Launch Instance” and wait.
• Find out public IP address and open a web browser with the address. e.g. http://[IP address]:38080 Don’t forget the port number 38080

Access to MGEScan Instance¶

Once the MGEScan instance is launched and accessible, galaxy scientific workflow system for MGEScan and SSH connection are avabilable through given dns name.

Ready To Use¶

The MGEScan is now ready to conduct your experiment on Amazon EC2.

Note

Do not forget to terminate your virtual instance after all analysis completed. Amazon Cloud charges use of VM instances hourly.

Terminating AWS Instance:

Note¶

Add a script to auto-start Galaxy after reboot in /etc/rc.local

su ec2-user -c 'source ~/.mgescanrc;cd $GALAXY_HOME;nohup sh run.sh &'

Description¶

MGEScan-LTR identifies all types of LTR retrotransposons, i.e., young intact, old intact, and solo LTR retrotransposons, without relying on a library of known elements. It uses approximate string matching, protein domain analysis, and profile Hidden Markov Models to identify intact LTR retrotransposons.

For details, please read following references.

• Rho, M., et al. (2007) De novo identification of LTR retrotransposons in eukaryotic genomes. BMC Genomics, 8, 90.
• Rho, M., et al. (2010) LTR retroelements in the genome of Daphnia pulex. BMC Genomics, 11, 425.

Running the program¶

To run MGEScan-LTR, follow the steps below,

• Specify options that you like to have:
  • Check repeatmasker if you want to preprocess
  • Check scaffold if the input file has all scaffolds.
• Update values:
  • min_dist: minimum distance(bp) between LTRs.
  • max_dist: maximum distance(bp) between LTRS
  • min_len_ltr: minimum length(bp) of LTR.
  • max_len_ltr: maximum length(bp) of LTR.
  • ltr_sim_condition: minimum similarity(%) for LTRs in an element.
  • cluster_sim_condition: minimum similarity(%) for LTRs in a cluster
  • len_condition: minimum length(bp) for LTRs aligned in local alignment.
• Click ‘Execute’

Options¶

• RepeatMasker: Yes / No
• file path for multiple sequences to divide
• settings for LTRs
  • minimum distance(bp) between LTRs
  • maximum distance(bp) between LTRs
  • minimum length(bp) of LTR
  • maximum length(bp) of LTR
  • minimum similarity(%) for LTRs in an element
  • minimum similarity(%) for LTRs in a cluster
  • minimum length(bp) for LTRs aligned in local alignment

Results¶

Upon completion, MGEScan-LTR generates a file ltr.out. This output file has information about clusters and coordinates of LTR retrotransposons identified. Each cluster of LTR retrotransposons starts with the head line of [cluster_number]———, followed by the information of LTR retrotransposons in the cluster. The columns for LTR retrotransposons are as follows.

• LTR_id: unique id of LTRs identified. It consist of two components, sequence file name and id in the file. For example, chr1_2 is the second LTR retrotransposon in the chr1 file.
• start position of 5 LTR.
• end position of 5 LTR.
• start position of 3 LTR.
• end position of 3 LTR.
• strand: + or -.
• length of 5 LTR.
• length of 3 LTR.
• length of the LTR retrotransposon.
• TSD on the left side of the LTR retotransposons.
• TSD on the right side of the LTR retrotransposons.
• di(tri)nucleotide on the left side of 5LTR
• di(tri)nucleotide on the right side of 5LTR
• di(tri)nucleotide on the left side of 3LTR
• di(tri)nucleotide on the right side of 3LTR

License¶

Copyright 2015. You may redistribute this software under the terms of the GNU General Public License.

Description¶

MGEScan-nonLTR identifies non-LTR retrotransposons based on Gaussian Bayes classifiers and generalized hidden Markov models consisting of twelve super states that correspond to different clades or closely related clades.

For details, please read following reference.

• Rho, M., Tang, H. (2009) MGEScan-non-LTR: computational identification and classification of autonomous non-LTR retrotransposons in eukaryotic genomes. Nucleic Acids Research, 37(21), e143.

Running the program¶

To run MGEScan-nonLTR, follow the steps below:

• Select genome files a select box. You can upload your genome files through ‘Get Data’ at Tools menu bar.
• Click ‘Execute’ button. This tool reads your genome files and runs the whole process.

Options¶

• hmmmsearch options e.g. -E 0.00001 : reports sequences smaller than 0.00001 E-value threshold in output
• URL of the profile files for RT and APE
• EMBOSS transeq options

Results¶

Upon completion, MGEScan-nonLTR generates output, “info” in the data directory you specified. In this “info” directory, two sub-directories (“full” and “validation”) are generated.

The “full” directory is for storing sequences of elements. Each subdirectory in “full” is the name of clade. In each directory of clade, the DNA sequences of nonLTRs identified are listed. Each sequence is in fasta format. The header contains the position information of TEs identified, [genome_file_name]_[start position in the sequence] For example, >chr1_333 means that this element start at 333bp in the “chr1” file. - The “validation” directory is for storing Q values. In the files “en” and “rt”, the first column corresponds to the element name and the last column Q value.

License¶

Copyright 2015. You may redistribute this software under the terms of the GNU General Public License.

UCSC Genome Browser¶

Source Code¶

In MGEScan source code, ltr/toGFF.py and nonltr/toGFF.py are used to convert results to GFF format developed by Wazim Mohammmed Ismail.

D. melanogaster (dm3)¶

• dm3.gff3
• dm3.ltr.out
• dm3.en
• dm3.rt

C. intestinalis (KH)¶

• KH.gff3
• KH.ltr.out
• KH.en
• KH.rt

D. pulex (GCA_000187875.1)¶

• dpulex.gff3
• dpulex.ltr.out
• dpulex.en
• dpulex.rt