layout: true class: inverse --- class: special, center  # Reference Genomes in Galaxy **Slides: @blankenberg, @Slugger70** .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- layout: true class: left, inverse --- class: left, middle, center  ## Please interrupt *We are here to answer questions!* .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Overview .large[ * **Intro to built in datasets** * Built in data hierarchy * Some problems * Data Managers ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Built in Data  .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Data, what data? .large[ * Some genomes are large! Human, Mouse, Coral * Some tools require indices of the genomes. * The indices take a long time to build! * Better to pre-build the indices. ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Overview .large[ * Intro to built in datasets * **Built in data hierarchy** * Some problems * Data Managers ] --- ##  Data schematics in Galaxy  .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Using reference data in a tool #### bwa.xml ``` xml <conditional name="reference_source"> <param name="reference_source_selector" type="select" label="Will you select a reference genome from your history or use a built-in index?" help="Built-ins were indexed using default options. See 'Indexes' section of help below"> <option value="cached">Use a built-in genome index</option> <option value="history">Use a genome from history and build index</option> </param> <when value="cached"> <param name="ref_file" type="select" label="Using reference genome" help="Select genome from the list"> <options from_data_table="bwa_mem_indexes"> <filter type="sort_by" column="2" /> <validator type="no_options" message="No indexes are available" /> </options> <validator type="no_options" message="A built-in reference genome is not available for the build associated with the selected input file"/> </param> </when> <when value="history"> ``` .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Where are the data tables? #### tool_data_table_conf.xml (Usually located in `galaxy/config/`) ``` xml <tables> <!-- Locations of indexes in the BWA mapper format --> <table name="bwa_indexes" comment_char="#" allow_duplicate_entries="False"> <columns>value, dbkey, name, path</columns> <file path="tool-data/bwa_index.loc" /> </table> </tables> ``` .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  "loc" files - Short for location! Not *"sending me loco"* bwa_index.loc ``` text ... # #<unique_build_id> <dbkey> <display_name> <file_path> # ... bosTau7 bosTau7 Cow (bosTau7) /mnt/galaxyIndices/genomes/bosTau7/bwa_mem_index/bosTau7/bosTau7.fa ce10 ce10 C. elegans (ce10) /mnt/galaxyIndices/genomes/ce10/bwa_mem_index/ce10/ce10.fa danRer7 danRer7 Zebrafish (danRer7) /mnt/galaxyIndices/genomes/danRer7/bwa_mem_index/danRer7/danRer7.fa dm3 dm3 D. melanogaster Apr. 2006 (BDGP R5/dm3) (dm3) /mnt/galaxyIndices/genomes/dm3/bwa_mem_index/dm3/dm3.fa hg19 hg19 Human (hg19) /mnt/galaxyIndices/genomes/hg19/bwa_mem_index/hg19/hg19.fa hg38 hg38 Human (hg38) /mnt/galaxyIndices/genomes/hg38/bwa_mem_index/hg38/hg38.fa mm10 mm10 Mouse (mm10) /mnt/galaxyIndices/genomes/mm10/bwa_mem_index/mm10/mm10.fa ... ``` .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Overview .large[ * Intro to built in datasets * Built in data hierarchy * **Some problems** * Data Managers ] --- ##  Some Problems! .large[ * Time consuming! * ~30 minutes work just to add a new genome to 1 tool! * Administrator needs to know: * how to index **every** tool * expected format of the reference data * format of the .loc file ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Typical conversation .middle[] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Typical conversation .middle[] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Typical conversation .middle[] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Typical conversation .middle[] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Other concerns .large[ * **Accessible?** * Manually download genome FASTA files * Download, compile, run bwa index; which options? * **Reproducible?** * Only if the person performing manual steps keeps good notes * **Transparent?** * Send email to sysadmin asking for notes * Restart Galaxy server for new entries ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- class: left ##  Overview .large[ * Intro to built in datasets * Built in data hierarchy * Some problems * **Data Managers** ] (now we're onto the good stuff!) --- ##  Data Managers .large[ * Allows for the **creation of built-in** (reference) data * underlying data * data tables * \*.loc files * Specialized Galaxy tools that can only be accessed by an admin * Defined **locally** or installed from **ToolShed** ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Data Managers .large[ * **Flexible** framework * Not just genomic data * Run Data Managers through UI * Workflow compatible * API * Examples * Adding new genome builds (dbkeys) * Fetching genome (fasta) sequences * Building short read mapper indices for genomes ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Special class of Galaxy tool Looks just like a normal Galaxy tool!  .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  What does it do? The output of the data manager is a JSON description of the new data table entry  This gets turned into a new data table entry  The index files themselves get placed in the appropriate location. .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Data Managers Admin .large[ * Located on the Galaxy's Admin Tab under **Local Data** ]  .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Data Managers Admin .large[ * UI tools to fetch reference genomes/build indices * View progress of index build jobs * View contents of tool data tables ]  .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Resources / further reading .large[ * Galaxy Wiki Page on Data Managers * Details * Building * Examples https://wiki.galaxyproject.org/Admin/Tools/DataManagers ] .footnote[\#usegalaxy \#GAT2017 / @galaxyproject] --- ##  Exercise Time! .footnote[\#usegalaxy \#GAT2017 / @galaxyproject]