Changes between Version 36 and Version 37 of BigCompute


Ignore:
Timestamp:
Dec 21, 2010 5:33:25 PM (14 years ago)
Author:
Barbera van Schaik
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • BigCompute

    v36 v37  
    6060* Available disk space on the grid storage elements / worker nodes
    6161
     62== Data location on grid ==
     63
     64=== Data ===
     65
     66Browse to the following directory with the Vbrowser (this directory is only readable/writable for the gvnl group):
     67lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data
     68
     69Each component in the workflow generates a new sub directory.
     70
     71If you would like to retrieve the analysis results of the picard-qc component of one of the samples, you go to the directory:
     72* bwa_databasename_parametername/samplename_libraryname_lanename/picard1.32qc/
     73* Example: lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/A4a_HUModqRBVDIBAPE_L7
     74
     75The results for the analysis where we have splitted the fastq-files is located in the directories that start with:
     76* split_fileprefix_1/
     77
     78Note that not everything is there yet and that the data structure may change, since we are still testing.
     79
     80=== Workflows and databases ===
     81
     82These directories are open to all members of the vlemed VO
     83
     84* Workflows: lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_WF
     85* Databases: lfn://lfc.grid.sara.nl:5010/grid/vlemed/AMC-e-BioScience/Sequence_DB
     86
     87The directories that contain the workflows have the following structure:
     88|| '''Directory''' || '''Description''' ||
     89|| bin || dependent binaries like bwa, samtools ||
     90|| GasW || component description, describes which executable has to run on grid and it specifies the in and output files/parameters ||
     91|| Java || dependent jar files, e.g. the GATK jar ||
     92|| parameterFiles || text files that contain exactly one line with the parameters that you would like to provide to bwa or another component ||
     93|| Scufl || old workflow description files, can be ignored ||
     94|| shFiles || these files are executed on the grid and described by the GASW descriptor, the gvnl shFiles are based on the Groningen templates ||
     95|| Workflows || Workflow descriptions, clicking on them will start the Moteur plugin. Input files/parameters can be specified in the fields. If you click on "Run" the jobs are submitted to the grid ||
     96More info in [http://www.ncbi.nlm.nih.gov/pubmed/21156038 Luyf and van Schaik et al (2010)]
     97
     98The shell files can (in most cases) run on any linux cluster. In that case you need to place the shell file and the dependent executable(s) in one directory. At the start of each shell file is an example on how to run them.
     99
    62100== Workflow execution ==
    63101
    64102''' On mini pilot. Split lines-per-file: 8,000,000. Start: 18-12-2010 16:40 '''
    65103|| '''Lane''' || '''split BWA merge''' || '''Comments''' || '''Elapsed time (s)''' ||
    66 || A4a_L4_HUModqRBUDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-cOunUZ/html/workflow-cOunUZ.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100826_I176_FC804GCABXX_L6_HUModqRBUDIBAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/A4a_HUModqRBUDIBAPE_L4/mergebam lfc] || 22740 ||
    67 || A4a_L6_HUModqRADDIAAPE  || [https://hangplek.nikhef.nl:443/workflows/workflow-pqWdc1/html/workflow-pqWdc1.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100826_I124_FC20813ABXX_L7_HUModqRADDIAAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/A4a_HUModqRADDIAAPE_L6/mergebam lfc] || 37750 ||
     104|| A4a_L4_HUModqRBUDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-cOunUZ/html/workflow-cOunUZ.html done] || || 22740 ||
     105|| A4a_L6_HUModqRADDIAAPE  || [https://hangplek.nikhef.nl:443/workflows/workflow-pqWdc1/html/workflow-pqWdc1.html done] || || 37750 ||
    68106|| A4a_L7_HUModqRBVDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-OqeNdv/html/workflow-OqeNdv.html failed] || || ||
    69 || A4b_L3_HUModqRAFDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-GRDX0X/html/workflow-GRDX0X.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100829_I168_FC804GBABXX_L1_HUModqRAFDIBAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/A4b_HUModqRAFDIBAPE_L3/mergebam lfc] || 32256 ||
     107|| A4b_L3_HUModqRAFDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-GRDX0X/html/workflow-GRDX0X.html done] || || 32256 ||
    70108|| A4b_L6_HUModqRBTDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-ZRH98o/html/workflow-ZRH98o.html failed] || || ||
    71109|| R2A _L1_HUModqRADDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-1lNDQh/html/workflow-1lNDQh.html failed] || || ||
    72 || R2A _L1_HUModqRAFDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-rufel6/html/workflow-rufel6.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100809_I125_FC2083DABXX_L1_HUModqRAFDIBAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/R2A_HUModqRAFDIBAPE_L1/mergebam lfc] || 35819 ||
     110|| R2A _L1_HUModqRAFDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-rufel6/html/workflow-rufel6.html done] || || 35819 ||
    73111|| R2A _L5_HUModqRAEDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-wVuMjg/html/workflow-wVuMjg.html failed] || || ||
    74 || R2B _L3_HUModqRBTDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-L3m5Yi/html/workflow-L3m5Yi.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100809_I174_FC2085PABXX_L6_HUModqRBTDIAAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/R2B_HUModqRBTDIAAPE_L3/mergebam lfc] || 23754 ||
     112|| R2B _L3_HUModqRBTDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-L3m5Yi/html/workflow-L3m5Yi.html done] || || 23754 ||
    75113|| R2B _L4_HUModqRBUDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-alGMJK/html/workflow-alGMJK.html failed] || pair 1 not in correct gzip format? || ||
    76114|| R2B _L6_HUModqRBTDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-LCaDkY/html/workflow-LCaDkY.html failed] || || ||
    77 || R2C _L2_HUModqRBUDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-gVXW9i/html/workflow-gVXW9i.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100810_I171_FC20828ABXX_L3_HUModqRBUDIBAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/R2C_HUModqRBUDIBAPE_L2/mergebam lfc] || 40763 ||
     115|| R2C _L2_HUModqRBUDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-gVXW9i/html/workflow-gVXW9i.html done] || || 40763 ||
    78116|| R2C _L2_HUModqRBVDIBAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-8bma4j/html/workflow-8bma4j.html failed] || || ||
    79 || R2C _L7_HUModqRBVDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-Rjh1lC/html/workflow-Rjh1lC.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100810_I128_FC2087PABXX_L7_HUModqRBVDIAAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/R2C_HUModqRBVDIAAPE_L7/mergebam lfc] || 39374 ||
    80 || Unknown_L6_HUModqRBUDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-IG0HAm/html/workflow-IG0HAm.html done] || [lfn://lfc.grid.sara.nl:5010/grid/vlemed/gvnl/Data/split_100804_I124_FC201RNABXX_L6_HUModqRBUDIAAPE_1/bwa_gatk_index_basespace_human_g1k_v37.fasta.tar.gz_bwa_4_threads.txt/Unknown_HUModqRBUDIAAPE_L6/mergebam lfc] || 30307 ||
     117|| R2C _L7_HUModqRBVDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-Rjh1lC/html/workflow-Rjh1lC.html done] || || 39374 ||
     118|| Unknown_L6_HUModqRBUDIAAPE || [https://hangplek.nikhef.nl:443/workflows/workflow-IG0HAm/html/workflow-IG0HAm.html done] || || 30307 ||
    81119Note: The Merge component is red (marked as failed) in the finished workflows. The bam files where produced successfully.
    82120