wiki:BigComputeAnalysisProgress

Version 7 (modified by Barbera van Schaik, 13 years ago) (diff)

--

Workflow execution / progress data analysis

First batch

First alignment step

# Sample WF Status Start
1 first-batch A (17 lanes) workflow-0a2b547d partly done 29-04-2011 21:40
2 first-batch G (50 lanes) workflow-72df4adf partly done 29-04-2011 21:44
3 first-batch (121 lanes) workflow-ecccffa5 28 done 05-05-2011 17:01
4 first-batch (93 lanes + 1 from second batch) workflow-9c7ae662 running 09-05-2011 09:50

Fastqc

# Sample WF Status Start
1 first-batch (366 fastq files) workflow-4888e67c 140 done 09-05-2011 08:57
2 first-batch (226 fastq files) workflow-b8a55600 running 11-05-2011 13:05

Second batch

First alignment step

# Sample WF Status Start
1 A4a F F F workflow-425d9ceb done
2 Vartest workflow-490b15f8 done
3 Iteration test workflow-bf48aff1 failed
4 Iteration test workflow-923c6588 done
5 60-samples-batch (15 lanes) workflow-d80b5767 10 / 15 done 11-02-2011 19:30
6 60-samples-batch A (55 lanes) workflow-cbaca6e5 15 / 55 done 12-02-2011 13:55
7 60-samples-batch A remaining 1 (17 lanes) workflow-835250b1 failed (grid very busy) 07-03-2011 17:45
8 60-samples-batch A remaining 1 (17 lanes) workflow-31eb952d 1/17 done 08-03-2011 14:15
9 60-samples-batch A remaining (27 lanes) workflow-fd98c7c8 11/27 done 15-03-2011 10:45
10 60-samples-batch G (27/54 lanes) workflow-a781209c 3/27 done 15-03-2011 20:37
11 second-batch R10-11-12 (27 lanes) workflow-6fe1cb10 6/27 done 16-03-2011 10:57
12 second-batch R13-14-15 (27 lanes) workflow-77b82882 3/27 done 18-03-2011 19:07
13 second-batch R16-17 (24 lanes) workflow-18ec0160 7/24 done 19-03-2011 11:00
14 second-batch R18 (11 lanes) workflow-7596a676 2/11 done 19-03-2011 18:33
15 second-batch R19-20-21 (31 lanes) workflow-58ea16f1 4 done, scheduled downtime, rest will fail 20-03-2011 11:50
16 second-batch R22-8-9 (altered submission scheme: submits 1 job/5 min) workflow-ea6b36a6 4 done, scheduled downtime, rest will fail 20-03-2011 12:37
17 second-batch R10-R17 A9-23 (206 lanes) workflow-7d2b2b5b 5 done, scheduled downtime, rest will fail 21-03-2011 11:59
18 second-batch (244 lanes) workflow-c902ebf3 This WF is cancelled, because we only want to run these jobs on Gina and HTC 23-03-2011 09:48
19 second-batch (237 lanes) workflow-62e0f254 135 done 25-03-2011 12:20
20 second-batch (102 lanes) workflow-b00af623 33 done 31-03-2011 08:21
21 second-batch (69 lanes) workflow-d5709592 4 done 04-04-2011 18:17
22 second-batch (65 lanes) workflow-7d8743c1 25 done 06-04-2011 23:24
23 second-batch (40 lanes) workflow-d8cdf017 16 done 15-04-2011 18:07
24 second-batch (24 lanes) workflow-177917d4 12 done 18-04-2011 11:25
25 second-batch (12 lanes) workflow-289c831f 5 done 20-04-2011 21:15
26 second-batch (7 lanes) workflow-96f414e3 4 done 22-04-2011
27 second-batch (3 lanes) workflow-63ecec34 1 done 27-04-2011 15:22
28 second-batch (2 lanes) workflow-dd84f331 1 done 02-05-2011 11:05
29 second-batch (1 lane) workflow-8a720670 running 05-05-2011 16:58

Fastqc analysis

# Sample WF Status Start
1 second-batch (2x295 lanes = 590 fastq files) workflow-417415cc 396 done 23-04-2011 02:44
2 second-batch (194 fastq files) workflow-6ded80d3 189 done 23-04-2011 11:40
3 second-batch (5 fastq file) workflow-2bb76072 5 done 23-04-2011 14:40

RESULTS: log-fastqc20110423.xls - contains information about run time and disk usage on the compute nodes and info about the number of sequences per lane

THROUGHPUT: workflow run time was 12 hrs, total CPU run time was 12 days

STATUS: done

First and second batch

Mark-duplicates analysis on all files that are aligned so far

# Sample WF Status Start
1 351 lanes workflow-a5a7a078 212 done 04-05-2011 12:54
2 139 lanes workflow-87b3994d 115 done 05-05-2011 15:29
3 24 lanes workflow-7e395331 failed 06-05-2011 08:22
4 24 lanes workflow-6207c67d 19 done 06-05-2011 12:55
5 5 lanes workflow-62e2653e failed 07-05-2011 14:08
6 37 lanes workflow-05988a60 running 11-05-2011 15:00

Samtools flagstat on bam files

# Sample WF Status Start
1 381 lanes workflow-a7a0acbf 181 done 09-05-2011 16:56
2 200 lanes workflow-39699118 running 11-05-2011 14:46

Notes

Monitor clusters