Changes between Initial Version and Version 1 of BIOS_VirtualMachine

Timestamp:

Sep 19, 2016 11:57:14 AM (8 years ago)

Author:

rick

Comment:

--

BIOS_VirtualMachine

@@ +1 @@
+= BIOS data location and access overview =
+
+Processed files are stored at the BIOS VM. BIOS VM is a virtual machine, a digital computer, running at SURFsara's HPC cloud. It behaves like a normal Linux server would, though it can be connected to graphically.
+
+BIOS Metadatabase can be accessed via a web browser or any client program supporting HTTPS.
+
+Raw files are stored at [http://doc.grid.surfsara.nl/en/latest/Pages/Advanced/grid_storage.html SURFsara's Grid]. SURFsara's Grid is an online data storage system. Accessing Grid requires a certain level of technical skills and to follow [http://doc.grid.surfsara.nl/en/latest/ a registration procedure]. 
+
+[[Image(BIOSdataInfrastructure.png, 600px)]]
+
+= BIOS virtual machine provided by SURFsara =
+
+For safety and privacy reasons, BIOS data (genome, transcriptome, methylome and phenome) is only accessible for downstream analysis at a SURFsara virtual machine (VM). The BIOS VM is managed by Martijn Vermaat and Leon Mei. Since the resources and capacity of this VM are limited, it should only be used for downstream analysis. If you want to work on many BAM files or similar expensive analysis you would normally use a cluster for, it's probably better to get acquainted with working on the grid directly. If you are not sure contact Martijn or Leon.
+
+The current test VM runs these specs:
+ * 16 processors
+ * 128 Gb RAM
+ * 4.5Tb disk space mounted at /virdir. This is the place you could keep your analysis data. The files in /virdir/Backup folder will be backed up about once per month. The files in /virdir/Scratch are not backed up.
+ * 2GB soft limit and 3GB hard limit per user at /home
+
+== BIOS VM Access ==
+
+To get access, please send a request to Leon Mei (`h.mei[at]lumc.nl`) or Martijn Vermaat (`m.vermaat.hg[at]lumc.nl`) with your '''public''' SSH key ([wiki:FgSshKey instructions]).
+
+For remote access from a Linux or Mac OSX terminal, type
+{{{
+ssh username@bios-vm.bbmrirp3-lumc.vm.surfsara.nl
+}}}
+where your private SSH key is in the standard location `~/.ssh/id_rsa` (alternatively, specify it with `-i`).
+
+For terminal access from Windows, use the [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] tool and configure the VM IP address, your username, and your private SSH key. 
+
+For graphical access from windows use [http://mobaxterm.mobatek.net/ MobaXterm] as advised by SURF in the HPC cloud documentation (https://doc.hpccloud.surfsara.nl/access-your-VM).
+
+Alternatively for graphical access from Windows or Mac OSX, use [http://wiki.x2go.org/doku.php X2Go] and configure the VM IP address, your username, your private SSH key and the session type/desktop manager (Gnome). Or, use a remote desktop connection client (for mac: http://www.microsoft.com/nl-nl/download/details.aspx?id=18140).
+
+[wiki:FgSshKey Step by step instructions for using a public/private SSH pair for access to the VM]
+
+[wiki:FgConnectTroubleshooting Connection troubleshooting page (under construction)]
+
+== Rstudio server ==
+There is a Rstudio server running on the BIOS VM: http://bios-vm.bbmrirp3-lumc.vm.surfsara.nl:8787
+
+You could log in using your username and password as your ssh session.
+
+== UCSC Genome Browser tracks ==
+
+Viewing RP3 data in the UCSC Genome Browser can be done by using the [wiki:FgStorageInTheCloud#VirDir WWW export directory on the virdir] and selecting the exported URLs as custom tracks.
+
+Please note that no privacy sensitive data should be stored here, as it will be world-readable.
+
+Example sessions:
+
+ * [http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=martijnvermaat&hgS_otherUserSessionName=rp3 Coverage tracks for 10 random samples, meta exon track, and the PolyA binning track]
+
+= Grid SRM access from the BIOS VM =
+In case you need to have access to some raw BIOS data (e.g., RNAseq, methylation), you will have to download them from the Grid SRM storage to the BIOS VM. Here are instructions on how to do it.
+
+'''Note:''' Requesting access to the SRM takes quite some time and using the SRM itself is not the easiest thing to learn. As such, if there already is someone at your institute with access and experience using the SRM, it might be faster and easier to ask that person for help.
+
+== Grid SRM Access ==
+
+Before proceeding, follow the steps in [wiki:FgObtainingGridAccess Obtaining access to grid infrastructure].
+
+=== Prepare a proxy ===
+
+To download data from the Grid SRM to the BIOS VM you'll need a proxy and your keys (the grid certificate). You should have access to a UI to start a proxy. For example gb-ui-lumc.lumc.nl, ui.lsg.psy.vu.nl or another site.
+
+* On the UI there should be a `.globus` folder in your home dir, that contains the grid certificate. Copy the `.globus` directory from your local home folder to the UI home folder. Make sure the permissions are set accordingly. (log into an UI and issue the commands `chmod 644 usercert.pem` and `chmod 400 userkey.pem`). These files don't need to be renewed. 
+  {{{
+  .globus/:
+  total 8
+  -rw-r--r-- 1 mgalen mgalen 1769 Aug 14 16:55 usercert.pem
+  -r-------- 1 mgalen mgalen 1751 Aug 14 16:55 userkey.pem
+  }}}
+
+* The proxy can be started by logging into an UI and use `startGridSession`:
+  {{{
+  startGridSession bbmri.nl:/bbmri.nl/RP3
+  }}}
+
+* This creates your own x509 in the `/tmp` dir on the UI which looks something like this.
+  {{{
+  -rw------- 1 mgalen   6.1K Aug 27 09:55 x509up_u40208
+  }}}
+* You may have to change the permissions of this file using `chmod 644 x509up_u40208`. Copy this file to a place at the BIOS VM for later use. (Maybe to `/tmp` also.) Make sure you copy the x509 file associated with your username. This is valid for 7 days, you need to renew this weekly. 
+
+* Now log in the BIOS VM and use this command to fetch the file you just created on the UI to the BIOS VM.
+  {{{
+  scp mgalen@uimd.grid.sara.nl:/tmp/x509up_u1234 /tmp    (replace 'uimd.grid.sara.nl'  with the address of your UI)
+  }}}
+
+=== Downloading files ===
+
+* Once these files are in place, you can copy data from the Grid SRM to the BIOS VM using curl. For example, login to the VM and issue the following command, where `-E` points to the path where you put the proxy file. Don't forget to redirect the output from curl to a local filename.
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/README >README
+  }}}
+
+* You can also upload data to the Grid SRM from the BIOS VM using curl. To upload a local file `test.txt`, use the `--upload-file` (or `-T`) argument:
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/test.txt --upload-file test.txt
+  }}}
+  (In practice, of course use a more appropriate directory on the Grid SRM instead of the project root.)
+  Instead of specifying the full target name including filename, you can also just specify the target directory ending in a `/`. Curl will than use your local filename also on the Grid SRM:
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/ --upload-file test.txt
+  }}}
+
+* If you want to delete a file from the Grid SRM, use `-X DELETE` (use this with caution):
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/test.txt -X DELETE
+  }}}
+
+* Just checking if a file exists, without really downloading it, can be done with the `-I` option (A response with `200 OK` means the file exists, `404 Not Found` means it doesn't):
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/README
+  HTTP/1.1 200 OK
+  Date: Tue, 28 Jan 2014 15:26:17 GMT
+  ETag: 0000EADAD57CE41D47F5A8F069A7C24F8003_-1773128220
+  Last-Modified: Wed, 15 Jan 2014 14:31:15 GMT
+  Content-Length: 154
+  Server: Jetty(7.3.1.v20110307)
+  }}}
+  {{{
+  mgalen@cloud-KVM:~$ curl --CApath /etc/grid-security/certificates/ -E /tmp/x509up_u40208 -L https://fly1.grid.sara.nl:2882/pnfs/grid.sara.nl/data/bbmri.nl/RP3/nonexisting
+  HTTP/1.1 404 Not Found
+  Content-Type: text/html
+  Transfer-Encoding: chunked
+  Server: Jetty(7.3.1.v20110307)
+  }}}
+
+
+== Data Storage ==
+
+Read [wiki:FgStorageInTheCloud Storage in the cloud]