Changes between Version 15 and Version 16 of DataManagement


Ignore:
Timestamp:
Aug 19, 2011 5:19:47 PM (13 years ago)
Author:
laurent
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • DataManagement

    v15 v16  
    11This page is work-in-progress regarding the data Management of the GoNL.
    22
    3 See
    4 * DataManagement/ProjectData - how are rawdata, intermediate data and result data organized on the compute clusters
    5 * DataManagement/SftpServer - how to access the SFTP for data sharing
    6 * DataManagement/ProjectResources - where resources and tools that are used by the pipelines
    7 * DataManagement/FileNameConventions - how are files named so we understand eachother
     3'''Important note''': The block size on the storage is 6MB, which means that each file  -regardless of its real size- will occupy at least 6MB on the file  system. This means that data should rather be kept in big files rather  than a multitude of small files whenever possible. Typically things like  logs, old submit scripts, etc. should be compressed into 1 file for  archiving.
     4
     5See
     6
     7 * DataManagement/ProjectData - how are rawdata, intermediate data and result data organized on the compute clusters
     8 * DataManagement/SftpServer - how to access the SFTP for data sharing
     9 * DataManagement/ProjectResources - where resources and tools that are used by the pipelines
     10 * DataManagement/FileNameConventions - how are files named so we understand eachother