User Tools
Sidebar
asc:workbook_data
Where to put data
There are several places to store the data. This depends on the size of your data (D3PD/NTUP/AOU/ESD):
- Small size (<5 GB) - you may copy it to you home directory (under /users/[yourname])
- Medium size (<50 GB) - you may copy it data directory on each server or desktop (under /data1/[yourname])
- Large size data (above 50 GB). Usually, we copy data from the grid to the atlas4 file server
Please do not keep data on NFS where your home directory is (/users/). There is a significant performance penalty when running your jobs, plus it is impossible to backup your data
To run over a large data sample, we need to copy data on the computer farm and then you can process the data using Arcond/Condor. Read here more. Please contact S.Chekanov if you need to copy data on servers of the computer farm.
Usually (but not always), we keep two copies of data: one on the central storage and second version distributed (staged) on the farm. Typically, you need about 1 day to process 1 TB of data. On the computer farm, it should take less than 30 min.
How to get the data from the grid is described here. You can use use either dq2-get or subscription Using data subscription.
asc/workbook_data.txt · Last modified: 2013/05/30 18:53 (external edit)
Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
