User Tools

Site Tools


asc:workbook_data

<<back

Where to put data

There are several places to store the data. This depends on the size of your data (D3PD/NTUP/AOU/ESD):

  • Small size (<5 GB) - you may copy it to you home directory (under /users/[yourname])
  • Medium size (<50 GB) - you may copy it data directory on each server or desktop (under /data1/[yourname])
  • Large size data (above 50 GB). Usually, we copy data from the grid to the atlas4 file server
Please do not keep data on NFS where your home directory is (/users/). There is a significant performance penalty when running your jobs, plus it is impossible to backup your data

To run over a large data sample, we need to copy data on the computer farm and then you can process the data using Arcond/Condor. Read here more. Please contact S.Chekanov if you need to copy data on servers of the computer farm.

Usually (but not always), we keep two copies of data: one on the central storage and second version distributed (staged) on the farm. Typically, you need about 1 day to process 1 TB of data. On the computer farm, it should take less than 30 min.

How to get the data from the grid is described here. You can use use either dq2-get or subscription Using data subscription.

asc/workbook_data.txt · Last modified: 2013/05/30 18:53 by 127.0.0.1