[[asc:workbook| <<back]]

====== Getting data to ANL from the grid ======

===== Using dq2-get command =====

For small data samples, use the dq2-get command. dq2-get are set automatically on atlas16-18 if one uses the setup described on the setup page. Read [[asc:workbook_introduction]] how to setup atlas software. 

One can also  setup the dq2-get client [[asc:workbook_introduction#using_cvmfs_file_system|using the CVFMS file system]]

The DQ2 client is described in [[https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2ClientsHowTo|this HowTo]] and [[https://twiki.cern.ch/twiki/bin/view/Atlas/WorkBookDQ2|this manual]] with further information [[http://twiki.mwt2.org/bin/view/DataServices/UsingDQ2AtUC|here]]


To setup dq2-get using CVFMS file system, use this script:

<code bash setup.sh>
#!/bin/bash
export ALRB_localConfigDir="/share/sl6/cvmfs/localConfig"
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
localSetupDQ2Client  --skipConfirm
voms-proxy-init -voms atlas -valid 96:00
</code>

Put it in a file "setup.sh" and then "source setup.sh".

Say "yes" and type your password.


The most often used dq2 commands are below dq2-xxx --help will show the options:

   * dq2-get -copies data from local storage or over the grid
   * dq2-ls -lists datasets matching a given pattern
   * dq2-put -allows the user to create user-defined datasets
   * dq2-ls -P -creates PoolFileCatalog.xml




In this example, we use  the standard (proxy for 3 days):
<code>
grid-proxy-init -valid  96:00 
</code>

Before start using dq2-get, check the contents of your Grid proxy with the command:

<code>
voms-proxy-info -all 
</code>

If there is a problem run these commands to debug them:

<code>
setupATLAS
diagnostics
gridCert
</code>


Then check the location of your data:

<code>
dq2-ls -r -f user.CCC.0108173.topmix_Egamma.AOD.v2
</code>

Then pull the data:

<code>
dq2-get -a  -s <Data location> <Data set name to copy>
</code>
Note: if looking for containers (a collection of data sets that end with a /) *dq2-ls* will not work. You need to do *dq2-list-datasets-container*.


====== Using Pathena commands ======

Do the setup as above then then type:

<code bash>
localSetupPandaClient
</code>

The general ATLAS instructions for pathena can be found [[https://twiki.cern.ch/twiki/bin/view/Atlas/DAonPanda|here]]. 

Also look at:
[[https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ATLASLocalRootBase#Using_PandaClient | Using_PandaClient]] how to setup Panda.
(attention: do not setup dq2-get together with panda!)


===== Putting files on the grid =====
(thanks to Taylor for checking)

Setup Atlas release. For SL6, source this code:

<code bash>
#!/bin/bash
export AVERS=17.7.0
export TEST_AREA=$HOME/testarea
export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
asetup --release=$AVERS --cmtconfig=x86_64-slc6-gcc46-opt --testarea=$TEST_AREA
localSetupDQ2Client  --skipConfirm
voms-proxy-init -voms atlas -valid 96:00
localSetupPandaClient --noAthenaCheck
</code>

Then put files (assumes MWT2_UC_SCRATCHDISK):

<code>
dq2-put -d -L MWT2_UC_SCRATCHDISK  -s ./<folder of files> user.<username>.<custom name>
</code>

 Create and register it before doing this:

You can also register it on the grid.
<code>
dq2-register-dataset user.<name>.test1
dq2-register-location user.<name>.test1  MWT2_UC_SCRATCHDISK
</code>

====== Replicate data on the grid ======


Go to: https://rucio-ui.cern.ch/

Use your X509 certificate go login

(On top click) Data Transfer (R2D2) > Request new rules > click “List of DIDS” > copy your dataset list to the box and click Search > in step 2 use "BNL-OSG2_SCRATCHDISK” or something that you have permission to write > in step 3 you can change lifetime, I suggest to put 365 > then continue and finish the procedure and it will tell you that your new rule is established or something similar.



===== Using subscription =====


Read section [[asc:workbook_grid_subscribe|using data subscription]].

 --- //[[chekanov@gmail.com|Sergei Chekanov]] 2013/03/18 09:21//