ATLAS Support Center

User Tools

Site Tools

Trace:
community:hepsim:dev_hepsim

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
community:hepsim:dev_hepsim [2016/10/25 22:00]
asc 		community:hepsim:dev_hepsim [2016/12/22 15:53]
asc [Summary]
Line 3: Line 3:
 ====== How to link private samples to HepSim ====== ====== How to link private samples to HepSim ======
  
-Your files can be  published on HepSim and you  can maintain your own Hepim repository with PROMC/SLCIO/ROOT files. This is useful if:+Your files can be  published on [[http://atlaswww.hep.anl.gov/hepsim/|HepSim]] and you  can maintain your own HepSim repository with PROMC/SLCIO/ROOT files. This is useful if:
  
   * You want to share your own files, so they will be available from HepSim   * You want to share your own files, so they will be available from HepSim
-  * You want to copy your dataset to mirros in other places +  * You want to make a  mirror of data on a server close to your location for faster download
-  * You want to make a copy of certain HepSim datasets since the network connection to the existing mirrors is slow +
  
-If the dataset is new, you will see a new entry in the HepSim database. 
-If you have made a mirror, your server will be added as a mirror for a given dataset. 
  
-Your authorship will be preserved inside HepSim.+If the dataset is new, you will see a new entry in the HepSim database. Your authorship will be preserved on the HepSim web page. 
 +If you have made a mirror of the existing data, your server will be added as a mirror for a given dataset. 
  
 ====== What do you need ====== ====== What do you need ======
Line 18: Line 17:
 You need the following: You need the following:
  
-  * A web server that can hold your data (>1 TB is recommended)+  * A web server that can hold your data (>1 TB for /var/www/html is recommended). It is advisable to create a separate partition with RAID10, say, /data, and link this directory to /var/www/html/.
   * Linux OS (any flavor)   * Linux OS (any flavor)
-  * Installed Apache2 with PHP module and Java+  * Apache2 with PHP module that can serve/var/www/html/ 
 +  * Java 8 JDK or JRE (optionally, for tools to check data)
  
  
-The root directory of a typical HepSim repository is "events". This directory should  +The root directory of a typical HepSim repository has the name  "events". This directory should  
-be served by Apache. You will need to make the directory structure such as:+be served by Apache, i.e. it should be visible from /var/www/html. You will need to make the directory structure such as:
  
-<code> +<code bash> 
-  |-events +  |-events               # HepSim root directory 
-  |  |-pp +  |  |-pp                # process type: pp, ee, ep, misc (single particles) 
-  |  |  |-14tev +  |  |  |-14tev          # CM energy. Can be 100tev, 250gev, 500gev 
-  |  |  |  |-qcd_pythia8 +  |  |  |  |-qcd_pythia8 # this directory contains ProMC files for EVGEN 
-  |  |  |  |  |-rfast001 +  |  |  |  |  |-rfast001 # subdirectory with ROOT/Delphes files 
-  |  |  |  |  |-rfull001 +  |  |  |  |  |-rfull001 # subdirectory with SLCIO files 
-  |  |  |  |  |-macros+  |  |  |  |  |-macros   # files for truth validation (optional)
 </code> </code>
-(it shows a data sample with the name "qcd_pythia8"). +(it shows a data sample for 14 TeV pp with the name "qcd_pythia8"). 
-The directory "events/pp/14tev/qcd_pythia8" should contain truth-level ProMC files. The subdirectories "rfast001" contains fast simulation files (tag "rfast001"), while "rfull001" should contain LCIO files with full simulation. The directory "macros" contains macro files written in Jython for validation, as well as images. +The directory "events/pp/14tev/qcd_pythia8" should contain truth-level ProMC files. The sub-directories "rfast001" contains fast simulation files (tag "rfast001"), while "rfull001" should contain LCIO files with full simulation. The directory "macros" contains macro files written in Jython for validation, as well as images (optional).
  
  
-After populating the directory "events/pp/14tev/qcd_pythia8" with ProMC files, you will need to index all files. Install the needed packages. Download the example from {{:community:hepsim:hepsim_web.tgz|hepsim_web.tgz}}+After populating the directory "events/pp/14tev/qcd_pythia8" with ProMC files, you will need to index all files (including reconstruction tags). Install the needed packages as explained below.  
 +Download the example from {{:community:hepsim:hepsim_web.tgz|hepsim_web.tgz}}
 Assuming that you are in the directory above the directory "events", run these commands:  Assuming that you are in the directory above the directory "events", run these commands: 
  
Line 54: Line 55:
   * ** web_post** directory should be outside the Apache area. This directory is used to index your files in the directory "events"    * ** web_post** directory should be outside the Apache area. This directory is used to index your files in the directory "events" 
  
-This package has 2 example files in "events/ee/250gev/pythia6_higgs_gamgam" to illustrate how indexing works.+This package has 2 example files in "events/ee/250gev/pythia6_higgs_gamgam" to illustrate how the indexing works.
  
  
-Now, index the files located in "events".  Open the script "A_RUN_hepsim.sh" and specify the location of the "events" directory.  +Now, index all ProMC/ROOT/SLCIO  files located in "events".  Open the script "A_RUN_hepsim.sh" and specify the location of the "events" directory.  
-Then run "bash A_RUN_hepsim.sh".+Then run "bash ./A_RUN_hepsim.sh".
  It will process the directory "events/". You can edit the script to change the indexed directory to ""events/pp/100tev/". After indexing, several new files should appear, such as "files.zip", "metadata.txt" and "dirs.idx".  It will process the directory "events/". You can edit the script to change the indexed directory to ""events/pp/100tev/". After indexing, several new files should appear, such as "files.zip", "metadata.txt" and "dirs.idx".
  
-To make sure that everything works, look with the browser in the HTTP link "http://yourserver/events/". You should see your files and the directory structure. Note that what you see is not what you will see in HepSim, which does not use the "index.php" file.+To make sure that everything works, look with the browser in the HTTP link "http://yourserver/events/". You should see your files and the directory structure. Note that what you see is not what you  see in HepSim, which does not use the "index.php" file.
  
 Now you can: Now you can:
  
-  * populate the directory tree with your  files (for a new dataset)+  * populate the directory tree with your  files (for a new dataset). ProMC is used for EVGEN files, abd SLCIO/ROOT files for directories "rfullXXX/rfastXXX", where XXX is a tag number.
   * copy existing files from the HepSim and put them to the correct directory (will be mirror)   * copy existing files from the HepSim and put them to the correct directory (will be mirror)
  
 In both cases, make sure that Linux system administrator set correct permission for the directory "events", so you can copy the files. The directory for indexing should be inside your private directory outside the Web area. In both cases, make sure that Linux system administrator set correct permission for the directory "events", so you can copy the files. The directory for indexing should be inside your private directory outside the Web area.
  
-If you have indexed the files successfully, send a request to "[email protected]" (or [email protected]) to include your repository to HepSim database. Please include a short description of your files and your name.+If you have indexed the files successfully, send a request to "[email protected]" (or [email protected]) to include your repository to HepSim database. Please include a short description of your files and your name (it will be shown on the web page). 
 + 
 + 
 +====== How to mirror entire dataset ====== 
 + 
 +You can mirror an entire dataset by copying HepSim files to your mirror web server (or local computer). 
 +You can use the command "hs-mirror" from the "hs-toolkit" package. For example, if you want 
 +to mirror a dataset with known URL, use this example: 
 + 
 +<code bash> 
 +wget http://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz; 
 +source hs-toolkit/setup.sh # setup HepSim programs 
 +# now copy a dataset from URL to a new location 
 +SOURCE=http://cepcgit.ihep.ac.cn:81/hepsim/events/ee/250gev/pythia6_higgs_gamgam_test 
 +OUTPUT_DIR=/var/www/html/ 
 +hs-mirror -i $SOURCE-o $OUTPUT_DIR 
 +</code> 
 +This example creates the directory "/var/www/html/events/ee/250gev/pythia6_higgs_gamgam_test" and copies all files from the 
 +URL. The URL can be found using the HepSim web page.  
 + 
 +If you want to download only EVGEN files, without reconstructed events, use this command: 
 +<code bash> 
 +hs-mirror -i $SOURCE-o $OUTPUT_DIR -t evgen 
 +</code> 
 + 
 + 
 +Generally, you do not need to index files in the mirror directory.  
 + 
 +====== Summary ====== 
 + 
 +When everything is done, this is what you should expect:
  
 +  * You files will be visible on the main HepSim web page and on the current mirrors (takes ~ few days to propagate the changes)
 +  * You can search your files etc. using the main HepSim web page
 +  * You can search, list and download files using hs-toolkit commands (hs-find, hs-ls, hs-get etc)
 +  * Since your files are public, there is good chance that somebody will make a mirror to reduce the load on your server
 +  * You server will be listed on the page [[http://atlaswww.hep.anl.gov/hepsim/summary.php|statistics summary]]
  
  
  
  --- //[[[email protected]|Sergei Chekanov]] 2016/04/28 21:31//  --- //[[[email protected]|Sergei Chekanov]] 2016/04/28 21:31//
 + --- //[[[email protected]|Sergei Chekanov]] 2016/10/24 21:31//
community/hepsim/dev_hepsim.txt · Last modified: 2016/12/22 15:53 by asc

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
Public Domain Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki