Differences

This shows you the differences between two versions of the page.

--- hepsim:dev_hepsim [2017/02/07 02:46]
hepsim17
+++ hepsim:dev_hepsim [2020/06/26 21:24] (current)
hepsim17
@@ Line 3: / Line 3: @@
 [[:|<< back to HepSim manual]]
-====== Linking event storages======
+====== Linking event storage ======
-Your files can be  published on [[http://atlaswww.hep.anl.gov/hepsim/|HepSim]] and you  can maintain your own HepSim repository with PROMC/SLCIO/ROOT files. This is useful if:
+Monte Carlo files from any location can be  published on [[https://atlaswww.hep.anl.gov/hepsim/|HepSim]].
+HepSim is not a storage, but a catalog of files stored in multiple URL locations. It is your responsibility
+to maintain Monte Carlo data visible from HepSim.  HepSim tools allow:
-  * You want to share your own files, so they will be available from HepSim
+  * to index files so they will be available from HepSim;
-  * You want to make a  mirror of data on a server close to your location for faster download
+  * to make mirrors of data on a server close to your location for faster download;
 If the dataset is new, you will see a new entry in the HepSim database. Your authorship will be preserved on the HepSim web page.
 If you have made a mirror of the existing data, your server will be added as a mirror for a given dataset.
+If you created  Monte Carlo files, you can publish them using hepsim#petrel project, and link such files
+to HepSim after you index them as explained below. In this case you do not need to maintain  a web server.
 ====== What do you need ======
-You need the following:
+If you decide to publish Monte Carlo files (or make a mirror of the existing files), you need the following:
   * A web server that can hold your data (>1 TB for /var/www/html is recommended). It is advisable to create a separate partition with RAID10, say, /data, and link this directory to /var/www/html/.
   * Linux OS (any flavor)
   * Apache2 with PHP module that can serve/var/www/html/
   * Java 8 JDK or JRE (optionally, for tools to check data)
+You there is no web server, one can add files to the "petrel#hepsim" project using [[https://www.globus.org/|Globus]]. See the description later.
@@ Line 46: / Line 54: @@
 <code bash>
-wget http://atlaswww.hep.anl.gov/hepsim/doc/lib/exe/fetch.php?media=hepsim:hepsim_web.tgz -O hepsim_web.tgz
+wget https://atlaswww.hep.anl.gov/hepsim/doc/lib/exe/fetch.php?media=hepsim:hepsim_web.tgz -O hepsim_web.tgz
 tar -zvxf hepsim_web.tgz
 cd hepsim_web/web_post
-wget http://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz;
+wget https://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz;
 source hs-toolkit/setup.sh # this checks if Java is installed
 </code>
@@ Line 71: / Line 79: @@
   * copy existing files from the HepSim and put them to the correct directory (will be mirror)
-In both cases, make sure that Linux system administrator set correct permission for the directory "events", so you can copy the files. The directory for indexing should be inside your private directory outside the Web area.
+In both cases, make sure that Linux system administrator set the correct permission for the directory "events", so you can copy the files. The directory for indexing should be inside your private directory outside the Web area.
 If you have indexed the files successfully, send a request to "[email protected]" (or [email protected]) to include your repository to HepSim database. Please include a short description of your files and your name (it will be shown on the web page).
@@ Line 83: / Line 91: @@
 <code bash>
-wget http://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz;
+wget https://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz;
 source hs-toolkit/setup.sh # setup HepSim programs
 # now copy a dataset from URL to a new location
-SOURCE=http://cepcgit.ihep.ac.cn:81/hepsim/events/ee/250gev/pythia6_higgs_gamgam_test
+SOURCE=https://cepcgit.ihep.ac.cn:81/hepsim/events/ee/250gev/pythia6_higgs_gamgam_test
 OUTPUT_DIR=/var/www/html/
-hs-mirror -i $SOURCE-o $OUTPUT_DIR
+hs-mirror -i $SOURCE -o $OUTPUT_DIR
 </code>
 This example creates the directory "/var/www/html/events/ee/250gev/pythia6_higgs_gamgam_test" and copies all files from the
@@ Line 95: / Line 103: @@
 If you want to download only EVGEN files, without reconstructed events, use this command:
 <code bash>
-hs-mirror -i $SOURCE-o $OUTPUT_DIR -t evgen
+hs-mirror -i $SOURCE -o $OUTPUT_DIR -t evgen
 </code>
 Generally, you do not need to index files in the mirror directory.
+====== How to index directories  ======
+If you have made new files for HepSim (and added new file to the mirrored directory), you will need to index all files
+Assume your files are located inside the directory "events/ep/pythi8_dis". Then run "hs-index" script:
+<code bash>
+wget https://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz;
+source hs-toolkit/setup.sh # setup HepSim programs
+hs-index events/ep/pythi8_dis
+</code>
+If you need to re-index all subdirectories inside the directory "events", pass the upper directory to this script:
+<code bash>
+hs-index events
+</code>
+This script creates:
+  * "files.zip" - with the list of files
+  * "metadata.txt" - with some information from ProMC files
+  * "dirs.txt" - a list with subdirectories
+====== Using Petrel from Globus======
+Monte Carlo files can be added to the "petrel#hepsim" project (https://press3.mcs.anl.gov/petrel/) on the
+grid [[https://www.globus.org/|Globus]], so hey can be accessed via the grid and https for public view. In this case there is no need to run a private web server. The description of Petrel can be found in https://press3.mcs.anl.gov/petrel/documentation/.
+You will need a [[https://www.globus.org/|Globus]] account to be able to store files on Petrel.
+You can run your personal Petrel projects (with enabled https) with the files with HepSim-supported files,
+or you can add your files to the existing petrel#hepsim project.
+The directory with Monte Carlo files should be properly indexed as described above, and then you can copy such directory to Petrel Web-based storage using the transfer tools.
+A typical workflow is following:
+  * create directory with MC files on OSG or other resources
+  * index files using the tools described above
+  * Move the directory with the files to a petrel project (or ask to be included to petrel#hepsim)
+  * Send a URL location of your files to [email protected]
 ====== Summary ======
@@ Line 105: / Line 155: @@
 When everything is done, this is what you should expect:
-  * You files will be visible on the main HepSim web page and on the current mirrors (takes ~ few days to propagate the changes)
+  * Your files will be visible on the main HepSim web page and on the current mirrors (takes ~ few days to propagate the changes)
   * You can search your files etc. using the main HepSim web page
   * You can search, list and download files using hs-toolkit commands (hs-find, hs-ls, hs-get etc)