Differences

This shows you the differences between two versions of the page.

@@ Line 1: / Line 1: @@
+{{indexmenu_n>2}}
+[[:|<< back to HepSim manual]]
+======  HepSim data files ======
+======  EVGEN events ======
+Truth-level (EVGEN) data are stored in a platform-independent format called [[https://atlaswww.hep.anl.gov/asc/promc/ | ProMC]] that allows very effective compression using a variable-byte encoding.  This data format is
+supported by popular programming languages (C++, Java, Python) on major  operating system (Windows, Mac, Linux, etc.).
+Open access to such files via the http protocol is the central element  in the design of HepSim, since
+further processing using fast and full simulations is done using computer resources from multiple locations.
+The compact ProMC files optimized for web streaming, together with the http protocol optimized
+to handle many (relatively small) files, is one of the distinct features of HepSim compared to other production systems.
+Simulated data after fast and full detector simulations are kept in ROOT and LCIO  formats.
+======  HepSim software toolkit ======
+In order to work with HepSim database, install the HepSim software toolkit. For Linux/Mac with the bash shell, use:
+<code bash>
+wget https://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz -O - | tar -xz
+source hs-toolkit/setup.sh
+</code>
+or using curl:
+<code bash>
+curl https://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz | tar -xz
+source hs-toolkit/setup.sh
+</code>
+This creates the directory "hs-toolkit" with HepSim commands. To run them,
+[[https://www.java.com/en/download/|Java7/8]] and above should be installed.
+One can also download  [[http://atlaswww.hep.anl.gov/hepsim/soft/hs-toolkit.tgz|hs-toolkit.tgz]].
+Most commands are platform-independent and based on the
+[[http://atlaswww.hep.anl.gov/hepsim/hepsim.jar|hepsim.jar]] that can be used on Windows/Mac.
+The setup command  adds all commands to the execution path.
+You can view the available commands from this package by typing:
+<code bash>
+hs-help
+</code>
+<note tip>The programs included in **hs-toolkit** can be used for searching and downloading ProMC files, and other files after fast and full simulations.
+If you need to analyse and display reconstructed events in the LCIO format, use the [[https://atlaswww.hep.anl.gov/asc/jas4pp|Jas4PP program]] which includes **hs-toolkit**, but also adds packages for processing SLCIO files. This program can also process ProMC files.
+</note>
+======  Event record browser ======
+Use the event-record browser to view separate EVGEN events, cross sections and log files. On Linux/Mac, use:
+<code bash>
+hs-view https://mc.hep.anl.gov/asc/hepsim/events/pp/14tev/mg5_httbar/tev14_mg5_Httbar_001.promc
+</code>
+Here we have looked at one file of the [[http://atlaswww.hep.anl.gov/hepsim/info.php?item=141 | Pythia8 (QCD) sample]].
+Of course, one can look at the local file as well:
+<code bash>
+wget https://mc.hep.anl.gov/asc/hepsim/events/pp/14tev/mg5_httbar/tev14_mg5_Httbar_001.promc
+hs-view tev14_mg5_Httbar_001.promc
+</code>
+after you downloaded it.
+On Windows, download [[https://atlaswww.hep.anl.gov/asc/hepsim/hepsim.jar| hepsim.jar]] and click the "hepsim.jar" file. Then  open the ProMC file using the "File" menu.
+You will see a pop-up GUI browser which displays the MC record. You can search for a given particle name, view data layouts and log files using the [Menu]:
+<hidden>
+{{:hepsim:promc_browser.png| ProMC browser}}
+</hidden>
+This works for full parton-shower simulations with detailed information on particles.
+Unlike the usual parton shower Monte Carlo, this  browser has a detailed information on event weights, PDF uncertainties and scale uncertainties (in some cases). The browser can show 4-momenta of each event as well as the total cross sections (for NLO, you need to read all events to get an accurate cross section). Look at the [[https://atlaswww.hep.anl.gov/asc/promc/| ProMC file]] description.
+======  File validation ======
+Check the consistency of the EVGEN file and additional information as:
+<code bash>
+hs-info https://mc.hep.anl.gov/asc/hepsim/events/pp/14tev/mg5_httbar/tev14_mg5_Httbar_001.promc
+</code>
+The last argument can be a file path on the local disk (faster than URL!). The output of the above command is:
+<code>
+File          = http://mc.hep.anl.gov/asc/hepsim/events/pp/14tev/mg5_httbar/tev14_mg5_Httbar_001.promc
+ProMC version = 4
+Last modified = 2015-10-03 12:06:52
+Description   = run_Httbar_14tev
+Events        = 10000
+Sigma    (pb) = 5.61176E-1 ± 3.3035E-3
+Lumi   (pb-1) = 1.78197E4
+Varint units  = E:100000 L:1000
+Log file:     = logfile.txt
+####  The file is healthy!  ####
+</code>
+All entries are self-explanatory. Varint units - values used to multiply energy (momenta) to convert to variable-byte integers.
+The "E:100000" means that all px, py, pz, e, mass values are multiplied by 100000, while all distances (x,y,z,t) are multiplied by  1000.
+See the [[https://atlaswww.hep.anl.gov/asc/promc/ | ProMC archive format]].
+One can look at separate events using the above command after passing an integer argument that specifies
+the event to be looked at. This command prints the event number 100:
+<code bash>
+hs-info https://mc.hep.anl.gov/asc/hepsim/events/pp/14tev/mg5_httbar/tev14_mg5_Httbar_001.promc 100
+</code>
+====== List available data files ======
+Let us show how to find all files associated with a given Monte Carlo event sample.
+Go to [[https://atlaswww.hep.anl.gov/hepsim/ | HepSim database]]. Look at the links "Files". It list the available files.
+Then find the files as:
+<code bash>
+      hs-ls [name]
+</code>
+where [name] is the dataset name, or the URL of the Info page, or the URL of the location of all files.
+This command shows a table with file names and their sizes.
+Here is an example illustrating how to list all files from the [[https://atlaswww.hep.anl.gov/hepsim/info.php?item=2|Higgs to ttbar]]
+Monte Carlo sample:
+<code bash>
+hs-ls tev100pp_higgs_ttbar_mg5
+</code>
+If you want to create a simple list without decoration to make a file to be read by other program, use
+<code bash>
+hs-ls tev100pp_higgs_ttbar_mg5 simple
+</code>
+The string "simple" removes the decorations. If you want a list with full URL and without decorations, use:
+<code bash>
+hs-ls tev100pp_higgs_ttbar_mg5 simple-url
+</code>
+Similarly, one can use the Info or Download URL path:
+<code bash>
+hs-ls https://atlaswww.hep.anl.gov/hepsim/info.php?item=2
+</code>
+or
+<code bash>
+hs-ls https://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/higgs_ttbar_mg5/
+</code>
+The commands show this table:
+<hidden>
+<code>
+Looking for http://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/higgs_ttbar_mg5
+File name                size (kB)
+------------------------------------
+mg5_Httbar_100tev_001.promc 24464
+mg5_Httbar_100tev_002.promc 24360
+mg5_Httbar_100tev_003.promc 24488
+mg5_Httbar_100tev_004.promc 24516
+mg5_Httbar_100tev_005.promc 24448
+mg5_Httbar_100tev_006.promc 24484
+mg5_Httbar_100tev_007.promc 24468
+mg5_Httbar_100tev_008.promc 24516
+mg5_Httbar_100tev_009.promc 24440
+mg5_Httbar_100tev_010.promc 24592
+mg5_Httbar_100tev_011.promc 24456
+mg5_Httbar_100tev_012.promc 24568
+mg5_Httbar_100tev_013.promc 24436
+mg5_Httbar_100tev_014.promc 24416
+mg5_Httbar_100tev_015.promc 24408
+mg5_Httbar_100tev_016.promc 24484
+mg5_Httbar_100tev_017.promc 24400
+mg5_Httbar_100tev_018.promc 24460
+mg5_Httbar_100tev_019.promc 24452
+mg5_Httbar_100tev_020.promc 24420
+------------------------------------
+-> Summary: Nr of files=20  Total size=489 MB
+</code>
+</hidden>
+====== Searching for a dataset  ======
+One can find a URL that corresponds to a dataset using a search word. The syntax is:
+<code bash>
+hs-find [search word]
+</code>
+For example, this commands list all datasets that have the keyword  "higgs" in the name of the dataset,
+in the name of Monte Carlo models, or in the file description:
+<code bash>
+hs-find higgs
+</code>
+Note that there is no need to use any character such as "*" or "%" as for the usual regular expression.
+The match is done using any dataset that has the sub-string "higgs".
+The same functionality is implemented in the "Search" menu of the HepSim interface.
+If you are interested in a specific reconstruction tag, use "%" to separate the search string and the tag name.
+Example:
+<code bash>
+hs-find pythia%rfast001
+</code>
+It will search for Pythia samples after a fast detector simulation with the tag "001". To search for a full detector simulation, replace
+"rfast" with "rfull".
+====== Downloading truth-level files ======
+One can download all files for a given dataset as:
+<code bash>
+hs-get [name] [OUTPUT_DIR]
+</code>
+where [name] is either the name of the dataset, or the URL of Info page [[https://atlaswww.hep.anl.gov/hepsim/ | HepSim repository]], or a direct URL pointing to the locations of ProMC files.
+This example downloads dataset "tev100pp_higgs_ttbar_mg5" to the directory "data":
+<code bash>
+hs-get tev100pp_higgs_ttbar_mg5 data
+</code>
+You will be prompted to use certain mirror (if there are alternative mirrors). Select the mirror and start downloading the files.
+Alternatively, this example downloads files using the URL of the Info page:
+<code bash>
+hs-get https://atlaswww.hep.anl.gov/hepsim/info.php?item=2 data
+</code>
+Or, if you know the download URL with the file locations, use this command:
+<code bash>
+hs-get https://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/higgs_ttbar_mg5 data
+</code>
+All these examples will download all files from the "tev100pp_higgs_ttbar_mg5" event sample.
+<note important>If you see that the download is slow, use an alternative URL from the mirror list which is given for each dataset.
+</note>
+You can stop downloading using [Ctrt-D]. Next time you start it, it will continue the download.
+By default, we use 2 threads. One  can increase the number of threads by adding an integer number to the end of this command.
+If a second integer is given, it will be used to set the maximum number of files to download.
+This example shows how to download 10 files using 3  threads:
+<code bash>
+hs-get https://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/higgs_ttbar_mg5 higgs_ttbar_mg5 3 10
+</code>
+Instead of [Download URL], one can use the URL of the info page, or the name of the dataset.
+Here are 2 identical examples to download 5 files using single (1) thread and the output directory "data":
+Using the URL of the info page:
+<code bash>
+hs-get https://atlaswww.hep.anl.gov/hepsim/info.php?item=2  data 1 5
+</code>
+or, when using the dataset name given on the info page:
+<code bash>
+hs-get tev100pp_higgs_ttbar_mg5  data 1 5
+</code>
+You can also download files that have certain pattern in the names. If a directory contains files generated with different pt cuts,
+the names are usually have the sub-string "pt", followed by the pT cuts. In this case, one can download such files as:
+<code bash>
+hs-get tev13pp_higgs_pythia8_ptbins data 2 5 pt100_
+</code>
+The last argument shows that all the downloaded files should have the string "pt100_" in their names (in this case, it tells that the
+file are generated with pT>100 GeV).
+The general usage of the hs-get command requires 2, 3, 4 or 5 arguments:
+<code bash>
+hs-get [URL] [OUTPUT_DIR] [Nr of threads (optional)] [Nr of files (optional)] [pattern (optional)].
+</code>
+where [URL] is either info URL, [Download URL], or the dataset name.
+It is not recommended to use more than 6 threads for downloads.
+====== Downloading files after detector simulation ======
+Some datasets contain reconstructed files after detector simulations.
+Reconstructed files are stored inside the directory "rfastNNN" (fast simulation) or "rfullNNN" (full simulation),
+where "NNN" is the version number. For example,
+[[https://atlaswww.hep.anl.gov/hepsim/info.php?item=15|tev100pp_ttbar_mg5]] sample includes the link "rfast001" (Delphes
+fast simulation, version 001). To download the reconstructed events for the reconstruction tag "rfast001", use this syntax:
+<code bash>
+hs-ls  tev100pp_ttbar_mg5%rfast001      # list reco files with the tag "rfast001"
+hs-get tev100pp_ttbar_mg5%rfast001 data # download to the "data" directory
+</code>
+The symbol "%" separates the sample name (tev100pp_ttbar_mg5) from the reconstruction tag (rfast001).
+If you want to download 10 files in 3 threads, use this:
+<code>
+hs-get  tev100pp_ttbar_mg5%rfast001 data 3 10
+</code>
+As before, one can also download the files using the URL approach:
+<code bash>
+hs-ls https://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/ttbar_mg5/rfast001/ # list all files
+hs-get https://mc.hep.anl.gov/asc/hepsim/events/pp/100tev/ttbar_mg5/rfast001/ data
+</code>
+Note that the reconstruction tag "rfast001" is separated by backslash as for the usual directory.
+As before, the URL can be taken from the list with mirrors. For example, if you want to download  files from Nersc, use this URL:
+<code bash>
+hs-ls  http://portal.nersc.gov/project/m1758/data/events/pp/100tev/ttbar_mg5/rfast001/ # list all files
+hs-get http://portal.nersc.gov/project/m1758/data/events/pp/100tev/ttbar_mg5/rfast001/ data
+</code>
+Written by  //[[[email protected]|Sergei Chekanov (ANL)]] 2016/02/08 10:26//