Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
hepsim:dev_hepsim [2017/05/23 17:51] hepsim17 [How to mirror entire dataset] |
hepsim:dev_hepsim [2020/06/26 21:24] hepsim17 |
||
---|---|---|---|
Line 3: | Line 3: | ||
[[: | [[: | ||
- | ====== Linking event storages====== | + | ====== Linking event storage |
- | Your files can be published on [[http:// | + | Monte Carlo files from any location |
+ | HepSim | ||
+ | to maintain Monte Carlo data visible from HepSim. | ||
- | * You want to share your own files, so they will be available from HepSim | + | * to index files so they will be available from HepSim; |
- | * You want to make a mirror | + | * to make mirrors |
If the dataset is new, you will see a new entry in the HepSim database. Your authorship will be preserved on the HepSim web page. | If the dataset is new, you will see a new entry in the HepSim database. Your authorship will be preserved on the HepSim web page. | ||
If you have made a mirror of the existing data, your server will be added as a mirror for a given dataset. | If you have made a mirror of the existing data, your server will be added as a mirror for a given dataset. | ||
+ | |||
+ | If you created | ||
+ | to HepSim after you index them as explained below. In this case you do not need to maintain | ||
====== What do you need ====== | ====== What do you need ====== | ||
- | You need the following: | + | If you decide to publish Monte Carlo files (or make a mirror of the existing files), you need the following: |
- | * A web server that can hold your data (>1 TB for / | + | * A web server that can hold your data (>1 TB for / |
* Linux OS (any flavor) | * Linux OS (any flavor) | ||
* Apache2 with PHP module that can serve/ | * Apache2 with PHP module that can serve/ | ||
* Java 8 JDK or JRE (optionally, | * Java 8 JDK or JRE (optionally, | ||
+ | |||
+ | |||
+ | You there is no web server, one can add files to the " | ||
Line 46: | Line 54: | ||
<code bash> | <code bash> | ||
- | wget http:// | + | wget https:// |
tar -zvxf hepsim_web.tgz | tar -zvxf hepsim_web.tgz | ||
cd hepsim_web/ | cd hepsim_web/ | ||
- | wget http:// | + | wget https:// |
source hs-toolkit/ | source hs-toolkit/ | ||
</ | </ | ||
Line 71: | Line 79: | ||
* copy existing files from the HepSim and put them to the correct directory (will be mirror) | * copy existing files from the HepSim and put them to the correct directory (will be mirror) | ||
- | In both cases, make sure that Linux system administrator set correct permission for the directory " | + | In both cases, make sure that Linux system administrator set the correct permission for the directory " |
If you have indexed the files successfully, | If you have indexed the files successfully, | ||
Line 83: | Line 91: | ||
<code bash> | <code bash> | ||
- | wget http:// | + | wget https:// |
source hs-toolkit/ | source hs-toolkit/ | ||
# now copy a dataset from URL to a new location | # now copy a dataset from URL to a new location | ||
- | SOURCE=http:// | + | SOURCE=https:// |
OUTPUT_DIR=/ | OUTPUT_DIR=/ | ||
hs-mirror -i $SOURCE -o $OUTPUT_DIR | hs-mirror -i $SOURCE -o $OUTPUT_DIR | ||
Line 100: | Line 108: | ||
Generally, you do not need to index files in the mirror directory. | Generally, you do not need to index files in the mirror directory. | ||
+ | |||
+ | ====== How to index directories | ||
+ | |||
+ | If you have made new files for HepSim (and added new file to the mirrored directory), you will need to index all files | ||
+ | Assume your files are located inside the directory " | ||
+ | |||
+ | <code bash> | ||
+ | wget https:// | ||
+ | source hs-toolkit/ | ||
+ | hs-index events/ | ||
+ | </ | ||
+ | |||
+ | If you need to re-index all subdirectories inside the directory " | ||
+ | |||
+ | <code bash> | ||
+ | hs-index events | ||
+ | </ | ||
+ | |||
+ | This script creates: | ||
+ | |||
+ | * " | ||
+ | * " | ||
+ | * " | ||
+ | |||
+ | |||
+ | ====== Using Petrel from Globus====== | ||
+ | |||
+ | Monte Carlo files can be added to the " | ||
+ | grid [[https:// | ||
+ | You will need a [[https:// | ||
+ | |||
+ | You can run your personal Petrel projects (with enabled https) with the files with HepSim-supported files, | ||
+ | or you can add your files to the existing petrel# | ||
+ | The directory with Monte Carlo files should be properly indexed as described above, and then you can copy such directory to Petrel Web-based storage using the transfer tools. | ||
+ | |||
+ | A typical workflow is following: | ||
+ | |||
+ | * create directory with MC files on OSG or other resources | ||
+ | * index files using the tools described above | ||
+ | * Move the directory with the files to a petrel project (or ask to be included to petrel# | ||
+ | * Send a URL location of your files to [email protected] | ||
+ | |||
====== Summary ====== | ====== Summary ====== | ||
Line 105: | Line 155: | ||
When everything is done, this is what you should expect: | When everything is done, this is what you should expect: | ||
- | * You files will be visible on the main HepSim web page and on the current mirrors (takes ~ few days to propagate the changes) | + | * Your files will be visible on the main HepSim web page and on the current mirrors (takes ~ few days to propagate the changes) |
* You can search your files etc. using the main HepSim web page | * You can search your files etc. using the main HepSim web page | ||
* You can search, list and download files using hs-toolkit commands (hs-find, hs-ls, hs-get etc) | * You can search, list and download files using hs-toolkit commands (hs-find, hs-ls, hs-get etc) |