====== Using singularity ======
===== On a desktop/laptop =====
Singularity can serve to provide a portable complete environment for EIC studies.  Due to the ubiquity of Docker, a convenient approach is to maintain a Docker image which can be used to generate Singularity images.  The generated Singularity image can then be used on the end user's computer and transported to machines with greater resources when needed.

As an example, let's use an established Docker image.  First, Singularity 2.3 or newer must be installed on your system (see [[http://singularity.lbl.gov]]).  Then, we will create an image from an image hosted on Docker Hub.  **WARNING:** this will take ~10 GB!  The image can be readily deleted when it is no longer needed.
<code bash>
singularity pull docker://argonneeic/fpadsim:v1.4
</code>

The file "fpadsim-v1.4.img" now contains an complete environment necessary for simulating and reconstructing EIC events.  In order to use the image, we have a few options.  The most direct way to get into a fully set up environment is
<code bash>
singularity exec fpadsim-v1.4.img bash -l
</code>
.  Barring any errors, you are now running inside a container.  It is a container configured to be quite transparent, and you should be able to see all the files within your home directory as well as the processes that are running outside of the container when you run "top".  This is in contrast to the default configuration of a Docker container, where isolation is desired.  While your home directory is mounted within the container, the system files have been replaced by the contents of the container.  Go ahead and try commands like "slic", "promc_browser", "jaspp", etc.

====== Useful commands ======

Download the image, and look inside. 
<code>
sudo singularity shell --writable --contain  fpadsim-v1.4.img
</code>
Now you can add files.

===== On the Open Science Grid (OSG) =====
While documentation for using Singularity images on the OSG can be found [[https://support.opensciencegrid.org/support/solutions/articles/12000024676-singularity-containers|here]], this section will discuss the specific use of the same Docker image as used above.

This particular Docker image (and indeed any image providing a complete chain of tools for GEANT4 simulation, reconstruction, and analysis of EIC events) is fairly large (~5GB uncompressed).  To facilitate the distribution of large images, the OSG can auto-generate Singularity images from a list of Docker repositories for distribution over [[https://cernvm.cern.ch/portal/filesystem|cvmfs]].  That list is maintained [[https://github.com/opensciencegrid/cvmfs-singularity-sync/blob/master/docker_images.txt|here]].

First, ssh access to OSG Connect must be established.  On the OSG Connect login server, you should see all of the available Singularity images in an uncompressed form mounted at <code>/cvmfs/singularity.opensciencegrid.org</code>.  To run the same image as in the above section, simply run the following line.
<code>
singularity exec /cvmfs/singularity.opensciencegrid.org/argonneeic/fpadsim\:v1.4/ bash -l
</code>

For submitting Condor jobs, it's necessary to require that the target machine as Singularity installed by adding <code>HAS_SINGULARITY == TRUE</code> to your "Requirements" in your Condor submit file.  The optional (but recommended) way of loading the required image on the target machine is by adding the following lines to your Condor submit file,
<code>
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/argonneeic/fpadsim:v1.4"
+SingularityBindCVMFS = False
</code>,
Where SingularityBindCVMFS can be changed to True if the cvmfs mount is required inside the container.  With these options added, the use of the Singularity image by your OSG job wrapper script is fairly seamless, and the job executable will be run inside of the image via the default shell, /bin/sh.  For this particular image, the default shell will not load the environment setup scripts, so it's usually necessary to run commands inside of a bash login shell by either using the shebang
<code bash>
#!/bin/bash -l
</code>
, or by wrapping commands like
<code bash>
bash -lc "..."
</code>
.  This environment can be tested on the login server by running
<code>
singularity shell /cvmfs/singularity.opensciencegrid.org/argonneeic/fpadsim\:flat/
</code>.

If more control over loading the image is needed, then the above options can be omitted and replaced by
<code>
+SingularityAutoLoad = False
</code>.
With this option, the target machine will avoid loading even the default Singularity image so that one may set the specific options needed when loading the image in, e.g., a job wrapper script.