Using UTEP Jakar

This is a draft of how to compile the DFTSW code and use it on UTEP Jakar. A proper write-up article will follow.

1. Log in with: ssh username@jakar.utep.edu

2. Download the source code to your home directory with: git clone git@quantum.utep.edu:implicit_none_blockmesh.git

You may see o_home directory. This contains your old hpc files. This is there for backup purposes, and you should avoid using it for running calculations.

3. Edit makefile: vim Makefile

In your makefile, prepend the library path manually to make dftsw compile on jakar. This may be done in bash_profile.
module load gnu8 openmpi3 scalapack
LOPTIONS = -L/path_to_liblapack.a/ -L/opt/ohpc/pub/libs/gnu8/openblas/0.3.7/lib/ -llapack -lopenblas
LOPTIONS += -L/opt/ohpc/pub/libs/gnu8/openmpi3/scalapack/2.0.2/lib -lscalapack

Jakar does not have a working lapack. But you can compile it in your home directory. You only need to do this once.

wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.0.tar.gz
tar zxvf v3.10.0.tar.gz
cd lapack-3.10.0/
cp make.inc.example make.inc
make
*this make command will complain about lapack_testing.py but you can ignore it since you have liblapack.a at that point.
Copy the liblapack.a in your /home/username/lib and use that path as the path_to_liblapack.a

4. If everything is set up correctly in the Makefile, you can hit: make

5. Now you have a dftsw_exe binary file.

6. partition info. (Note: do not use the phys partitions)
#SBATCH -p generic: 24 proc/ 63.G (4 sockets of 6 core CPUs)
#SBATCH -p phys: 80 proc/ 377G (4 sockets of 20 core CPUs)

Old article "Using UTEP HPC cloud" is kept in below.

Update on 2021-7-5: UTEP-HPC is being decommissioned. It will be replaced with jakar.

In order to perform HPC calculations on UTEP's new research cluod, these steps must be followed:

• A reservation to a PC must be made, you must load an MPI linux image (to compile MPI code), login to the reservation, transfer files and compile your code. However, you cannot submit jobs from here.
• A separate login to the HPC server must be made, here you can submit jobs, but cannot compile.

Detailed steps

1. Browse to http://hpcvcl00.utep.edu/vcl (the browser may give you a warning that this page poses a security risk) this will get you into the Virtual Computing Lab, here you need to provide your login name and password.
2. Once you login, click on the "New Reservation" tab.
3. Make sure you select the cenots6IntelFM environment from the drop down list, as this environment is the only one that provides a Linux image with a FORTRAN compiler with the MPI environment.
4. Select the hours you need to work with this environment (8hr maximum limit).
5. When the reservation is ready, the page will give you the ip address of your assigned node, your username and password on the node (the password is different that the VCL password and is valid only for this reservation).
6. You can now login to this node with the provided information ( in Windows you can use putty or ssh).
7. If it is your first login you must setup the PATH system variable to find the FORTRAN mpif90 compiler, to do this (only once), open in your home directory a file called .bashrc and put this into it:
   export PATH=$PATH:/usr/lib64/qt-3.3/bin:/opt/intel/composer_xe_2013.0.079/bin/intel64:/opt/intel/composer_xe_2013.0.079/mpirt/bin/intel64:/opt/intel/composer_xe_2013.0.079/bin/intel64:/opt/intel/composer_xe_2013.0.079/bin/intel64_mic:/opt/intel/composer_xe_2013.0.079/debugger/gui/intel64:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/mpich2/bin
8. For linking, here is an excerpt from the Makefile:
   create this variable
   UT_MKL_LIB = /opt/intel/mkl/lib/intel64/
   the linking is:
   $(FFF) $(OBJ) -o cluster -Wl,-rpath,$(UT_MKL_LIB) -L$(UT_MKL_LIB) -Wl,--start-group \
   -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -lpthread -lm
9. Once the code compiles you need to login to the HPC server at 10.91.31.5 (or just ssh to hpc) to submit your job, with the username and password from the VCL login, (the main password for the site, not the reservation password)
10. Here is a sample job script:
    #BSUB -n 72
    #BSUB -W 24:00
    #BSUB -q high_priority
    #BSUB -e error.%J
    #BSUB -o output.%J
    #BSUB -J test1

    /shared/mpi/bin/mpirun -np 72 ./cluster > log.txt

    Where the options are:
    -n number of processors on which to execute (must be a multiple of 12).
    -W maximum execution time (in hours).
    -q queue to submit job.
    -e file to print error information (if you job executes correctly, this file should be empty).
    -o file to print output information (prints mostly cluster usage).
    -J job name (small text to identify calculation).
    The last line is the MPI execution of the NRLMOL binary file called cluster on 72 processors (this number must be the same as the one in the -n option), all the output will be redirected to a file called. log.txt.

11. The queues you can use are:
    • normal_priority up to 32 cores.
    • medium_priority up to 48 cores.
    • high_priority up to 60 cores.
12. Submit your job with: bsub < job (it does not tell you what your jobnumber is if it is accepted)
13. You can monitor your jobs with: bjobs (you can get the jobnumber from here)
14. You can kill jobs with: bkill #jobnumber
15. You can also submit jobs through the HPC server website, goto http://10.91.31.5 (or http://hpc.utep.edu) and login, you can create and submit jobs from there.