Commit f61854f7 authored by Rafał Lalik's avatar Rafał Lalik
Browse files

Redo paths to be relative to DATA_DIR

parent b03f3202
......@@ -52,7 +52,7 @@ For all parameters, except for the location of DST analysis output files, order
- Location of DST analysis output files (input for the program). First parameter in order, input without an index prefacing it. Example: `../../output_dir_Pre_fRPC_Cal_New/be2104506572506.hld_dst_feb21.root` (multiple files can be selected using asterisks or question-marks, e.g. `../../output_dir_Pre_fRPC_Cal_New/be21045*.hld_dst_feb21.root`).
- Location of parameters file, prefaced with index `-p`. If not specified defaults to `./feb21_dst_params.txt`. Example: `-p ../../feb21_dst_params.txt`.
- Name of the root output file containing histograms (the name is also used to name the .txt file with calculated calibration parameters) prefaced with index `-o`. If not specified, defaults to `output.root`. Example: `-o output.root`.
- Path to where both output files (root file with histograms and txt file with calculated calibration parameters) should be saved (given folder needs to be manualy created, if path to nonexistent folder is given, the files won't be saved) prefaced with index `-d`. If not specified defaults to the same directory the program was executed in. Example: `-d ./out`.
- Path to where both output files (root file with histograms and txt file with calculated calibration parameters) should be saved (given folder needs to be manually created, if path to nonexistent folder is given, the files won't be saved) prefaced with index `-d`. If not specified defaults to the same directory the program was executed in. Example: `-d ./out`.
- Number of events to analyse, prefaced with index `-e`. If not specified, all events in selected files will be analysed. Example: `-e 500000`.
- Stage of calibration. Select one of:
- `--cal-frpc-stage1` - First step of calibration (thresholds).
......@@ -73,3 +73,99 @@ Optionally `2>&1 | tee fRPC_progress.txt` can also be added at the end of the co
```bash
./build/bin/elastics_monitor inputFiles
```
## Online full-dst production
The setup provides tools to monitor new hld files and submit dst production jobs to the farm. All the scripts will be installed with make install command to the `$CMAKE_INSTALL_PREFIX/bin` directory.
Before start, one need to prepare config file. There is provided default one `config.sh.example` so:
```bash
cp config.sh.example config.sh
```
The default configuration:
```bash
# general
DATA_DIR=../data
# hld watcher and crawler
HLD_WATCH_PID_FILE=/tmp/.hld_watcher.pid
HLD_WATCH_LOG_FILE=/tmp/.hld_watcher.log
HLD_LIST_FILE=hld_list.txt
HLD_DIR=/lustre/hades/raw/feb22/22
HLD_MASK=*/be*.hld
HLD_DST_STATUS_FILE=hld_dst_status.txt
HLD_DST_SUBMIT_FILE=hld_dst_submit.txt
DST_DIR=data/out
DST_EVENTS=1000000
DST_LOG_PREFIX=dst_slurm
DST_SUBMIT_SCRIPT=submit_dst_job.sh
DST_BATCH_SCRIPT=dst_batch_script.sh
DST_CHECK_SCRIPT=
DST_LIST_FILE=../data/dst_list.txt
DST_MASK=*/be*.hld_*.root
# dst watcher and crawler
DST_LUMI_WATCH_PID_FILE=/tmp/.dst_lumi_watcher.pid
DST_LUMI_WATCH_LOG_FILE=/tmp/.dst_lumi_watcher.log
DST_LUMI_SUBMIT_SCRIPT=submit_lumi_job.sh
DST_LUMI_BATCH_SCRIPT=lumi_batch_script.sh
DST_LUMI_SUBMIT_FILE=dst_lumi_submit.txt
LUMI_LOG_PREFIX=dst_lumi_slurm
LUMI_TOOL=lumi_monitor
LUMI_DIR=lumi
LUMI_LIST_FILE=lumi_list.txt
```
includes several variables, some of them are worth to mention:
* `DATA_DIR` - directory where results will be written, this path will be prepend to the following variables:
* `HLD_LIST_FILE` - keeps list of all found hld files
* `HLD_DST_SUBMIT_FILE` - keeps list of all submitted dst jobs (files)
* `HLD_DST_STATUS_FILE` - keeps status of the dst job, whether dst succeeded or failed
* `DST_LIST_FILE` - keeps list of all dst files
* `DST_DIR` - place where dst output files will be stored
* `LUMI_LIST_FILE` - list of the found dst inputs
* `DST_LUMI_SUBMIT_FILE` - keeps list of all submitted lumi inputs (files)
* `LUMI_DIR` - place where lumi files will be stored
* `*_PID_FILE` and `*_LOG_FILE`store process id and outputs for apps running as daemons
* `*_LOG_PREFIX` - prefix for slurm log files, which will be stored in `$DATA_DIR/slurm_log/$XXX_LOG_PREFIX-slurm-%j.log`
* `HLD_DIR` - directory to browse or monitor for new hld files
* `HLD_MASK` - search mask (glob), the files are searched inside `$HLD_DIR/$HLD_MASK`
* `DST_EVENTS` - number of events to analyze from hld file
* `*_SUBMIT_SCRIPT` - script which executes `sbatch` command, takes file as input, other variables are read out from config file
* `*_BATCH_SCRIPT` - script which is executed on the batch farm
* `DST_CHECK_SCRIPT` - program to check whether the dst production is correct
* `DST_MASK` - similar like `HLD_MASK`
* `LUMI_TOOL` - luminosity monitoring tool
* `LUMIDIR` output for lumi tool, relative to `DATA_DIR`
* `LUMI_LIST_FILE`- list of run dst files for lumi too
For most of the cases, one needs to adjust only `HLD_DIR` and `DATA_DIR`.
### Online dst
It is recommend to set up installation path with `cmake -DCMAKE_INSTALL_PREFIX=some_dir` and install files with `make install`
The basic usage is to start hld monitoring tool. Assuming that we are located in `some_dir/bin`:
```bash
./hld_watcher.py
```
Thsi will monitor every file created in `HLD_DIR` matching the `HLD_MASK` and submit the job if the new file appears. The app can be also run as a daemon with `-d` option, then the PID will be stored in `HLD_WATCH_PID_FILE` and out log in `HLD_WATCH_LOG_FILE`.
In case that some files could me missed (watcher was not working) one can run
```bash
./hld_watcher.py -l
```
which will list the directory and submit those files which were not processed yet. It is recommended to do it only a few times per day, e.g. using cron.
Each submitted file is stored in `HLD_DST_SUBMIT_FILE` therefore the tools above know which one were already send and will not submit the second time. If there is need to restart some jobs, one can remove these files from `HLD_DST_SUBMIT_FILE` and run command
```bash
./hld_watcher.py -s
```
which will go through `HLD_LIST` (so it will not list directory again) and submit those files which are not present in `HLD_DST_SUBMIT_FILE`.
See
```bash
./hld_watcher.py -h
```
for list of options.
### Luminosity monitoring
The muli is running on the DST files. The `lumi_watcher.py` program monitors the `DST_OUT` file for new files and submits jobs. It works in same way like the `hld_watcher.py` including options.
......@@ -10,7 +10,6 @@ import subprocess
# global variables
DST_LUMI_WATCH_PID_FILE="~/.dst_lumi_watcher.pid"
DST_LUMI_WATCH_LOG_FILE="~/.dst_lumi_watcher.log"
DST_LUMI_WATCHER_DAEMONIZE=False
def wildcard_to_regex(pattern, decorate=False):
......@@ -37,7 +36,7 @@ def parse_shell_config(fn):
if len(l) == 0 or l.startswith('#'):
continue
g = p.match(l).groups()
g = p.match(l.partition('#')[0].rstrip()).groups()
if len(g) == 2:
k = g[0]
v = g[1].strip('"')
......@@ -49,22 +48,6 @@ def parse_shell_config(fn):
print(f" - Override {k}={globals()[k]}")
def test_required_variables(var_list):
"""
Check whether required variables given as list of strings, are set up.
"""
print("* Check required variables:")
if not isinstance(var_list, list):
var_list=list(var_list)
for v in var_list:
if not v in globals():
print(f" *** ERROR *** - Variable {v} not defined")
sys.exit(1)
else:
print(f" - {v}={globals()[v]}")
def make_abs(path, wd):
if not os.path.isabs(path):
return wd+'/'+path
......@@ -73,15 +56,15 @@ def make_abs(path, wd):
WD=os.path.dirname(os.path.realpath(__file__))
parse_shell_config(make_abs('config.sh', WD))
test_required_variables([
'DATA_DIR', 'DST_DIR', 'DST_MASK', 'DST_LIST_FILE'
])
full_name = make_abs(DATA_DIR + '/' + DST_MASK, WD)
re_full_name = wildcard_to_regex(full_name, decorate=True)
p_full_name = re.compile(re_full_name)
ABS_LUMI_LIST_FILE=make_abs(DATA_DIR + '/' + LUMI_LIST_FILE, WD)
ABS_DST_DIR=make_abs(DATA_DIR + '/' + DST_DIR, WD)
def submit_file(name):
if len(DST_LUMI_SUBMIT_SCRIPT):
......@@ -130,14 +113,14 @@ class EventHandler(pyinotify.ProcessEvent):
#print("Modifying", event.pathname)
def process_IN_CLOSE_WRITE(self, event):
if p_full_name.match(event.pathname):
add_file_to_list(event.pathname, make_abs(LUMI_LIST_FILE, WD))
add_file_to_list(event.pathname, ABS_LUMI_LIST_FILE)
def on_loop(notifier):
pass
dirname = os.path.dirname(make_abs(LUMI_LIST_FILE, WD))
dirname = os.path.dirname(ABS_LUMI_LIST_FILE)
try:
os.mkdir(dirname)
except FileExistsError:
......@@ -153,16 +136,16 @@ if __name__ == "__main__":
if args.list:
r = glob.glob(full_name)
add_file_to_list(r, make_abs(DST_LIST_FILE, WD))
add_file_to_list(r, ABS_LUMI_LIST_FILE)
elif args.submit:
l = get_file_from_list(make_abs(DST_LIST_FILE, WD))
l = get_file_from_list(ABS_LUMI_LIST_FILE)
for ll in l:
submit_file(ll)
else:
wm = pyinotify.WatchManager()
handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wm.add_watch(DST_DIR,
wm.add_watch(ABS_DST_DIR,
pyinotify.ALL_EVENTS,
rec=True)
......
......@@ -12,7 +12,7 @@ function make_abs()
if [[ "$1" = /* ]]; then
local myresult=$1
else
local myresult=$2/$1
local myresult=$(readlink -e $2/$1)
fi
eval $__returnval="'$myresult'"
......@@ -20,6 +20,11 @@ function make_abs()
[ -f $WD/config.sh ] && . $WD/config.sh || exit 1
# data dir must be defined
[[ -n $DATA_DIR ]] || exit 1
make_abs "$DATA_DIR" "$WD" ABS_DATA_DIR
mkdir $ABS_DATA_DIR -p
# batch script must exists
make_abs "$DST_LUMI_BATCH_SCRIPT" "$WD" ABS_DST_LUMI_BATCH_SCRIPT
[[ -f $ABS_DST_LUMI_BATCH_SCRIPT ]] || exit 2
......@@ -28,21 +33,17 @@ make_abs "$DST_LUMI_BATCH_SCRIPT" "$WD" ABS_DST_LUMI_BATCH_SCRIPT
make_abs "$LUMI_TOOL" "$WD" ABS_LUMI_TOOL
[[ -n $ABS_LUMI_TOOL ]] || exit 3
# data dir must be defined
[[ -n $DATA_DIR ]] || exit 6
make_abs "$DATA_DIR" "$WD" ABS_DATA_DIR
mkdir $ABS_DATA_DIR -p
# lumi dir must be defined
[[ -n $LUMI_DIR ]] || exit 6
make_abs "$LUMI_DIR" "$WD" ABS_LUMI_DIR
ABS_LUMI_DIR=$ABS_DATA_DIR/$LUMI_DIR
mkdir $ABS_LUMI_DIR -p
# submit file, containing directory must exists
make_abs "$DST_LUMI_SUBMIT_FILE" "$WD" ABS_DST_LUMI_SUBMIT_FILE
ABS_DST_LUMI_SUBMIT_FILE=$ABS_DATA_DIR/$DST_LUMI_SUBMIT_FILE
mkdir $(dirname $ABS_DST_LUMI_SUBMIT_FILE) -p
[ ! -e $ABS_DST_LUMI_SUBMIT_FILE ] && touch $ABS_DST_LUMI_SUBMIT_FILE
make_abs "$DST_LUMI_STATUS_FILE" "$WD" ABS_DST_LUMI_STATUS_FILE
ABS_DST_LUMI_STATUS_FILE=$ABS_DATA_DIR/$DST_LUMI_STATUS_FILE
mkdir $(dirname $ABS_DST_LUMI_STATUS_FILE) -p
if ! grep -q "^$1" $ABS_DST_LUMI_SUBMIT_FILE; then
......
......@@ -4,18 +4,18 @@ DATA_DIR=../data
# hld watcher and crawler
HLD_WATCH_PID_FILE=/tmp/.hld_watcher.pid
HLD_WATCH_LOG_FILE=/tmp/.hld_watcher.log
HLD_LIST_FILE=../data/hld_list.txt
HLD_LIST_FILE=hld_list.txt
HLD_DIR=/lustre/hades/raw/feb22/22
HLD_MASK=*/be*.hld
HLD_DST_STATUS_FILE=../data/hld_dst_status.txt
HLD_DST_SUBMIT_FILE=../data/hld_dst_submit.txt
DST_DIR=../data/out
HLD_DST_STATUS_FILE=hld_dst_status.txt
HLD_DST_SUBMIT_FILE=hld_dst_submit.txt
DST_DIR=out
DST_EVENTS=1000000
DST_LOG_PREFIX=dst_slurm
DST_SUBMIT_SCRIPT=submit_dst_job.sh
DST_BATCH_SCRIPT=dst_batch_script.sh
DST_CHECK_SCRIPT=
DST_LIST_FILE=../data/dst_list.txt
DST_LIST_FILE=dst_list.txt
DST_MASK=*/be*.hld_*.root
# dst watcher and crawler
......@@ -23,8 +23,8 @@ DST_LUMI_WATCH_PID_FILE=/tmp/.dst_lumi_watcher.pid
DST_LUMI_WATCH_LOG_FILE=/tmp/.dst_lumi_watcher.log
DST_LUMI_SUBMIT_SCRIPT=submit_lumi_job.sh
DST_LUMI_BATCH_SCRIPT=lumi_batch_script.sh
DST_LUMI_SUBMIT_FILE=../data/dst_lumi_submit.txt
DST_LUMI_SUBMIT_FILE=dst_lumi_submit.txt
LUMI_LOG_PREFIX=dst_lumi_slurm
LUMI_TOOL=lumi_monitor
LUMI_DIR=../data/lumi
LUMI_LIST_FILE=../data/lumi_list.txt
LUMI_DIR=lumi
LUMI_LIST_FILE=lumi_list.txt
......@@ -10,7 +10,6 @@ import subprocess
# global variables
HLD_WATCH_PID_FILE="~/.hld_watcher.pid"
HLD_WATCH_LOG_FILE="~/.hld_watcher.log"
HLD_WATCHER_DAEMONIZE=False
def wildcard_to_regex(pattern, decorate=False):
......@@ -37,7 +36,7 @@ def parse_shell_config(fn):
if len(l) == 0 or l.startswith('#'):
continue
g = p.match(l).groups()
g = p.match(l.partition('#')[0].rstrip()).groups()
if len(g) == 2:
k = g[0]
v = g[1].strip('"')
......@@ -49,22 +48,6 @@ def parse_shell_config(fn):
print(f" - Override {k}={globals()[k]}")
def test_required_variables(var_list):
"""
Check whether required variables given as list of strings, are set up.
"""
print("* Check required variables:")
if not isinstance(var_list, list):
var_list=list(var_list)
for v in var_list:
if not v in globals():
print(f" *** ERROR *** - Variable {v} not defined")
sys.exit(1)
else:
print(f" - {v}={globals()[v]}")
def make_abs(path, wd):
if not os.path.isabs(path):
return wd+'/'+path
......@@ -73,15 +56,14 @@ def make_abs(path, wd):
WD=os.path.dirname(os.path.realpath(__file__))
parse_shell_config(make_abs('config.sh', WD))
test_required_variables([
'DATA_DIR', 'HLD_DIR', 'HLD_MASK', 'HLD_LIST_FILE'
])
full_name = make_abs(HLD_DIR + '/' + HLD_MASK, WD)
re_full_name = wildcard_to_regex(full_name, decorate=True)
p_full_name = re.compile(re_full_name)
ABS_HLD_LIST_FILE=make_abs(DATA_DIR + '/' + HLD_LIST_FILE, WD)
def submit_file(name):
if len(DST_SUBMIT_SCRIPT):
......@@ -130,14 +112,14 @@ class EventHandler(pyinotify.ProcessEvent):
#print("Modifying", event.pathname)
def process_IN_CLOSE_WRITE(self, event):
if p_full_name.match(event.pathname):
add_file_to_list(event.pathname, make_abs(HLD_LIST_FILE, WD))
add_file_to_list(event.pathname, ABS_HLD_LIST_FILE)
def on_loop(notifier):
pass
dirname = os.path.dirname(make_abs(HLD_LIST_FILE, WD))
dirname = os.path.dirname(ABS_HLD_LIST_FILE)
try:
os.mkdir(dirname)
except FileExistsError:
......@@ -153,9 +135,9 @@ if __name__ == "__main__":
if args.list:
r = glob.glob(full_name)
add_file_to_list(r, make_abs(HLD_LIST_FILE, WD))
add_file_to_list(r, ABS_HLD_LIST_FILE)
elif args.submit:
l = get_file_from_list(make_abs(HLD_LIST_FILE, WD))
l = get_file_from_list(ABS_HLD_LIST_FILE)
for ll in l:
submit_file(ll)
else:
......
......@@ -12,7 +12,7 @@ function make_abs()
if [[ "$1" = /* ]]; then
local myresult=$1
else
local myresult=$2/$1
local myresult=$(readlink -e $2/$1)
fi
eval $__returnval="'$myresult'"
......@@ -20,6 +20,11 @@ function make_abs()
[ -f $WD/config.sh ] && . $WD/config.sh || exit 1
# data dir must be defined
[[ -n $DATA_DIR ]] || exit 1
make_abs "$DATA_DIR" "$WD" ABS_DATA_DIR
mkdir $ABS_DATA_DIR -p
# batch script must exists
make_abs "$DST_BATCH_SCRIPT" "$WD" ABS_DST_BATCH_SCRIPT
[[ -f $ABS_DST_BATCH_SCRIPT ]] || exit 2
......@@ -29,23 +34,19 @@ make_abs "$HLD_CHECK_SCRIPT" "$WD" ABS_HLD_CHECK_SCRIPT
[[ -z $HLD_CHECK_SCRIPT ]] || [[ -f $ABS_HLD_CHECK_SCRIPT ]] || exit 3
# status file must be set if the check script is set
make_abs "$HLD_DST_STATUS_FILE" "$WD" ABS_HLD_DST_STATUS_FILE
[[ -z $HLD_CHECK_SCRIPT ]] || [[ -n $HLD_DST_STATUS_FILE ]] || exit 4
mkdir $(dirname $HLD_DST_STATUS_FILE) -p
ABS_HLD_DST_STATUS_FILE=$ABS_DATA_DIR/$HLD_DST_STATUS_FILE
mkdir $(dirname $ABS_HLD_DST_STATUS_FILE) -p
# dst output dir must be defined
[[ -n $DST_DIR ]] || exit 5
make_abs "$DST_DIR" "$WD" ABS_DST_DIR
mkdir $(dirname $ABS_DST_DIR) -p
# data dir must be defined
[[ -n $DATA_DIR ]] || exit 6
make_abs "$DATA_DIR" "$WD" ABS_DATA_DIR
mkdir $(dirname $ABS_DATA_DIR) -p
ABS_DST_DIR=$ABS_DATA_DIR/$DST_DIR
mkdir $ABS_DST_DIR -p
# submit file, containing directory must exists
make_abs "$HLD_DST_SUBMIT_FILE" "$WD" ABS_HLD_DST_SUBMIT_FILE
ABS_HLD_DST_SUBMIT_FILE=$ABS_DATA_DIR/$HLD_DST_SUBMIT_FILE
mkdir $(dirname $ABS_HLD_DST_SUBMIT_FILE) -p
[ ! -e $ABS_HLD_DST_SUBMIT_FILE ] && touch $ABS_HLD_DST_SUBMIT_FILE
if ! grep -q "^$1" $ABS_HLD_DST_SUBMIT_FILE; then
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment