Command line tools
lsxfel
Examine the contents of an EuXFEL proposal directory, run directory, or HDF5 file:
# Proposal directory
lsxfel /gpfs/exfel/exp/XMPL/201750/p700000
# Run directory
lsxfel /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002
# Single file
lsxfel /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0002/CORR-R0034-AGIPD00-S00000.h5
- --detail <source-pattern>
Show more detail on the keys and data of the sources selected by a pattern like
*/XGM/*
. Only applies when inspecting a single run or file. Can be used several times to select different patterns.This option can make
lsxfel
considerably slower.
extra-data-validate
Check the structure of an EuXFEL run or HDF5 file:
extra-data-validate /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002
If it finds problems with the data, the program will produce a list of them and exit with status 1. See Checking data files for details of what it checks.
karabo-bridge-serve-run
Stream data from a run, by proposal & run number, in the Karabo bridge format. See Streaming data over ZeroMQ for more information.
# Proposal run
karabo-bridge-serve-run 700000 40 --port 4321 \
--include 'SPB_IRDA_JF4M/DET/JNGFR*:daqOutput' \
--include '*/MOTOR/*[*Position]'
- --port <port>
Either a numeric TCP port, e.g.
4321
, or a ZMQ endpoint address, e.g.tcp://0.0.0.0:4321
. You will need to give the receiving code this port number or address as well.If no port is specified, the program will pick an unused port, and display the endpoint address as it starts.
- --include <pattern>
Sources matching the pattern will be included in the streamed data. You can also match keys by putting a key pattern in
[]
square brackets at the end of the source pattern.You must specify at least one
--include
pattern, and you can use the option several times to expand the selection.If data is flowing slower than you expect, see if you can use more specific patterns to avoid sending unnecessary sources.
- --allow-partial
By default, trains are only sent if they contain all the data selected by
--include
. This option also sends trains where some of that data is missing.
- --append-detector-modules
If the file data contains multiple detector modules as separate sources, i. e. for big area detectors (AGIPD, LPD and DSSC), append these into one single source.
- --dummy-timestamps
Add mock timestamps if missing in the original meta-data.
These two options above - appended module sources and dummy timestamps - are required if streamed data shall be provided to OnDA.
- -z <type>, --socket-type <type>
The ZMQ socket type to use, one of
PUB
,PUSH
orREP
. Default:REP
.
- --use-infiniband
Use the infiniband network interface (
ib0
) if it’s present. This is ignored if--port
is used with a full ZMQ endpoint address.
karabo-bridge-serve-files
Stream data from files in the Karabo bridge format. See Streaming data over ZeroMQ for more information.
For streaming data from a run directory, we recommend the newer karabo-bridge-serve-run command in place of this.
karabo-bridge-serve-files /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0005 4321
- --source <source>
Only sources matching the string <source> will be streamed. Default is ‘*’, serving as a global wildcard for all sources.
- --key <key>
Only data sets keyed by the string <key> will be streamed. Default is ‘*’, serving as a global wildcard for all keys.
- --append-detector-modules
If the file data contains multiple detector modules as separate sources, i. e. for big area detectors (AGIPD, LPD and DSSC), append these into one single source.
- --dummy-timestamps
Add mock timestamps if missing in the original meta-data.
These two options above - appended module sources and dummy timestamps - are required if streamed data shall be provided to OnDA.
- -z <type>, --socket-type <type>
The ZMQ socket type to use, one of
PUB
,PUSH
orREP
. Default:REP
.
- --use-infiniband
Use the infiniband network interface (
ib0
) if it’s present.
extra-data-make-virtual-cxi
Make a virtual CXI file to access AGIPD/LPD/JUNGFRAU detector data from a specified run:
extra-data-make-virtual-cxi /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0003 -o xmpl-3.cxi
- -o <path>, --output <path>
The filename to write. Defaults to creating a file in the proposal’s scratch directory.
- --min-modules <number>
Include trains where at least N modules have data (default: half+1 of all detector modules).
- --n-modules <number>
Number of detector modules in the experiment setup. Should be used only for JUNGFRAU data.
- --fill-value <dataset> <value>
Set the fill value for dataset (one of
data
,gain
ormask
). The defaults are different in different cases:data (raw, uint16): 0
data (proc, float32): NaN
gain: 0
mask: 0xffffffff
- --exc-suspect-trains
Exclude ‘Suspect’ train IDs from the data to assemble. This can fix some problems with bad train IDs.
extra-data-locality
Check how the files are stored:
extra-data-locality /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002
The file reading may hang for a long time if files are unavailable or require staging in dCache from the tape. The program helps finding problem files.
If it finds problems with the data locality, the program will produce a list of files located on tape, lost or at unknown locality and exit with the non-zero status.