Accessing LPD data
The Large Pixel Detector (LPD) 1M is made of 16 modules which record data separately. extra_data includes convenient interfaces to access this data together.
This example stands by itself, but if you need more generic access to the data, please see other examples, including Reading data to analyse in memory and Reading data train by train.
The example uses some example data stored at the European XFEL Maxwell cluster. First, let’s load a run containing LPD data:
[1]:
from extra_data import open_run, by_index
run = open_run(proposal=700002, run=117)
# Using only the first three trains to keep this example light:
run = run.select_trains(by_index[:3])
run.instrument_sources
[1]:
frozenset({'FXE_AUXT_LIC/DOOCS/BAM_1932M:output',
'FXE_AUXT_LIC/DOOCS/BAM_1932S:output',
'FXE_DET_LPD1M-1/CORR/0CH0:output',
'FXE_DET_LPD1M-1/CORR/10CH0:output',
'FXE_DET_LPD1M-1/CORR/11CH0:output',
'FXE_DET_LPD1M-1/CORR/12CH0:output',
'FXE_DET_LPD1M-1/CORR/13CH0:output',
'FXE_DET_LPD1M-1/CORR/14CH0:output',
'FXE_DET_LPD1M-1/CORR/15CH0:output',
'FXE_DET_LPD1M-1/CORR/1CH0:output',
'FXE_DET_LPD1M-1/CORR/2CH0:output',
'FXE_DET_LPD1M-1/CORR/3CH0:output',
'FXE_DET_LPD1M-1/CORR/4CH0:output',
'FXE_DET_LPD1M-1/CORR/5CH0:output',
'FXE_DET_LPD1M-1/CORR/6CH0:output',
'FXE_DET_LPD1M-1/CORR/7CH0:output',
'FXE_DET_LPD1M-1/CORR/8CH0:output',
'FXE_DET_LPD1M-1/CORR/9CH0:output',
'FXE_DET_LPD1M-1/DET/0CH0:xtdf',
'FXE_DET_LPD1M-1/DET/10CH0:xtdf',
'FXE_DET_LPD1M-1/DET/11CH0:xtdf',
'FXE_DET_LPD1M-1/DET/12CH0:xtdf',
'FXE_DET_LPD1M-1/DET/13CH0:xtdf',
'FXE_DET_LPD1M-1/DET/14CH0:xtdf',
'FXE_DET_LPD1M-1/DET/15CH0:xtdf',
'FXE_DET_LPD1M-1/DET/1CH0:xtdf',
'FXE_DET_LPD1M-1/DET/2CH0:xtdf',
'FXE_DET_LPD1M-1/DET/3CH0:xtdf',
'FXE_DET_LPD1M-1/DET/4CH0:xtdf',
'FXE_DET_LPD1M-1/DET/5CH0:xtdf',
'FXE_DET_LPD1M-1/DET/6CH0:xtdf',
'FXE_DET_LPD1M-1/DET/7CH0:xtdf',
'FXE_DET_LPD1M-1/DET/8CH0:xtdf',
'FXE_DET_LPD1M-1/DET/9CH0:xtdf',
'FXE_RR_DAQ/ADC/1:network',
'FXE_SMS_MOV/CAM/SIDE:daqOutput',
'SA1_XTD2_XGM/DOOCS/MAIN:output',
'SPB_XTD9_XGM/DOOCS/MAIN:output'})
Source names with */DET/* represent raw detector data, and */CORR/* - data produced with the offline calibration pipeline. Normal access methods give us each module separately:
[2]:
data_module0 = run['FXE_DET_LPD1M-1/CORR/0CH0:output', 'image.data'].ndarray()
data_module0.shape
[2]:
(3, 256, 256)
The class extra_data.components.LPD1M can piece these together:
[3]:
from extra_data.components import LPD1M
lpd = LPD1M(run)
lpd
[3]:
<LPD1M: Data interface for detector 'FXE_DET_LPD1M-1' - proc data with 16 modules>
[4]:
image_data = lpd['image.data'].xarray()
print("Data shape:", image_data.shape)
print("Dimensions:", image_data.dims)
Data shape: (16, 3, 256, 256)
Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')
Note: This class pulls the data together, but it doesn’t know how the modules are physically arranged, so it can’t produce a detector image. Other examples show how to use detector geometry to produce images.
You can also select only certain modules of the detector. For example, modules 2 (Q1M3), 7 (Q2M4), 8 (Q3M1) and 13 (Q4M2) are the four modules around the center of the detector:
[5]:
lpd_center = LPD1M(run, modules=[2, 7, 8, 13])
image_data = lpd_center['image.data'].xarray()
print("Data shape:", image_data.shape)
print("Dimensions:", image_data.dims)
print()
print("Data for one pulse:")
print(image_data.sel(train=1713878183, pulse=8))
Data shape: (4, 3, 256, 256)
Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')
Data for one pulse:
<xarray.DataArray (module: 4, slow_scan: 256, fast_scan: 256)> Size: 1MB
array([[[ 32.326824 , 13.826709 , 0. , ..., 47.376667 ,
67.57162 , 9.247609 ],
[ -8.861919 , 32.85273 , 72.682076 , ..., 45.043106 ,
39.2642 , 3.1752753 ],
[ 33.83956 , 13.316033 , 27.64509 , ..., 31.393215 ,
21.25742 , 17.279358 ],
...,
[ 0. , 0. , 0. , ..., 0. ,
-2.1826582 , -0. ],
[ -0. , 0. , 0. , ..., 0. ,
0. , 0. ],
[ 0. , 0. , 0. , ..., 11.430056 ,
-15.085268 , -8.816615 ]],
[[ 12.569354 , 27.73656 , 32.966244 , ..., 9.83708 ,
4.870547 , 36.752926 ],
[-14.2927 , 16.467594 , -37.383083 , ..., 18.955885 ,
0. , 10.458132 ],
[-14.309514 , 26.83833 , 13.27472 , ..., -7.302546 ,
-3.7747538 , -0.40490106],
...
[ 11.1194515 , -3.1674623 , 10.65781 , ..., 77.86272 ,
16.799505 , -7.242821 ],
[ 25.012783 , 27.332106 , -5.850013 , ..., 56.722233 ,
70.28 , 29.09655 ],
[ 0. , 0. , 0. , ..., 0. ,
-13.461099 , 37.82373 ]],
[[ 12.360411 , 21.77143 , 23.096016 , ..., 22.170837 ,
98.64886 , 18.146912 ],
[ 5.122873 , -3.149296 , 15.456235 , ..., 19.706753 ,
44.125072 , 14.149995 ],
[ 29.712288 , 5.144479 , 17.113897 , ..., 104.83531 ,
90.16106 , 35.92096 ],
...,
[ 0. , 0. , -0. , ..., 49.876316 ,
-12.8638 , 57.798405 ],
[ -0. , -0. , 0. , ..., 31.26613 ,
9.25259 , 18.415903 ],
[ 0. , 0. , 0. , ..., -2.557403 ,
17.151001 , 3.9669514 ]]], dtype=float32)
Coordinates:
train_pulse object 8B (1713878183, 8)
train uint64 8B 1713878183
pulse uint64 8B 8
* module (module) int64 32B 2 7 8 13
Dimensions without coordinates: slow_scan, fast_scan
The returned array is an xarray object with labelled axes. See Indexing and selecting data in the xarray docs for more on what you can do with it.
LPD data may also be recorded in parallel gain mode, resulting in high-, medium- and low-gain frames for each pulse. To read this kind of data with the correct labels, use LPD1M(run, parallel_gain=True). This will retrieve data with an extra gain dimension, labelled with 0, 1 and 2 for high-, medium- and low-gain respectively.
Masking bad detector pixels
Some detector pixels may be defective and provide wrong signal readings. Most of such abnormal pixels are identified by the offline calibration pipeline and stored as a dynamic (per detector image) mask. extra_data includes a convenient method to apply such mask to the detector data:
[6]:
masked_data = lpd.masked_data().xarray()
print("Data shape:", masked_data.shape)
print("Dimensions:", masked_data.dims)
print()
print("Slice [0, 0, 29:34, 126:131] from the masked data:")
print(masked_data[0, 0, 29:34, 126:131])
Data shape: (16, 3, 256, 256)
Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')
Slice [0, 0, 29:34, 126:131] from the masked data:
<xarray.DataArray (slow_scan: 5, fast_scan: 5)> Size: 100B
array([[ nan, nan, 5.3925676, -2.014706 , 9.91627 ],
[ nan, nan, nan, -1.9685062, -12.091958 ],
[ nan, nan, nan, nan, 2.643778 ],
[ 15.758542 , 36.188587 , 2.4503093, 43.39787 , 15.549667 ],
[ 43.927513 , 7.209188 , 9.21519 , 65.89971 , 25.33027 ]],
dtype=float32)
Coordinates:
train_pulse object 8B (1713878183, 8)
train uint64 8B 1713878183
pulse uint64 8B 8
module int64 8B 0
Dimensions without coordinates: slow_scan, fast_scan
Here signal values from the defective pixels have been replaced with NaNs.
In case there is a need to inspect the dynamic mask itself - it can be read with:
[7]:
mask = lpd['image.mask'].xarray()
print("Slice [0, 0, 29:34, 126:131] from the dynamic mask:")
print(mask[0, 0, 29:34, 126:131])
Slice [0, 0, 29:34, 126:131] from the dynamic mask:
<xarray.DataArray (slow_scan: 5, fast_scan: 5)> Size: 100B
array([[35, 35, 0, 0, 0],
[35, 35, 32, 0, 0],
[35, 35, 32, 32, 0],
[ 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0]], dtype=uint32)
Coordinates:
train_pulse object 8B (1713878183, 8)
train uint64 8B 1713878183
pulse uint64 8B 8
module int64 8B 0
Dimensions without coordinates: slow_scan, fast_scan
This mask represents good pixels with 0 and bad pixels are bitwise encoded according to the enumeration of bad pixel reasons.
Processing detector data in chunks
For processing large sets of data it is useful to split them into chunks:
[8]:
for chunk in lpd.split_trains(trains_per_part=1):
chunk_data = chunk.masked_data().xarray()
print("Shape of the chunked data array:", chunk_data.shape)
print("Slice [0, 0, 29:34, 126:131] from the masked data:")
print(chunk_data[0, 0, 29:34, 126:131])
print()
Shape of the chunked data array: (16, 1, 256, 256)
Slice [0, 0, 29:34, 126:131] from the masked data:
<xarray.DataArray (slow_scan: 5, fast_scan: 5)> Size: 100B
array([[ nan, nan, 5.3925676, -2.014706 , 9.91627 ],
[ nan, nan, nan, -1.9685062, -12.091958 ],
[ nan, nan, nan, nan, 2.643778 ],
[ 15.758542 , 36.188587 , 2.4503093, 43.39787 , 15.549667 ],
[ 43.927513 , 7.209188 , 9.21519 , 65.89971 , 25.33027 ]],
dtype=float32)
Coordinates:
train_pulse object 8B (1713878183, 8)
train uint64 8B 1713878183
pulse uint64 8B 8
module int64 8B 0
Dimensions without coordinates: slow_scan, fast_scan
Shape of the chunked data array: (16, 1, 256, 256)
Slice [0, 0, 29:34, 126:131] from the masked data:
<xarray.DataArray (slow_scan: 5, fast_scan: 5)> Size: 100B
array([[ nan, nan, 33.254166 , 1.007353 , 19.83254 ],
[ nan, nan, nan, 13.779544 , 21.985378 ],
[ nan, nan, nan, nan, 21.150225 ],
[ 8.342757 , 26.872713 , 29.053667 , 5.585864 , 26.779982 ],
[-15.76885 , -9.912634 , -1.3164557, 15.124523 , 50.121597 ]],
dtype=float32)
Coordinates:
train_pulse object 8B (1713878184, 8)
train uint64 8B 1713878184
pulse uint64 8B 8
module int64 8B 0
Dimensions without coordinates: slow_scan, fast_scan
Shape of the chunked data array: (16, 1, 256, 256)
Slice [0, 0, 29:34, 126:131] from the masked data:
<xarray.DataArray (slow_scan: 5, fast_scan: 5)> Size: 100B
array([[ nan, nan, 15.278942 , 7.0514708 , 67.21028 ],
[ nan, nan, nan, 0. , 12.091958 ],
[ nan, nan, nan, nan, 17.625187 ],
[29.663137 , 1.7915142 , 59.157467 , 60.585144 , 58.743187 ],
[-6.7580786 , 5.406891 , -0.43881857, -6.4819384 , 80.302345 ]],
dtype=float32)
Coordinates:
train_pulse object 8B (1713878185, 8)
train uint64 8B 1713878185
pulse uint64 8B 8
module int64 8B 0
Dimensions without coordinates: slow_scan, fast_scan