{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Accessing LPD data\n", "\n", "The Large Pixel Detector (LPD) 1M is made of 16 modules which record data separately.\n", "`extra_data` includes convenient interfaces to access this data together.\n", "\n", "This example stands by itself, but if you need more generic access to the data,\n", "please see other examples, including [Reading data to analyse in memory](xpd_examples.ipynb)\n", "and [Reading data train by train](iterate_trains.ipynb)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The example uses some example data stored at the European XFEL Maxwell cluster. First, let's load a run containing LPD data:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "frozenset({'FXE_AUXT_LIC/DOOCS/BAM_1932M:output',\n", " 'FXE_AUXT_LIC/DOOCS/BAM_1932S:output',\n", " 'FXE_DET_LPD1M-1/CORR/0CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/10CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/11CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/12CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/13CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/14CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/15CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/1CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/2CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/3CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/4CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/5CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/6CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/7CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/8CH0:output',\n", " 'FXE_DET_LPD1M-1/CORR/9CH0:output',\n", " 'FXE_DET_LPD1M-1/DET/0CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/10CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/11CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/12CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/13CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/14CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/15CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/1CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/2CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/3CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/4CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/5CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/6CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/7CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/8CH0:xtdf',\n", " 'FXE_DET_LPD1M-1/DET/9CH0:xtdf',\n", " 'FXE_RR_DAQ/ADC/1:network',\n", " 'FXE_SMS_MOV/CAM/SIDE:daqOutput',\n", " 'SA1_XTD2_XGM/DOOCS/MAIN:output',\n", " 'SPB_XTD9_XGM/DOOCS/MAIN:output'})" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from extra_data import open_run, by_index\n", "\n", "run = open_run(proposal=700002, run=117)\n", "# Using only the first three trains to keep this example light:\n", "run = run.select_trains(by_index[:3])\n", "\n", "run.instrument_sources" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Source names with `*/DET/*` represent raw detector data, and `*/CORR/*` - data produced with the offline calibration pipeline. Normal access methods give us each module separately:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(3, 256, 256)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_module0 = run['FXE_DET_LPD1M-1/CORR/0CH0:output', 'image.data'].ndarray()\n", "data_module0.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The class `extra_data.components.LPD1M` can piece these together:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from extra_data.components import LPD1M\n", "lpd = LPD1M(run)\n", "lpd" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data shape: (16, 3, 256, 256)\n", "Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')\n" ] } ], "source": [ "image_data = lpd['image.data'].xarray()\n", "print(\"Data shape:\", image_data.shape)\n", "print(\"Dimensions:\", image_data.dims)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** This class pulls the data together, but it doesn't know how the modules are physically arranged,\n", "so it can't produce a detector image. Other examples show how to use detector geometry to produce images." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also select only certain modules of the detector. For example, modules 2 (Q1M3), 7 (Q2M4), 8 (Q3M1) and 13 (Q4M2) are the four modules around the center of the detector:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data shape: (4, 3, 256, 256)\n", "Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')\n", "\n", "Data for one pulse:\n", " Size: 1MB\n", "array([[[ 32.326824 , 13.826709 , 0. , ..., 47.376667 ,\n", " 67.57162 , 9.247609 ],\n", " [ -8.861919 , 32.85273 , 72.682076 , ..., 45.043106 ,\n", " 39.2642 , 3.1752753 ],\n", " [ 33.83956 , 13.316033 , 27.64509 , ..., 31.393215 ,\n", " 21.25742 , 17.279358 ],\n", " ...,\n", " [ 0. , 0. , 0. , ..., 0. ,\n", " -2.1826582 , -0. ],\n", " [ -0. , 0. , 0. , ..., 0. ,\n", " 0. , 0. ],\n", " [ 0. , 0. , 0. , ..., 11.430056 ,\n", " -15.085268 , -8.816615 ]],\n", "\n", " [[ 12.569354 , 27.73656 , 32.966244 , ..., 9.83708 ,\n", " 4.870547 , 36.752926 ],\n", " [-14.2927 , 16.467594 , -37.383083 , ..., 18.955885 ,\n", " 0. , 10.458132 ],\n", " [-14.309514 , 26.83833 , 13.27472 , ..., -7.302546 ,\n", " -3.7747538 , -0.40490106],\n", "...\n", " [ 11.1194515 , -3.1674623 , 10.65781 , ..., 77.86272 ,\n", " 16.799505 , -7.242821 ],\n", " [ 25.012783 , 27.332106 , -5.850013 , ..., 56.722233 ,\n", " 70.28 , 29.09655 ],\n", " [ 0. , 0. , 0. , ..., 0. ,\n", " -13.461099 , 37.82373 ]],\n", "\n", " [[ 12.360411 , 21.77143 , 23.096016 , ..., 22.170837 ,\n", " 98.64886 , 18.146912 ],\n", " [ 5.122873 , -3.149296 , 15.456235 , ..., 19.706753 ,\n", " 44.125072 , 14.149995 ],\n", " [ 29.712288 , 5.144479 , 17.113897 , ..., 104.83531 ,\n", " 90.16106 , 35.92096 ],\n", " ...,\n", " [ 0. , 0. , -0. , ..., 49.876316 ,\n", " -12.8638 , 57.798405 ],\n", " [ -0. , -0. , 0. , ..., 31.26613 ,\n", " 9.25259 , 18.415903 ],\n", " [ 0. , 0. , 0. , ..., -2.557403 ,\n", " 17.151001 , 3.9669514 ]]], dtype=float32)\n", "Coordinates:\n", " train_pulse object 8B (1713878183, 8)\n", " train uint64 8B 1713878183\n", " pulse uint64 8B 8\n", " * module (module) int64 32B 2 7 8 13\n", "Dimensions without coordinates: slow_scan, fast_scan\n" ] } ], "source": [ "lpd_center = LPD1M(run, modules=[2, 7, 8, 13])\n", "image_data = lpd_center['image.data'].xarray()\n", "print(\"Data shape:\", image_data.shape)\n", "print(\"Dimensions:\", image_data.dims)\n", "\n", "print()\n", "print(\"Data for one pulse:\")\n", "print(image_data.sel(train=1713878183, pulse=8))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The returned array is an *xarray* object with labelled axes.\n", "See [Indexing and selecting data](https://xarray.pydata.org/en/stable/indexing.html) in the xarray docs\n", "for more on what you can do with it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "LPD data may also be recorded in *parallel gain* mode, resulting in high-, medium- and low-gain frames for each pulse. To read this kind of data with the correct labels, use `LPD1M(run, parallel_gain=True)`. This will retrieve data with an extra gain dimension, labelled with 0, 1 and 2 for high-, medium- and low-gain respectively." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Masking bad detector pixels" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some detector pixels may be defective and provide wrong signal readings. Most of such abnormal pixels are identified by the offline calibration pipeline and stored as a dynamic (per detector image) mask. `extra_data` includes a convenient method to apply such mask to the detector data:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data shape: (16, 3, 256, 256)\n", "Dimensions: ('module', 'train_pulse', 'slow_scan', 'fast_scan')\n", "\n", "Slice [0, 0, 29:34, 126:131] from the masked data:\n", " Size: 100B\n", "array([[ nan, nan, 5.3925676, -2.014706 , 9.91627 ],\n", " [ nan, nan, nan, -1.9685062, -12.091958 ],\n", " [ nan, nan, nan, nan, 2.643778 ],\n", " [ 15.758542 , 36.188587 , 2.4503093, 43.39787 , 15.549667 ],\n", " [ 43.927513 , 7.209188 , 9.21519 , 65.89971 , 25.33027 ]],\n", " dtype=float32)\n", "Coordinates:\n", " train_pulse object 8B (1713878183, 8)\n", " train uint64 8B 1713878183\n", " pulse uint64 8B 8\n", " module int64 8B 0\n", "Dimensions without coordinates: slow_scan, fast_scan\n" ] } ], "source": [ "masked_data = lpd.masked_data().xarray()\n", "print(\"Data shape:\", masked_data.shape)\n", "print(\"Dimensions:\", masked_data.dims)\n", "print()\n", "print(\"Slice [0, 0, 29:34, 126:131] from the masked data:\")\n", "print(masked_data[0, 0, 29:34, 126:131])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here signal values from the defective pixels have been replaced with NaNs.\n", "\n", "In case there is a need to inspect the dynamic mask itself - it can be read with:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Slice [0, 0, 29:34, 126:131] from the dynamic mask:\n", " Size: 100B\n", "array([[35, 35, 0, 0, 0],\n", " [35, 35, 32, 0, 0],\n", " [35, 35, 32, 32, 0],\n", " [ 0, 0, 0, 0, 0],\n", " [ 0, 0, 0, 0, 0]], dtype=uint32)\n", "Coordinates:\n", " train_pulse object 8B (1713878183, 8)\n", " train uint64 8B 1713878183\n", " pulse uint64 8B 8\n", " module int64 8B 0\n", "Dimensions without coordinates: slow_scan, fast_scan\n" ] } ], "source": [ "mask = lpd['image.mask'].xarray()\n", "print(\"Slice [0, 0, 29:34, 126:131] from the dynamic mask:\")\n", "print(mask[0, 0, 29:34, 126:131])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This mask represents good pixels with `0` and bad pixels are bitwise encoded according to the [enumeration of bad pixel reasons](https://extra.readthedocs.io/en/latest/calibration/#extra.calibration.BadPixels)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Processing detector data in chunks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For processing large sets of data it is useful to split them into chunks:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shape of the chunked data array: (16, 1, 256, 256)\n", "Slice [0, 0, 29:34, 126:131] from the masked data:\n", " Size: 100B\n", "array([[ nan, nan, 5.3925676, -2.014706 , 9.91627 ],\n", " [ nan, nan, nan, -1.9685062, -12.091958 ],\n", " [ nan, nan, nan, nan, 2.643778 ],\n", " [ 15.758542 , 36.188587 , 2.4503093, 43.39787 , 15.549667 ],\n", " [ 43.927513 , 7.209188 , 9.21519 , 65.89971 , 25.33027 ]],\n", " dtype=float32)\n", "Coordinates:\n", " train_pulse object 8B (1713878183, 8)\n", " train uint64 8B 1713878183\n", " pulse uint64 8B 8\n", " module int64 8B 0\n", "Dimensions without coordinates: slow_scan, fast_scan\n", "\n", "Shape of the chunked data array: (16, 1, 256, 256)\n", "Slice [0, 0, 29:34, 126:131] from the masked data:\n", " Size: 100B\n", "array([[ nan, nan, 33.254166 , 1.007353 , 19.83254 ],\n", " [ nan, nan, nan, 13.779544 , 21.985378 ],\n", " [ nan, nan, nan, nan, 21.150225 ],\n", " [ 8.342757 , 26.872713 , 29.053667 , 5.585864 , 26.779982 ],\n", " [-15.76885 , -9.912634 , -1.3164557, 15.124523 , 50.121597 ]],\n", " dtype=float32)\n", "Coordinates:\n", " train_pulse object 8B (1713878184, 8)\n", " train uint64 8B 1713878184\n", " pulse uint64 8B 8\n", " module int64 8B 0\n", "Dimensions without coordinates: slow_scan, fast_scan\n", "\n", "Shape of the chunked data array: (16, 1, 256, 256)\n", "Slice [0, 0, 29:34, 126:131] from the masked data:\n", " Size: 100B\n", "array([[ nan, nan, 15.278942 , 7.0514708 , 67.21028 ],\n", " [ nan, nan, nan, 0. , 12.091958 ],\n", " [ nan, nan, nan, nan, 17.625187 ],\n", " [29.663137 , 1.7915142 , 59.157467 , 60.585144 , 58.743187 ],\n", " [-6.7580786 , 5.406891 , -0.43881857, -6.4819384 , 80.302345 ]],\n", " dtype=float32)\n", "Coordinates:\n", " train_pulse object 8B (1713878185, 8)\n", " train uint64 8B 1713878185\n", " pulse uint64 8B 8\n", " module int64 8B 0\n", "Dimensions without coordinates: slow_scan, fast_scan\n", "\n" ] } ], "source": [ "for chunk in lpd.split_trains(trains_per_part=1):\n", " chunk_data = chunk.masked_data().xarray()\n", " print(\"Shape of the chunked data array:\", chunk_data.shape)\n", " print(\"Slice [0, 0, 29:34, 126:131] from the masked data:\")\n", " print(chunk_data[0, 0, 29:34, 126:131])\n", " print()" ] } ], "metadata": { "kernelspec": { "display_name": "xfel (current)", "language": "python", "name": "xfel-current" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 4 }