{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Reading data train by train\n", "\n", "If the data you want to work with is too big to load into memory all at once,\n", "one simple alternative is to process data from one train at a time.\n", "\n", "Other options such as [using Dask](dask_averaging.ipynb) may run faster,\n", "or make it easier to do certain kinds of processing.\n", "But code that iterates through the trains is probably easier to understand." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# of trains: 3392\n", "Duration: 0:05:39.2\n", "First train ID: 79726751\n", "Last train ID: 79730142\n", "\n", "16 XTDF detector modules (SPB_DET_AGIPD1M-1)\n", " e.g. module SPB_DET_AGIPD1M-1 0 : 512 x 128 pixels\n", " SPB_DET_AGIPD1M-1/DET/0CH0:xtdf\n", " 64 frames per train, up to 217088 frames total\n", "\n", "3 instrument sources (excluding XTDF detectors):\n", " - SA1_XTD2_XGM/XGM/DOOCS:output \n", " - SPB_IRU_SIDEMIC_CAM:daqOutput \n", " - SPB_XTD9_XGM/XGM/DOOCS:output \n", "\n", "13 control sources:\n", " - ACC_SYS_DOOCS/CTRL/BEAMCONDITIONS \n", " - SA1_XTD2_XGM/XGM/DOOCS \n", " - SPB_IRU_AGIPD1M/PSC/HV \n", " - SPB_IRU_AGIPD1M/TSENS/H1_T_EXTHOUS \n", " - SPB_IRU_AGIPD1M/TSENS/H2_T_EXTHOUS \n", " - SPB_IRU_AGIPD1M/TSENS/Q1_T_BLOCK \n", " - SPB_IRU_AGIPD1M/TSENS/Q2_T_BLOCK \n", " - SPB_IRU_AGIPD1M/TSENS/Q3_T_BLOCK \n", " - SPB_IRU_AGIPD1M/TSENS/Q4_T_BLOCK \n", " - SPB_IRU_AGIPD1M1/CTRL/MC1 \n", " - SPB_IRU_AGIPD1M1/CTRL/MC2 \n", " - SPB_IRU_VAC/GAUGE/GAUGE_FR_6 \n", " - SPB_XTD9_XGM/XGM/DOOCS \n", "\n" ] } ], "source": [ "from extra_data import open_run\n", "\n", "run = open_run(proposal=700000, run=2)\n", "run.info() # Show overview info about this data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To iterate through the trains in this run, we need the `.trains()` method.\n", "\n", "But first, it's always a good idea to select the sources and keys we want,\n", "so we don't waste time loading irrelevant data.\n", "Let's select the image data from all AGIPD modules:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "frozenset({'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/10CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/11CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/12CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/13CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/14CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/15CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/1CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/2CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/3CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/4CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/5CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/6CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/7CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/8CH0:xtdf',\n", " 'SPB_DET_AGIPD1M-1/DET/9CH0:xtdf'})" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sel = run.select('SPB_DET_AGIPD1M-1/DET/*CH0:xtdf', 'image.data')\n", "sel.all_sources" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [ "raises-exception" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processing train 79726751\n" ] }, { "ename": "KeyError", "evalue": "'image.data'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", "\u001b[0;32m/tmp/ipykernel_2455694/3854580742.py\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrains\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Processing train\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Detector data module 0 shape:\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'image.data'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;32mbreak\u001b[0m \u001b[0;31m# Stop after the first train to keep the demo quick\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mKeyError\u001b[0m: 'image.data'" ] } ], "source": [ "for tid, data in sel.trains():\n", " print(\"Processing train\", tid)\n", " print(\"Detector data module 0 shape:\", data['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf']['image.data'].shape)\n", "\n", " break # Stop after the first train to keep the demo quick" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Oops, we're missing data for this detector module.\n", "We can use the `require_all=True` parameter to skip over trains where some modules are missing data." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Processing train 79726787\n", "Detector data module 0 shape: (64, 2, 512, 128)\n" ] } ], "source": [ "for tid, data in sel.trains(require_all=True):\n", " print(\"Processing train\", tid)\n", " print(\"Detector data module 0 shape:\", data['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf']['image.data'].shape)\n", "\n", " break # Stop after the first train to keep the demo quick" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data for each train is organised in nested dictionaries: `data[source][key]`.\n", "As this is often used with multi-module detectors like AGIPD,\n", "the `stack_detector_data` function is a convenient way to combine data from multiple similar modules." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Detector data module 0 shape: (64, 2, 512, 128)\n", "Stacked data shape: (64, 2, 16, 512, 128)\n" ] } ], "source": [ "from extra_data import stack_detector_data\n", "\n", "for tid, data in sel.trains(require_all=True):\n", " print(\"Detector data module 0 shape:\", data['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf']['image.data'].shape)\n", " stacked = stack_detector_data(data, 'image.data')\n", " print(\"Stacked data shape:\", stacked.shape)\n", "\n", " break # Stop after the first train to keep the demo quick" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are also methods which can get one train in the same format, from either a train ID or an index within this data:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "tid, data = sel.train_from_id(79726787)\n", "tid, data = sel.train_from_index(36)" ] } ], "metadata": { "kernelspec": { "display_name": "xfel (current)", "language": "python", "name": "xfel-current" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 4 }