Skip Ribbon Commands
Skip to main content
Navigate Up
Sign In

An Office of Science User Facility

MATTER IN EXTREME CONDITIONS (MEC)

MEC Data Analysis

E. Galtier, H.J. Lee and B. Nagler

Email list:
egaltier@slac.stanford.edu, haelee@slac.stanford.edu and bnagler@slac.stanford.edu, 

Conference Poster:MEC - HEDLA.pdf​​

Content

  1. Online analysis tools
  2. Offline analysis tools
  3. Physics analysis tools

Introduction

This webpage describes all the common online/real-time and offline analysis tools available at MEC, as well as some basics such as instructions on how to use these tools, how the data is managed and stored at LCLS-MEC, how to transfer the data, how to use the computing farms at LCLS etc. It contains three parts: online analysis tools, offline analysis tools and physics analysis tools, as shown below.  


Section I. Online data analysis tools we use during FEL beam-time

The Data Acquisition​ (DAQ) software we use is made in house by the LCLS Data System group, and with in the LCLS DAQ we are provided with a tool called Analysis Monitoring Interface (AMI, AMI webpage​) which can perform basic real-time analyses such as displaying detector images at specified acquisition rate, making projections of the detector image on-the-fly onto specified axis, making the trending plots of some diode read-out, so on and so forth. Your experiment POC will operate or show users how to operate DAQ and AMI during the beam-time.

Here below we show some typical examples of use of AMI, please contact the MEC team in advance (more than 2 month) of the beam-time if there are analysis tools that are essential for your beam time and are missing from the common list of tools below.
  • how the interface looks like as a whole, you will see each individual detector, such as "MecTargetChamber-0|Cspad", "MecTargetChamber-0|Princeton-2", under the list in AMI. Clicking on the detector that you want to visualizes the data will bring up a separate GUI or Window as shown next.

AMI_gui.pngAMI_GUI_2.png

  • Visualizing raw images from detectors, such as CSPAD, Princeton camera, OPAL optical cameras. Here you can do simple things such as X/Y projections of the image, saving a background file as reference, for example in "ChB", and thereafter subtract that off for every single event in "ChA" and display the background subtracted image in real time, some simple arithmetic such as multiplication, addition to the images, etc. For CSPAD detectors, this background subtraction procedure is already taken into account once you deploy a dark run before taking the signal run (a tool called calibman is available to generate background calibrations from those dark runs: calibman webpage​​) so you don't need to manually save the reference to "ChB" for example, AMI will automatically pick up those background calibrations and display the background subtracted images.
AMI_detector_image.png

  • Making projections of detector images onto X or Y axis, for spatial and spectrally resolved signals, histograming of the images, some statistically functions such as integrals, RMS, mean ,etc. 

​​AMI_projection.png

  • For "scalar" data such as read-out from some diodes such as IPM2, IPM3 (they are called ​​Beam Line Device ,Bld in the data structures), gas detector energies for beam intensities, some EPICS PV variables for photon energy for example, you can access to all of them from this "Env" GUI in AMI. AMI will give you a full list of all the recorded data (shown on the right), and you will just need to select the desired one, and some simple arithmetic (shown in the middle) can be performed to the data. Usually it's the trending plot ("v Time") that is important for aligning purposes.
AMI_Env_GUI.png
AMI-trending.png

  • If you need some customized analysis tools which are not existing in AMI, you need contact the MEC team at the very beginning of beam-time preparations. We can discuss with LCLS DAQ group or Data Analysis group to see if these request can fit into the beam-time preparation schedule. If it gets implemented, it can either be within the AMI software, or a standalone python script called ipsana (psana python interface​).
  • Even if users are doing low-shot-rate analysis, tuning may be necessary that requires high-rate analysis. Users can find many ~20 line psana-python examples here. It is important to emphasize that users should practice writing high-rate analysis scripts WELL BEFORE the experiment starts.  To practice, all the examples on the above confluence page can be run by the users.​


Section II. Offline data analysis tool, mecana GUI interface for data pre-processing, data extraction, etc.


The primary offline data analysis tool is developed by the MEC team, and it's called mecana with a GUI interface (tutorial on how to use mecana: ​mecana_instructions.pdf​). It's basically designed for user's convenience in terms of extracting data to a user-friendly or human-readable format. For example, it can automatically generate TIFF images or ASCII files for all the detectors that are contained in the raw data (XTC) files, tables of all EPICS PV variable (for example photon energy), ASCII files for the beam intensity gas detectors, diodes that monitor the relative photon flux (IPM2, IPM3), etc. The mecana GUI also gives access to the physics analysis packages such as XRD analysis, VISAR analysis, etc.


Screen Shot 2016-04-05 at 10.54.03 PM.png

It is a parallelized analysis code, with Message Passing Interface (MPI), so you can run it on these several thousand cores or CPU's available at LCLS computing farms simultaneously.

The raw recorded data are always saved in XTC format and they are saved at: /reg/d/psdm/mec/mecxxxx/xtc/, as shown below. You can access the raw data files from any psana server machines (need ssh to pslogin first, and then ssh to psana), or psexport machines. There will be 4 xtc files for each run as shown below, s00, s01, s02, s03 standfing for stream numbering in the data system, the r0001, r0002 stand for vairous run number in the DAQ system.

mecana_data.png 

mecana will then generate all these intermediate data files under: /reg/d/psdm/mec/mecxxxx/scratch/, for each run.
The purpose here is really just a data pre-processing, extraction and reduction step, so that users just need to transfer these intermediate, relatively small data files to your local storage.

mecana_data2.png

Data can be transferred to out-side of LCLS using psexport machines, with some open-source applications such as FileZilla, WinSCP, etc.


Section III. Physics analysis packages, such as XRD, VISAR analysis


MEC scientists are also developing our own experiment-specific physics analysis tools, such as X-ray diffraction (XRD) analysis code, VISAR analysis code, phase retrieval analysis for Phase Contrast Imaging (PCI) experiment, etc. These work are the outcome of a collective effort from all MEC scientists, and proper acknowledgement of these tools are expected if users intend to collaborate with us and use these analysis tools.



See conference poster:
MEC HEDLA.pdf

SLAC SLAC National Accelerator Laboratory, Menlo Park, CA
Operated by Stanford University for the U.S. Dept. of Energy