Skip to content
Snippets Groups Projects

New design using analyse steps 1 - reducing openpmd files to columnar parquet...

Merged d.bertini requested to merge development into main
1 file
+ 4
4
Compare changes
  • Side-by-side
  • Inline
+ 4
4
@@ -15,9 +15,9 @@ particle merging methods. The data is then stored as reduced datasets. This step
datasets provide by te first step. This step is usually done many times and is then naturally optimized by the reduced datasets to be read.
To perform this 2-steps approach **pp-ana** contains two main post-processing components:
- (`opmdfilter.py`) the filtering program that processes in parallel OpenPMD data files and generates [Parquet files](https://parquet.apache.org/)
- (`opmd_filter.py`) the filtering program that processes in parallel OpenPMD data files and generates [Parquet files](https://parquet.apache.org/)
for both field and particle data
- (`analyze.py`) the main analysis program that reads the generated [Parquet files](https://parquet.apache.org/)
- (`opmd_pq_reader.py`) the main analysis program that reads the generated [Parquet files](https://parquet.apache.org/)
to produce histograms and visualizations ([matplotlib](https://matplotlib.org/)).
The code is designed to leverage parallel processing using [MPI (Message Passing Interface)](https://www.open-mpi.org/) via the [mpi4py python interface](https://mpi4py.readthedocs.io/en/stable/) and [Dask](https://www.dask.org/) for efficient data handling.
@@ -34,7 +34,7 @@ The code is designed to leverage parallel processing using [MPI (Message Passing
## Main components
### Filtering Script (`opmdfilter.py`)
### Filtering Script (`opmd_filter.py`)
- **Command Line Arguments**:
- `--opmd_dir` or `-d`: Directory containing OpenPMD input files.
@@ -48,7 +48,7 @@ The code is designed to leverage parallel processing using [MPI (Message Passing
- Normalizes and filters particle data based on energy thresholds.
- Saves electric field and particle data as [Parquet files](https://parquet.apache.org/) , with metadata for field information.
### Analysis Script (`analyze.py`)
### Analysis Script (`opmd_pq_reader.py`)
- **Command Line Arguments**:
- `--pq_dir` or `-d`: Directory containing the Parquet files.
Loading