Skip to content

GitLab

Commit 4b8ad318 authored by jonas's avatar jonas
Browse files

out_rte_filename + readme update +run.py example

parent 70e9d10d
Hide whitespace changes
Inline Side-by-side
README.md
View file @ 4b8ad318
# SO/PHI-HRT PIPELINE
# **SO/PHI-HRT PIPELINE**
Reduction software for SO/PHI-HRT instrument on the ESA Solar Orbiter
## PHI-HRT data reduction
## **PHI-HRT data reduction**
1. read in science data (+scaling) open path option + open for several scans at once
2. read in flat field (+scaling)- just accepts one flat field fits file
3. read in dark field (+scaling)
......@@ -12,34 +12,43 @@ Reduction software for SO/PHI-HRT instrument on the ESA Solar Orbiter
8. prefilter correction
9. read in field stop
10. apply field stop
11. demodulate with const demod matrix <br />
a) option to output demod to fits file <br />
11. demodulate with const demod matrix <br>
a) option to output demod to fits file <br>
12. normalise to quiet sun
13. calibration <br />
a) cross talk correction <br />
(if required) b) ghost correction - **not implemented yet** <br />
14. rte inversion with cmilos <br />
a) output rte data products to fits file <br />
13. calibration <br>
a) cross talk correction <br>
(if required) b) ghost correction - **not implemented yet** <br>
14. rte inversion with cmilos <br>
a) output rte data products to fits file <br>
## DOWNLOAD FILES
#### **CONFIGURATION**
EITHER: download from the PHI Image Database (recommended): https://www2.mps.mpg.de/services/proton/phi/imgdb/ <br />
Any and all steps can be turned oon or off as you wish using the keywords in the `phihrt_pipe` function
***
## **DOWNLOAD INPUT FILES**
EITHER: download from the PHI Image Database (recommended): https://www2.mps.mpg.de/services/proton/phi/imgdb/
Suggested filters for HRT science data:
- KEYWORD DETECTOR = 'HRT' <br />
- Filename* like \*L1_phi-hrt-ilam\*
- **KEYWORD DETECTOR = 'HRT'** <br >
- **Filename\* like \*L1_phi-hrt-ilam\***
To download via the command line (eg if you want to save the files on a server and not locally)
To download via the command line (eg: if you want to save the files on a server and not locally)
```
wget --user yourusername --password yourpassword the_web_address_of_the_file_you_want_from_the_database
wget --user yourusername --password yourpassword file_web_address
gunzip file.gz
```
Gunzip used to unpack the .gz to the file you want <br />
Gunzip used to unpack the .gz to the file you want <br>
OR : use download_files.py to download images from the attic repository
## SETUP
OR : use download_files.py to download images from the attic repository: https://www2.mps.mpg.de/services/proton/phi/fm/attic/
***
## **SETUP**
1. Compile milos:
......@@ -65,12 +74,54 @@ conda create --name <env_name> --file requirements.txt
```bash
python run.py
```
***
## **OUTPUT**
#### **Demod File**
Filename: `_reduced.fits `
Shape: [Y,X,POL,WAVE]
#### **RTE products**
- File: `_rte_data_products.fits`
Shape: [6,Y,X] <br>
First Index:
- 0: Continuum Intensity
- 1: Magnetic Field Strength |B| (Gauss)
- 2: Inclination (degrees)
- 3: Azimuth (degrees)
- 4: Vlos (km/s)
- 5: Blos (Gauss) </p>
- File: `_blos_rte.fits`
Shape: [1,Y,X] <br>
First Index: <br>
- 0: Blos (Gauss) </p>
- File: `_vlos_rte.fits`
Shape: [1,Y,X] <br>
First Index: <br>
- 0: Vlos (km/s) </p>
- File: `_Icont_rte.fits`
Shape: [1,Y,X] <br>
First Index:
- 0: Continuum Intensity
***
Authors: <br />
### **Authors**: <br>
Jonas Sinjan - Max Planck Institute for Solar System Research, Goettingen, Germany
Credit: <br />
### **Credit**: <br>
- SPGPylibs for the foundation, from which it was expanded upon
- CMILOS: RTE INVERSION C code for SOPHI (based on the ILD code MILOS by D. Orozco) Author: juanp (IAA-CSIC)
\ No newline at end of file
hrt_pipe.py
View file @ 4b8ad318
......@@ -76,7 +76,7 @@ def demod_hrt(data,pmp_temp):
def phihrt_pipe(data_f, dark_f = '', flat_f = '', scale_data = True, norm_f = True, clean_f = False,
sigma = 59, flat_states = 24, prefilter_f = None,flat_c = True, dark_c = True, fs_c = True,
demod = True, norm_stokes = True, out_dir = './', out_demod_file = False,
correct_ghost = False, ItoQUV = False, ctalk_params = None, rte = False):
correct_ghost = False, ItoQUV = False, ctalk_params = None, rte = False, out_rte_filename = ''):
'''
PHI-HRT data reduction pipeline
......@@ -144,6 +144,8 @@ def phihrt_pipe(data_f, dark_f = '', flat_f = '', scale_data = True, norm_f = Tr
cross talk parameters for ItoQUV, (2,3) numpy array required: first axis: Slope, Offset (Normalised to I_c) - second axis: Q,U,V
rte: str, DEFAULT: False
invert using cmilos, options: 'RTE' for Milne Eddington Inversion, 'CE' for Classical Estimates, 'CE+RTE' for combined
out_rte_filename: str, DEFAULT = ''
if '', takes last 10 characters of input scan filename (assumes its a DID), change if want other name
Returns
-------
......@@ -982,21 +984,26 @@ def phihrt_pipe(data_f, dark_f = '', flat_f = '', scale_data = True, norm_f = Tr
rte_data_products *= field_stop[np.newaxis,start_row:start_row + data_size[0],start_col:start_col + data_size[1]] #field stop, set outside to 0
if out_rte_filename == '':
filename_root = str(file_path.split('.fits')[0][-10:])
else:
filename_root = out_rte_filename
with fits.open(file_path) as hdu_list:
hdu_list[0].data = rte_data_products
hdu_list.writeto(out_dir+str(file_path.split('.fits')[0][-10:])+'_rte_data_products.fits', overwrite=True)
hdu_list.writeto(out_dir+filename_root+'_rte_data_products.fits', overwrite=True)
with fits.open(file_path) as hdu_list:
hdu_list[0].data = rte_data_products[5,:,:]
hdu_list.writeto(out_dir+str(file_path.split('.fits')[0][-10:])+'_blos_rte.fits', overwrite=True)
hdu_list.writeto(out_dir+filename_root+'_blos_rte.fits', overwrite=True)
with fits.open(file_path) as hdu_list:
hdu_list[0].data = rte_data_products[4,:,:]
hdu_list.writeto(out_dir+str(file_path.split('.fits')[0][-10:])+'_vlos_rte.fits', overwrite=True)
hdu_list.writeto(out_dir+filename_root+'_vlos_rte.fits', overwrite=True)
with fits.open(file_path) as hdu_list:
hdu_list[0].data = rte_data_products[0,:,:]
hdu_list.writeto(out_dir+str(file_path.split('.fits')[0][-10:])+'_Icont_rte.fits', overwrite=True)
hdu_list.writeto(out_dir+filename_root+'_Icont_rte.fits', overwrite=True)
printc('--------------------------------------------------------------',bcolors.OKGREEN)
printc(f"------------- CMILOS RTE Run Time: {np.round(time.time() - start_time,3)} seconds ",bcolors.OKGREEN)
......
run.py
View file @ 4b8ad318
from hrt_pipe import phihrt_pipe
import numpy as np
sciencedata_fits_filenames = ['solo_L1_phi-hrt-ilam_20210223T170002_V202106111612C_0142230201.fits']#['solo_L0_phi-hrt-ilam_20210421T120003_V202106080929C_0144210101.fits', 'solo_L0_phi-hrt-ilam_20210424T120003_V202106141014C_0144240101.fits']
# 'solo_L0_phi-hrt-ilam_20210425T120002_V202106141020C_0144250101.fits', 'solo_L0_phi-hrt-ilam_20210426T120002_V202106162118C_0144260101.fits',
# 'solo_L0_phi-hrt-ilam_20210427T120002_V202106162052C_0144270101.fits', 'solo_L0_phi-hrt-ilam_20210427T120002_V202106171444C_0144270101.fits',
# 'solo_L0_phi-hrt-ilam_20210427T120002_V202106171517C_0144270101.fits']
#['solo_L0_phi-hrt-ilam_0667414748_V202103221851C_0142230201.fits']#['solo_L0_phi-hrt-ilam_0667414905_V202103221851C_0142230602.fits', 'solo_L0_phi-hrt-ilam_0667415054_V202103221851C_0142230603.fits', 'solo_L0_phi-hrt-ilam_0667415205_V202103221851C_0142230604.fits', 'solo_L0_phi-hrt-ilam_0667415354_V202103221851C_0142230605.fits', 'solo_L0_phi-hrt-ilam_0667415505_V202103221851C_0142230606.fits', 'solo_L0_phi-hrt-ilam_0667415654_V202103221851C_0142230607.fits', 'solo_L0_phi-hrt-ilam_0667415805_V202103221851C_0142230608.fits']#['../fits_files/solo_L0_phi-hrt-ilam_0667414748_V202103221851C_0142230201.fits']
#sciencedata_fits_filenames = ['solo_L0_phi-hrt-ilam_0667414748_V202103221851C_0142230201.fits']
#sciencedata_fits_filenames = ['solo_L0_phi-hrt-ilam_0667414905_V202103221851C_0142230602.fits', 'solo_L0_phi-hrt-ilam_0667415054_V202103221851C_0142230603.fits', 'solo_L0_phi-hrt-ilam_0667415205_V202103221851C_0142230604.fits', 'solo_L0_phi-hrt-ilam_0667415354_V202103221851C_0142230605.fits', 'solo_L0_phi-hrt-ilam_0667415505_V202103221851C_0142230606.fits', 'solo_L0_phi-hrt-ilam_0667415654_V202103221851C_0142230607.fits', 'solo_L0_phi-hrt-ilam_0667415805_V202103221851C_0142230608.fits']
sciencedata_fits_filenames = ['solo_L0_phi-hrt-ilam_20210421T120003_V202106080929C_0144210101.fits', 'solo_L0_phi-hrt-ilam_20210424T120003_V202106141014C_0144240101.fits',
'solo_L0_phi-hrt-ilam_20210425T120002_V202106141020C_0144250101.fits', 'solo_L0_phi-hrt-ilam_20210426T120002_V202106162118C_0144260101.fits',
'solo_L0_phi-hrt-ilam_20210427T120002_V202106162052C_0144270101.fits', 'solo_L0_phi-hrt-ilam_20210427T120002_V202106171444C_0144270101.fits',
'solo_L0_phi-hrt-ilam_20210427T120002_V202106171517C_0144270101.fits']
flatfield_fits_filename = '../fits_files/solo_L0_phi-hrt-flat_0667134081_V202103221851C_0162201100.fits'
darkfield_fits_filename = '../fits_files/solo_L0_phi-fdt-ilam_20200228T155100_V202002281636_0022210004_000.fits'
......@@ -27,9 +23,9 @@ prefilter_f = '../fits_files/fitted_prefilter.fits'
c_talk_params = np.zeros((2,3))
q_slope = 0#0.0038
u_slope = 0#-0.0077
v_slope = 0#-0.0009
q_slope = 0.0038
u_slope = -0.0077
v_slope = -0.0009
q_int = -0.0056 #the offset, normalised to I_c
u_int = 0.0031
......@@ -43,10 +39,12 @@ c_talk_params[1,0] = q_int
c_talk_params[1,1] = u_int
c_talk_params[1,2] = v_int
print(sciencedata_fits_filenames)
phihrt_pipe(sciencedata_fits_filenames, flat_f = flatfield_fits_filename, dark_f = darkfield_fits_filename, scale_data = True, norm_f = True, clean_f = True,
sigma = 59, flat_states = 24, norm_stokes = True, prefilter_f = prefilter_f, dark_c = True, flat_c = True,
fs_c = True, demod = False, ctalk_params = c_talk_params, ItoQUV = True, out_demod_file = True,
out_dir = '/data/slam/home/sinjan/hrt_pipe_results/L1_test/', rte = 'RTE')
fs_c = True, demod = True, ctalk_params = c_talk_params, ItoQUV = True, out_demod_file = True,
out_dir = '/path/to/files', rte = 'RTE', out_rte_filename='your_desired_rte_filename')
"""
Input Parameters:
----------
......@@ -110,4 +108,7 @@ ctalk_params: numpy arr, DEFAULT: None
rte: str, DEFAULT: False
invert using cmilos, options: 'RTE' for Milne Eddington Inversion, 'CE' for Classical Estimates, 'CE+RTE' for combined
out_rte_filename: str, DEFAULT = ''
if '', takes last 10 characters of input scan filename (assumes its a DID), change if want other name
"""
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment