Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
jonas.sinjan
hrt_pipeline
Commits
1f7e0aed
Commit
1f7e0aed
authored
Jun 30, 2021
by
jonas
Browse files
database login
parent
0fd9853b
Changes
3
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
1f7e0aed
...
...
@@ -34,7 +34,7 @@ EITHER: download from the PHI Image Database (recommended): https://www2.mps.mpg
Suggested filters for HRT science data:
-
**KEYWORD DETECTOR = 'HRT'**
<br
>
-
**Filename\* like \*L1_phi-hrt-ilam\***
-
**Filename\* like \*L1_phi-hrt-ilam
_date
\***
To download via the command line (eg: if you want to save the files on a server and not locally)
```
...
...
@@ -43,7 +43,21 @@ gunzip file.gz
```
Gunzip used to unpack the .gz to the file you want
<br>
OR : use download_files.py to download images from the attic repository: https://www2.mps.mpg.de/services/proton/phi/fm/attic/
Can also use
`download_from_db.py`
to perform multi download from database
Instructions:
1.
From the database find the files you wish to download
2.
Copy the 'Download File List' that the database will generate
3.
Paste into the
`file_names.txt`
file
4.
Create a
`.env`
file with your MPS windows login:
<br>
```
text=
USER_NAME =
PHIDATAPASSWORD =
```
5.
Set the target download folder in the
`download_from_db.py`
file
6.
Run the file (will require dotenv python module to be installed)
OR : use
`download_files.py`
to download images from the attic repository: https://www2.mps.mpg.de/services/proton/phi/fm/attic/
## **SETUP**
...
...
download_from_db.py
0 → 100644
View file @
1f7e0aed
import
subprocess
import
os
from
dotenv
import
load_dotenv
load_dotenv
()
username
=
os
.
environ
.
get
(
'USER_NAME'
)
password
=
os
.
environ
.
get
(
'PHIDATAPASSWORD'
)
def
get_file_list
(
text
):
text_str
=
str
(
text
)
text_split
=
text_str
.
split
(
"
\n
"
)
if
text_split
[
0
][
0
]
==
'h'
:
print
(
"contains https links"
)
print
(
text_split
)
return
text_split
file
=
open
(
'./file_names.txt'
)
text
=
file
.
read
()
text_split
=
get_file_list
(
text
)
target_directory
=
'/scratch/slam/sinjan/solo_attic_fits/fits_files/'
os
.
chdir
(
target_directory
)
print
(
f
"Downloading to target directory:
{
target_directory
}
"
)
for
file
in
text_split
:
file
=
file
.
split
(
"/"
)[
-
1
]
if
len
(
file
)
<
20
:
print
(
"Error occured with the file name, skipping to the next one"
)
pass
else
:
subprocess
.
call
([
"wget"
,
"--user"
,
f
"
{
username
}
"
,
"--password"
,
f
"
{
password
}
"
,
f
"https://www2.mps.mpg.de/services/proton/phi/imgdb/
{
file
}
"
])
subprocess
.
call
([
"gunzip"
,
f
"
{
file
}
"
])
print
(
"Download and unpacking complete"
)
file_names.txt
0 → 100644
View file @
1f7e0aed
solo_L1_phi-hrt-ilam_20200529T205144_V202106111602C_0065260200.fits.gz
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment