How to download EODATA files using s3cmd on CODE-DE
LEGACY ARTICLE
This article is marked as a legacy document and may not reflect the latest information. Please refer to the following articles:
Establishing connection
At first, please verify your connection with your EODATA buckets by using the list command.
eouser@vm01:~$ s3cmd ls
You should obtain this output.
2017-11-15 10:40 s3://DIAS
2017-11-15 10:40 s3://EODATA
If you cannot see these buckets, please complete all the steps from the following tutorial: /eodata/How-to-access-EODATA-and-Object-Storage-using-s3cmd-on-Linux-on-CODE-DE.
Downloading an EOData product
An entire EOData product is available to download as a whole directory, not just a single file. Therefore, you have to use –recursive parameter in order to avoid skipping of any files.
Example of downloading a Sentinel-2 product:
s3cmd get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/
The above command will download the S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE product to your local folder from which you are issuing the command.
Files can be also redirected to a specific local folder. For our purposes it would be a downloads directory. Add the path to your local destination folder to the command.
eouser@vm01:~$ mkdir downloads
eouser@vm01:~$ s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/ downloads/
Troubleshooting
If nothing happens or you could not download a product properly, please use -d switch to see a debug output for further diagnosis:
s3cmd -d get --recursive s3://EODATA/Sentinel-2/MSI/L2A/2022/01/30/S2B_MSIL2A_20220130T115209_N0400_R123_T28SFB_20220130T125446.SAFE/
Alternative solution
For syncing, you can use the s3cmd sync command. In synchronizes a local directory tree with S3.
Firstly we can create a new directory for our product and download that product using the s3cmd sync command on that directory.
eouser@vm01:~$ mkdir example-product
eouser@vm01:~$ s3cmd sync s3://EODATA/Sentinel-1/SAR/SLC/2018/10/01/S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE/ example-product/
Now we will present the sync method in practice.
Go to your target directory and list the files.
eouser@vm01:~$ cd example-product/
eouser@vm01:~/example-product$ ls
S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE-report-20181002T024920.pdf
annotation
manifest.safe
measurement
preview
support
For our purposes we will remove the measurement directory.
eouser@vm01:~/example-product$ rm -rf measurement/
eouser@vm01:~/example-product$ ls
S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE-report-20181002T024920.pdf
annotation
manifest.safe
preview
support
In order to sync the data with your local folder, you have to put a dot (. → location of the current directory) as a last character.
eouser@vm01:~$ s3cmd sync s3://EODATA/Sentinel-1/SAR/SLC/2018/10/01/S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE/ ./
After invoking the s3cmd sync command all the missing files will be found and restored, as presented below.
Done. Downloaded 5347560568 bytes in 23.7 seconds, 214.86 MB/s.
eouser@vm01:~/example-product$ ls
S1B_WV_SLC__1SSV_20181001T234306_20181001T235605_012965_017F28_7E23.SAFE-report-20181002T024920.pdf
annotation
manifest.safe
measurement
preview
support