How to access EODATA using s3cmd on CODE-DE-Lab

When you want to access EODATA on CODE-DE-Lab cloud, one option is to mount it as a file system so that the cloud data appears as part of the local Linux directory tree. Another option, described in this article, is to access the EODATA repository directly with s3cmd on a virtual machine hosted on CODE-DE-Lab cloud.

This method is useful when you want to list directories, inspect product paths, and download selected files or full products without mounting the repository as a file system.

What We Are Going To Cover

Configuring s3cmd for access to the EODATA repository
Listing buckets and exploring the EODATA repository
Downloading single files and full products
Using sync to download products and restore missing files

Prerequisites

No. 1 Account

You need a CODE-DE-Lab hosting account with access to the Horizon interface: https://identity.cloudferro.com/auth/realms/CODE-DE3/account/.

No. 2 Obtained EC2 credentials

Sign in to Tenant Manager. Under Configuration → EODATA Access, create personal S3 credentials and save the access key and secret key. Follow the CODE-DE-Lab migration guide for the current EODATA access procedure.

No. 3 A virtual machine

You need a virtual machine hosted on CODE-DE-Lab cloud. This article uses Ubuntu 24.04 as an example. Other operating systems are outside the scope of this article.

The virtual machine must be able to connect to https://eodata.code-de.org over HTTPS.

To create a virtual machine hosted on CODE-DE-Lab cloud, follow one of these articles:

No. 4 s3cmd installed

You need to have s3cmd installed on your virtual machine.

To install it, follow this article: How to install s3cmd on Linux on CODE-DE-Lab

No. 5 Understanding the s3cmd configuration file

s3cmd stores its connection settings in configuration files. In this article, one configuration file is used only for EODATA access.

To learn more about configuration files, see: Configuration files for s3cmd command on CODE-DE-Lab

Configuring s3cmd for access to the EODATA repository

In this article, the example configuration file is /home/eouser/eodata-access. You can store it in a different location if needed.

Start the configuration process with:

s3cmd -c /home/eouser/eodata-access --configure

You will be asked a series of questions. Enter your own access key and secret key obtained in Prerequisite No. 2.

The endpoint and HTTPS values depend on the cloud region you are using.

For FRA1-3, answer the questions as follows:

Access Key [access]: YOUR_ACCESS_KEY
Secret Key [access]: YOUR_SECRET_KEY
Default Region [US]: s3.fra1-3.cloudferro.com
S3 Endpoint [s3.amazonaws.com]: eodata.code-de.org
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:
Encryption password:
Path to GPG program [/usr/bin/gpg]:
Use HTTPS protocol [Yes]: Yes
HTTP Proxy server name:

Leave the DNS-style bucket+hostname:port template empty unless your environment explicitly requires a different value.

Once you have answered all questions, s3cmd will ask whether you want to test the connection:

Test access with supplied credentials? [Y/n]

Answer Y and press Enter.

If the configuration is correct, you should see output similar to this:

Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.

You will then be asked whether you want to save the settings:

Save settings? [y/N]

Answer y and press Enter.

The configuration file should then be saved:

Configuration saved to '/home/eouser/eodata-access'

Since this file contains access credentials, it is a good idea to make it readable only by the current user:

chmod 600 /home/eouser/eodata-access

Checking the saved configuration file

After saving the configuration, open the file and verify the most important lines:

nano /home/eouser/eodata-access

For FRA1-3, verify these settings:

host_base = eodata.code-de.org
host_bucket = eodata.code-de.org
bucket_location = s3.fra1-3.cloudferro.com
use_https = True
signature_v2 = False

In these EODATA configurations, host_bucket should match host_base. Do not use a bucket-prefixed DNS template for EODATA access in these regions.

Exploring the EODATA repository

Use the eodata bucket for EODATA access.

To list the contents of the eodata bucket, execute:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/

You should get output similar to this:

DIR  s3://eodata/Aqua/
DIR  s3://eodata/C3S/
DIR  s3://eodata/CAMS/
DIR  s3://eodata/CEMS/
DIR  s3://eodata/CLMS/
DIR  s3://eodata/CLMS_archive/
DIR  s3://eodata/Envisat/
DIR  s3://eodata/Envisat-ASAR/
DIR  s3://eodata/Global-Mosaics/
DIR  s3://eodata/Jason-3/
DIR  s3://eodata/Landsat-5/
DIR  s3://eodata/Landsat-7/
DIR  s3://eodata/Landsat-8/
DIR  s3://eodata/Landsat-8-ESA/
DIR  s3://eodata/Landsat-9/
DIR  s3://eodata/SMOS/
DIR  s3://eodata/SRTM/
DIR  s3://eodata/Sentinel-1/
DIR  s3://eodata/Sentinel-1-RTC/
DIR  s3://eodata/Sentinel-2/
DIR  s3://eodata/Sentinel-3/
DIR  s3://eodata/Sentinel-5P/
DIR  s3://eodata/Sentinel-6/
DIR  s3://eodata/Suomi-NPP/
DIR  s3://eodata/Terra/
DIR  s3://eodata/TerraAqua/
DIR  s3://eodata/auxdata/

You can continue exploring the repository by listing deeper paths. For example, to list the contents of the Sentinel-1 directory, execute:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-1/

The output should look similar to this:

DIR  s3://eodata/Sentinel-1/AUX/
DIR  s3://eodata/Sentinel-1/SAR/

Ending paths with a slash when using s3cmd ls

When you use s3cmd ls on a directory-like path, end the path with a slash.

For example, this command:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/CLMS/

returns the contents of the CLMS path:

DIR  s3://eodata/CLMS/bio-geophysical/
DIR  s3://eodata/CLMS/land_cover_use_in_priority_areas/
DIR  s3://eodata/CLMS/landcover_landuse/

If you omit the final slash:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/CLMS

the output will show:

DIR  s3://eodata/CLMS/
DIR  s3://eodata/CLMS_archive/

Listing the contents of a product

To list all files in a product such as S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635, execute:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/

The output should look similar to this:

2023-02-20 07:06       DIROBJ  s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/

This output shows that the path behaves like a directory-like object. To inspect its contents or download the whole product, continue by using that path in the examples below. Before relying on a specific internal structure, it is a good idea to verify the actual downloadable contents of that path with s3cmd ls, because different EODATA resources may be exposed in slightly different ways.

Downloading files

The examples below show how to download one existing object and how to download a full product.

Example 1: Downloading a single file

First, list the available files in the directory from which you want to download data:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/

You should get output similar to this:

2025-01-24 00:24   1717236175  s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/S5P_OPER_AUX_CTMANA_20230812T000000_20230813T000000_20230820T143624.nc

In this case, the product is stored as a single file rather than as a directory containing multiple files. To download it, use the exact object path returned by the ls command:

s3cmd -c /home/eouser/eodata-access get s3://eodata/Sentinel-5P/AUX/AUX_CTMANA/2023/08/12/S5P_OPER_AUX_CTMANA_20230812T000000_20230813T000000_20230820T143624.nc

The download process should begin immediately. The file will be saved to your current working directory.

After the download is complete, you can use the ls command to verify that the file is now present locally.

Note

Do not copy the file size value, such as 1717236175, into the get command. That number is only part of the output of s3cmd ls. Use only the object path that starts with s3://.

This approach is safer than relying on a hard-coded example path, because the structure of some EODATA resources may change over time. By listing the directory first and then copying an existing object path, you make sure that the file you are trying to download is really available.

Example 2: Downloading a full product

To download all files from a product, use s3cmd get with the -r parameter, which allows recursive downloads.

For instance, to download all files from product S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635, execute this command:

s3cmd -c /home/eouser/eodata-access get -r s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635

The download process should begin. The files should be saved to your current working directory. Depending on how the selected EODATA path is exposed, the downloaded result may appear as one or more local files, directories, or both.

If you end the directory path in the command above with a slash, the contents of that directory will be placed in your current working directory. The directory itself will not be created. Therefore, if you want to download the directory together with its contents, do not end the path with a slash.

Using the sync command

You can also use s3cmd sync to download data from the EODATA repository to a selected local directory. This is useful when you want to keep a local copy of a product and restore any missing files later.

Downloading a product with sync

Assume that you want to download the product S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 from the EODATA repository to a local directory named product-path.

First, create the local directory:

mkdir -p product-path

Then execute:

s3cmd -c /home/eouser/eodata-access sync s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 ./product-path

The output should contain information about the download progress, like so:

download: 's3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc' -> './product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc'  [1 of 2]
 2156950172 of 2156950172   100% in   12s   168.37 MB/s  done
mkdir: 's3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/' -> './product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635/' [2 of 2]
Done. Downloaded 2156950172 bytes in 12.2 seconds, 168.10 MB/s.

Once the operation is completed, you can list the local directory to verify the downloaded content:

ls -R ./product-path

The output should look similar to this:

./product-path:
S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635
S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc

./product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635:

In this example, the synchronized result contains both a local .nc file and a directory created under product-path.

Syncing missing files

You can use the same sync command later to restore files that are missing from the local copy.

To test this behavior, remove the downloaded .nc file:

rm -i ./product-path/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635.nc

Using the -i parameter is optional. It asks for confirmation before removing the file.

To approve the deletion, answer the prompt with y and press Enter.

Then execute the same sync command again:

s3cmd -c /home/eouser/eodata-access sync s3://eodata/Sentinel-5P/TROPOMI/L1B/2023/02/20/S5P_OFFL_L1B_RA_BD8_20230220T011032_20230220T025202_27746_03_020100_20230220T043635 ./product-path

The missing file should be downloaded again.

By default, sync restores files that exist in the remote EODATA location but are missing locally. Local files that do not exist in the remote source are left intact unless you explicitly use options that remove them.

Troubleshooting

If this command works:

s3cmd -c /home/eouser/eodata-access ls

but this one fails:

s3cmd -c /home/eouser/eodata-access ls s3://eodata/

check the configuration file carefully.

Check that host_bucket matches host_base and that your signing region is correct:

host_base = eodata.code-de.org
host_bucket = eodata.code-de.org
bucket_location = s3.fra1-3.cloudferro.com
use_https = True
signature_v2 = False

Leave the DNS-style bucket template empty so that s3cmd uses eodata.code-de.org directly.

If two ls commands that differ only by the final slash return different-looking results, that is usually normal. In s3cmd, the presence or absence of the trailing slash affects how the path is interpreted.

To inspect the connection process in more detail, use debug mode:

s3cmd -d -c /home/eouser/eodata-access ls s3://eodata/

What To Do Next

s3cmd can also be used to access object storage from CODE-DE-Lab.

Learn more here: How to access object storage from CODE-DE-Lab using s3cmd

Also, access EODATA programmatically through Python boto3 library: How to access EODATA using boto3 on CODE-DE-Lab.