How to mount eodata using s3fs in Linux on CODE-DE

This article explains how to access EO DATA resources from a Linux virtual machine running on CODE-DE cloud by using s3fs.

The s3fs tool mounts an S3-compatible bucket as a local filesystem. On several default images, /eodata is already prepared and no manual configuration is required.

Manual configuration is needed only when /eodata is not already available, for example on the tested CentOS Stream 9 image or on custom images.

Prerequisites

Before you start, make sure that the following requirements are met.

No. 1 Account

You need a CODE-DE hosting account with access to the Horizon interface:

https://cloud.fra1-1.cloudferro.com/auth/login/?next=/

No. 2 Linux VM

You need a Linux virtual machine running on CODE-DE cloud.

The VM must be connected to the eodata network. Without access to this network, the VM cannot reach the EO DATA endpoint, even if s3fs is installed correctly.

You can create the VM from one of the default images, or you can upload and use your own custom image.

See also:

No. 3 Metadata access

For manual s3fs configuration, the VM must be able to read its OpenStack metadata.

The EO DATA access key, secret key, endpoint, bucket name, and mount point are provided through the metadata service inside the VM.

Check whether eodata is already available

First check whether your image already provides /eodata.

On the tested default images, /eodata is already available on:

  • Ubuntu,

  • AlmaLinux 9,

  • Rocky Linux 9.

On the tested CentOS Stream 9 image, /eodata is not present by default. For custom images, assume that /eodata is not present unless you configured it yourself.

Run:

ls /eodata

If the command lists EO DATA folders, the repository is already available and no manual s3fs configuration is required.

You can enter the directory and browse it:

cd /eodata
ls

The following examples show tested default images where /eodata is already present. The screenshots below show examples of the prepared /eodata directory on tested default images.

../_images/ubuntu-has-eodata-folder.png

Attention

Continue with the next section only if /eodata is not already available.

Configure s3fs manually

Use this procedure when /eodata is not already available. In the tested default-image set, this applies to CentOS Stream 9.

For custom images, use the same procedure if s3fs is available from the distribution repositories.

You will:

  • install s3fs,

  • create a credentials file,

  • create the /eodata mount point,

  • add an /etc/fstab entry,

  • mount and test the repository.

Step 1. Install s3fs

Use this step only when /eodata is not already available. On the tested default images, this applies to CentOS Stream 9.

Install s3fs-fuse from EPEL:

sudo dnf install epel-release
sudo dnf install s3fs-fuse

Verify that s3fs is available:

which s3fs

The command should return:

/usr/bin/s3fs

Use this path in the /etc/fstab entry.

Step 2. Get the EO DATA access details from metadata

On the tested CentOS Stream 9 image, get the EO DATA access details from the OpenStack metadata service.

Run:

curl http://169.254.169.254/openstack/latest/vendor_data2.json

The command returns JSON output similar to this:

{
  "nova": {
    "vmconfig": {
      "mountpoints": [
        {
          "s3_access_key": "EXAMPLE_ACCESS_KEY",
          "s3_secret_key": "EXAMPLE_SECRET_KEY",
          "s3_url": "eodata.cloudferro.com",
          "s3_bucket": "eodata",
          "repomount": "/eodata"
        }
      ]
    }
  }
}

In the output, note the following values:

  • s3_access_key,

  • s3_secret_key,

  • s3_url,

  • s3_bucket,

  • repomount.

In the tested setup, the bucket name is eodata and the mount point is /eodata.

Step 3. Create the s3fs credentials file

Create the credentials file:

nano ~/.passwd-eodata

Add the access key and secret key from the metadata output in this format:

s3_access_key:s3_secret_key

For example:

EXAMPLE_ACCESS_KEY:EXAMPLE_SECRET_KEY

Save the file and restrict its permissions:

chmod 600 ~/.passwd-eodata

The file must not be readable by other users. Otherwise, s3fs may refuse to use it.

The examples below assume that the VM user is eouser, so this file is stored as:

/home/eouser/.passwd-eodata

If your VM uses a different username, replace this path in the /etc/fstab entry.

Step 4. Create the mount point

Create the local directory where EO DATA will be mounted.

In the tested metadata output, this mount point is /eodata.

sudo mkdir -p /eodata

Step 5. Add the mount to /etc/fstab

Important

Keep the fstab entry on one line. Do not split it across several lines.

Open /etc/fstab:

sudo nano /etc/fstab

You can configure the mount in one of two ways:

  • automatic mounting during startup,

  • manual mounting only when needed.

For automatic mounting, the examples include:

  • _netdev, because this is a network filesystem,

  • nofail, so the VM can boot even if the EO DATA endpoint is temporarily unavailable,

  • x-systemd.mount-timeout=30, to avoid long boot delays.

Use this method if you want /eodata to be mounted when the VM starts.

Add this line to /etc/fstab:

/usr/bin/s3fs#eodata /eodata fuse passwd_file=/home/eouser/.passwd-eodata,_netdev,nofail,x-systemd.mount-timeout=30,allow_other,use_path_request_style,uid=0,gid=0,umask=0222,mp_umask=0222,url=eodata.cloudferro.com 0 0

After adding the line, save /etc/fstab.

The examples use /usr/bin/s3fs, which is the path returned by which s3fs on the tested CentOS Stream 9 image after installing s3fs-fuse.

If your credentials file is not /home/eouser/.passwd-eodata, replace that path as well.

Reload systemd:

sudo systemctl daemon-reload

Step 6. Mount and verify eodata

Mount /eodata:

sudo mount /eodata

Enter the directory and list its contents:

cd /eodata
ls

You should see output similar to this:

../_images/centos-has-eodata-folder.png

If EO DATA folders are listed, the mount is working.

Troubleshooting

Use the checks below if /eodata does not mount correctly.

Check endpoint access

First verify that the VM can resolve the EO DATA endpoint.

getent hosts eodata.cloudferro.com

If the endpoint cannot be resolved, check that the VM is connected to the eodata network.

If the mount command reports that s3fs cannot be found, run which s3fs and compare the result with the path used in /etc/fstab.

Check the credentials file

Verify that the credentials file exists:

ls -l /home/eouser/.passwd-eodata

The file must contain one line:

ACCESS_KEY:SECRET_KEY

The permissions must be restricted:

chmod 600 /home/eouser/.passwd-eodata

Check allow_other

The examples use allow_other, which allows users other than the mounting user to access the mounted filesystem.

If mounting fails with an allow_other error, check /etc/fuse.conf:

sudo nano /etc/fuse.conf

Make sure this line exists and is not commented out:

user_allow_other

Then try mounting again:

sudo mount /eodata

Unmount /eodata

To unmount the repository, run:

sudo umount /eodata

If the directory is busy, make sure no terminal session is inside /eodata and no process is reading files from it.

Collect debug output

For troubleshooting, you can run s3fs manually with debug options and mount EO DATA into a temporary directory.

The examples below assume that the VM user is eouser, so the credentials file is stored as /home/eouser/.passwd-eodata. If your VM uses a different username, replace this path in the commands.

Become root:

sudo -i

If /eodata is already mounted, unmount it:

umount /eodata

Create a temporary mount directory:

mkdir -p testing_dir

Mount eodata with debug output enabled.

s3fs eodata testing_dir -o passwd_file=/home/eouser/.passwd-eodata -o url=eodata.cloudferro.com -o use_path_request_style -o umask=0002 -o curldbg -d

Open another terminal session and test the temporary mount:

cd testing_dir
ls

You should see output similar to the regular /eodata listing.

../_images/eodata-ubuntu_creodias.png

The debug output is printed in the terminal where s3fs was started. Additional logs may also be available in /var/log/.

What to do next

After /eodata is mounted, you can browse and copy files with standard Linux commands:

cd /eodata
ls

To copy a selected product to your home directory, use cp:

cp /eodata/path/to/product/file.nc ~/

Note

EO DATA is object storage exposed as a filesystem. It is suitable for browsing and copying products, but large processing jobs should usually copy the required files to local storage first.