Automating file transfers from the spectrometer

This guide gives some guidance on how you can link the spectrometer computer to the server running RTMet. It’s up to you to decide how you want to do this, as it can be quite dependent on your specific setup, local network security policies, etc.

It is assumed that the spectrometer computer is running Windows, and the RTMet server is running a Linux distribution.

Changing the input strategy in workflow configuration

By default, rose-suite.conf[template variables]cfg__input_strategy is set to internal, which means that the workflow will look for .raw files in the raws/ subdirectory of the workflow run.

You can change this behavior by setting rose-suite.conf[template variables]cfg__input_strategy to local, and defining the rose-suite.conf[template variables]cfg__local_runs_dir config option to point to a general directory where the raw files are stored in the following fashion: for each workflow run, you should store .raw files in a subdirectory with the same name as the run.

Exemple configuration and filetree:

cylc-src/bioreactor-workflow/rose-suite.conf

# `internal | local`.
#
# * *internal*: will look for .raw files in `${CYLC_WORKFLOW_RUN_DIR}/raws`.
# * *local*: will look for .raw files in `{cfg__local_runs_dir}/${CYLC_WORKFLOW_RUN_NAME}`.
#
# :default: 'internal'
# :type: str
cfg__input_strategy='local'

# Used if `cfg__input_strategy` is set to *local*. If there is a
# subdirectory in the one defined here that shares the same name as the
# `workflow run`, the workflow will look for .raw files inside.
#
# :optional: yes
# :default: None
# :type: str (path)
cfg__local_runs_dir='~/documents/transfered_raw_files'

~/documents
├── transfered_raw_files
│   ├── 20240617_experiment
│   │   ├── file_1.raw
│   │   ├── file_2.raw
│   ├── fedbatch_test_run
│   │   ├── file_1.raw
│   │   ├── file_2.raw
│   │   ├── file_3.raw

If you launch the worflow as:

# vip: compound command to validate, install and play the workflow
cylc vip bioreactor-workflow -r 20240617_experiment

The workflow will look for .raw files in the ~/documents/transfered_raw_files/20240617_experiment directory.

The only part missing now is having the .raw files incrementally appear in the right directory when they are generated by the spectrometer. Again, it’s up to you to decide how you want to do this.

Note

When operating with cfg__input_strategy set to local, each .raw files is processed only when the next one appears in the directory, as there is no direct way to know when a .raw file has been fully written by the spectrometer. It means that the last .raw file won’t be processed.

Approaches for synchronized file access

Option 1: Mount a shared folder on local network with Samba

1. Set up a Samba server:

   • Install and configure Samba server software on the server running RTMet.

   • Create a shared directory accessible by the spectrometer computer.

   • Set up a user account and workgroup which will have permissions for file access.

2. Access the shared folder from the spectrometer:

   • On the spectrometer computer, log in as the user + workgroup created in step 1.

   • From Windows Explorer, map a network drive to the shared folder on the server.

3. Automate file retrieval:

   • Make sure to output .raw files to subdirectories of the shared folder.

   • Set rose-suite.conf[template variables]cfg__local_runs_dir to point to the shared folder.

   • Run the workflow using the subdirectory name as the run name.

See also

• Getting started with Samba for interoperability

• Mounting and mapping shares between Windows and Linux with Samba

Option 2: rsync over SSH (spectrometer as server)

1. Set up OpenSSH server on the spectrometer computer:

   • Install the OpenSSH Server Windows extension on the spectrometer computer.

   • Make sure you can SSH into the spectrometer computer from the RTMet server, using the existing Windows user and password.

   • Set up key-based authentication for passwordless connections.

   • Authorize SSH multiplexing for lightweight connections.

2. Add rsync to executables:

   • Install a Unix-like terminal emulator on the spectrometer computer: GitBash, Cygwin or WSL. Make it the default shell for SSH connections.

   • Install rsync if it’s not already available. See instructions (french) for doing it on GitBash.

3. Set up a cron job for automated file transfers:

   • reporawfile etc/deployment/crontab inside RTMet repository contains an example of a cron job that uses rsync to transfer files from the spectrometer to the RTMet server.

   • reporawfile etc/deployment/root_ssh_config contains an example of a SSH configuration file that can be used to use key-based authentication and multiplexing.

Other options

You have many other options to automate file transfers, such as using FTP, SCP, or other protocols.