.. _user-guide.file-transfers: =============================================== Automating file transfers from the spectrometer =============================================== This guide gives some guidance on how you can link the spectrometer computer to the server running RTMet. It's up to you to decide how you want to do this, as it can be quite dependent on your specific setup, local network security policies, etc. It is assumed that the spectrometer computer is running Windows, and the RTMet server is running a Linux distribution. Changing the input strategy in workflow configuration ===================================================== By default, :rose:conf:`rose-suite.conf[template variables]cfg__input_strategy` is set to *internal*, which means that the workflow will look for .raw files in the :file:`raws/` subdirectory of the :term:`workflow run`. You can change this behavior by setting :rose:conf:`rose-suite.conf[template variables]cfg__input_strategy` to *local*, and defining the :rose:conf:`rose-suite.conf[template variables]cfg__local_runs_dir` config option to point to a general directory where the raw files are stored in the following fashion: for each :term:`workflow run`, you should store .raw files in a subdirectory with the same name as the run. **Exemple configuration and filetree:** .. code-block:: ini :caption: cylc-src/bioreactor-workflow/rose-suite.conf # `internal | local`. # # * *internal*: will look for .raw files in `${CYLC_WORKFLOW_RUN_DIR}/raws`. # * *local*: will look for .raw files in `{cfg__local_runs_dir}/${CYLC_WORKFLOW_RUN_NAME}`. # # :default: 'internal' # :type: str cfg__input_strategy='local' # Used if `cfg__input_strategy` is set to *local*. If there is a # subdirectory in the one defined here that shares the same name as the # `workflow run`, the workflow will look for .raw files inside. # # :optional: yes # :default: None # :type: str (path) cfg__local_runs_dir='~/documents/transfered_raw_files' .. code-block:: bash ~/documents ├── transfered_raw_files │ ├── 20240617_experiment │ │ ├── file_1.raw │ │ ├── file_2.raw │ ├── fedbatch_test_run │ │ ├── file_1.raw │ │ ├── file_2.raw │ │ ├── file_3.raw If you launch the worflow as: .. code-block:: bash # vip: compound command to validate, install and play the workflow cylc vip bioreactor-workflow -r 20240617_experiment The workflow will look for .raw files in the :file:`~/documents/transfered_raw_files/20240617_experiment` directory. The only part missing now is having the .raw files incrementally appear in the right directory when they are generated by the spectrometer. Again, it's up to you to decide how you want to do this. .. note:: When operating with `cfg__input_strategy` set to *local*, each .raw files is processed only when the next one appears in the directory, as there is no direct way to know when a .raw file has been fully written by the spectrometer. It means that the last .raw file won't be processed. Approaches for synchronized file access ======================================= Option 1: Mount a shared folder on local network with Samba ----------------------------------------------------------- #. **Set up a Samba server**: * Install and configure Samba server software on the server running RTMet. * Create a shared directory accessible by the spectrometer computer. * Set up a user account and workgroup which will have permissions for file access. #. **Access the shared folder from the spectrometer**: * On the spectrometer computer, log in as the user + workgroup created in step 1. * From Windows Explorer, map a network drive to the shared folder on the server. #. **Automate file retrieval**: * Make sure to output .raw files to subdirectories of the shared folder. * Set :rose:conf:`rose-suite.conf[template variables]cfg__local_runs_dir` to point to the shared folder. * Run the workflow using the subdirectory name as the :term:`run` name. .. seealso:: * `Getting started with Samba for interoperability`_ * `Mounting and mapping shares between Windows and Linux with Samba`_ Option 2: rsync over SSH (spectrometer as server) ------------------------------------------------- #. **Set up OpenSSH server on the spectrometer computer**: * Install the `OpenSSH Server`_ Windows extension on the spectrometer computer. * Make sure you can SSH into the spectrometer computer from the RTMet server, using the existing Windows user and password. * Set up `key-based authentication`_ for passwordless connections. * Authorize `SSH multiplexing`_ for lightweight connections. #. **Add rsync to executables**: * Install a Unix-like terminal emulator on the spectrometer computer: GitBash, Cygwin or WSL. Make it the default shell for SSH connections. * Install rsync if it's not already available. See `instructions`_ (french) for doing it on GitBash. #. **Set up a cron job for automated file transfers**: * :reporawfile:`etc/deployment/crontab` inside RTMet repository contains an example of a cron job that uses rsync to transfer files from the spectrometer to the RTMet server. * :reporawfile:`etc/deployment/root_ssh_config` contains an example of a SSH configuration file that can be used to use key-based authentication and multiplexing. Other options ------------- You have many other options to automate file transfers, such as using FTP, SCP, or other protocols. .. _Getting started with Samba for interoperability: https://www.redhat.com/sysadmin/getting-started-samba .. _Mounting and mapping shares between Windows and Linux with Samba: https://www.redhat.com/sysadmin/samba-windows-linux .. _OpenSSH Server: https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse?tabs=gui .. _key-based authentication: https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Public_Key_Authentication .. _SSH multiplexing: https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing .. _instructions: https://tech.gamuza.fr/Installer-Rsync-en-complement-de-GitBash.html