Data transfers are similar to archiving. The /iss/etc/init.d/datasend module configures and controls data transfers. When enabled, it schedules itself with crontab to run at configurable hours of the day, at 5 minutes past the hour. There are two stages to the transfer: the first collects data files into a compressed tar file, the second actually sends the tar file across the network.
The collectNewer function in datasend traverses a data directory looking for files newer than some timestamp and adding them to a tar file. The data directories selected for transfer are listed in a series of calls:
collectNewer $tarfile class $timestamp "\.cdf" collectNewer $tarfile prof915h $timestamp "\.cdf" collectNewer $tarfile prof915l $timestamp "\.cdf" collectNewer $tarfile rass915 $timestamp "\.cdf" collectNewer $tarfile surface_met $timestamp "\.cdf" collectNewer $tarfile surface_two $timestamp "\.cdf" collectNewer $tarfile send $timestamp collectNewer $tarfile status $timestamp collectNewer $tarfile tklog $timestamp |
In the previous implementation, the data transfer time period specified a time window, and all data files modified within that window would be transferred. In the latest version, the time period is now a maximum window size. If a timestamp file exists from the last successful data transfer, and if that timestamp is within the maximum time period, then only data new or modified files since the timestamp file will be transferred. If the timestamp file is too old, such as when transfers have failed consecutively several times, then only the maximum time period of data will be transferred. The idea is to prevent overloading a slow or expensive connection with too much backlogged or outdated data. If the connection is fast or cheap, then the time window can be increased.