Skip to main content
Feedback

Best practices for MFT runtime deployment

This topic includes best practice recommendations for deploying and managing MFT runtimes, focusing on load balancing and optimizing performance, including the use of multiple runtimes on the same server.

Runtime deployment and workload splitting

Distributing the workload across multiple runtimes helps prevent any single runtime from becoming a bottleneck, especially with high traffic or a large number of flows.

  • Limit flow endpoints per runtime

    • For transfers to the cloud, limit the load to around 20 flow endpoints per runtime.
    • For local-to-local transfers, limit the load to around 50 flow endpoints per runtime.
    info

    These are not strict limits, but recommended thresholds for managing load.

  • Split the workload - We recommend splitting flow endpoints between multiple runtimes. You can do this by:

    • Deploying runtimes on different servers
    • Installing multiple runtimes on the same server
    tip

    Implement this strategy by installing a new runtime on a recommended drive, such as D: with fast disk I/O, on the existing server. Then, transfer a portion of the flows to the new runtime, which will immediately reduce the load on the initial runtime.

Resource allocation and disk optimization

The MFT runtime uses an internal disk cache for all transfers, making disk resources critical for performance.

  • Fast disk I/O - The server running the runtime must have fast disk I/O for the disk cache. Standard disks (often network storage) might be too slow and cause bottlenecks.

  • Sufficient disk space - Ensure the server and the drive where the runtime is installed (including its cache) have enough disk space to handle the largest potential influx of files. Running out of local cache storage can cause processing to stop.

  • Node installation location - Avoid installing on the C: drive. Use a dedicated, fast drive like D: for the installation and cache.

  • Sufficient resources - The server's overall resources, including network bandwidth, CPU, and disk I/O, must be sufficient to handle the entire traffic load, as every file will be copied into the internal cache before being sent to its destination.

Strategic traffic splitting

While splitting by flow count is effective, strategically separating transfer types can help mitigate specific bottlenecks.

  • Isolate cloud traffic - Consider separating flows by transfer direction to prevent bottlenecks caused by cloud-side performance.

    • Place flows that read from a local source and send to the cloud on one set of runtimes.
    • Place flows that read from the cloud and send to a local source on a different set of runtimes.

  • Flow endpoint execution - Node execution is determined per flow endpoint, not per entire flow. This means you can distribute different target endpoints of a single flow across multiple runtimes without limitation.

Addressing queued and slow transfers

To proactively address issues like queued files and slow transfers, we recommend the following steps:

  • Monitor activity - Use the Activity page to monitor file progress and identify if files are being transferred as expected.
  • Investigate queuing - The primary current troubleshooting focus is on understanding why files remain in a Queued state for extended periods, as this suggests an issue with queue processing, potentially separate from file removal or permissions errors.
  • Load testing - After implementing runtime splitting and upgrading resources, conduct load testing to confirm the system can handle the expected traffic volume and determine if resource limitations have been resolved.

Troubleshooting and log gathering

  • Review logs - Review the runtime logs, focusing on periods when slowness or queued transfers were observed. Permissions errors could indicate unauthorized access exceptions. File-not-found errors could indicate that a file was removed mid-transfer.
  • File deletion - Files are automatically deleted from the source upon pickup* as part of the transfer transaction. Manually removal mid-transfer can lead to errors as the runtime attempts to process the missing file.
On this Page