For organizations that utilize a heavy number of pipelines, it is important not to overrun your server's resources such as CPU time and memory or have pipeline processes compete and collide. It is important to take in the big picture of ALL the pipelines as a whole; not just each individual pipeline.
The following are best practices to consider when coming up with your organization's pipeline strategy.
Unstage Unused Pipelines
If a pipeline is not being used, take it offline by unstaging it. When polling directory watcher pipelines are deployed but not used, they are still checking network folders for changes at their regular interval times. This utilizes unnecessary CPU, disk and network time.
With a few pipelines, the usage is negligible and unnoticed. However, for many configured and deployed pipelines, unused pipelines consume bandwidth and CPU and can cause timing conflicts with other pipelines.
Avoid Short Intervals
When using a Directory Watcher trigger for local server files, jobs are triggered based on file events utilizing resources as needed. When using the Polling Directory Watcher on network folders, Centro is checking at every interval if there are changes. This means for each pipeline there is CPU, network and disk utilization.
Consider spacing multiple pipelines at different intervals (e.g. 2 with 15 minutes, 3 with 30 minutes, 5 with one hour). This will help prevent bottleneck situations where all the pipelines are suddenly querying the file system at the same time requiring heavy work from the CPU or disk storage.
Group Pipelines
To reduce the polling against your network and server, reduce the number of pipelines by grouping like pipelines into single pipelines watching one parent folder with filters to specify desired sub folders to include. Making use of dynamic properties for various action configurations can drastically extend the versatility of a single pipeline. Similarly, the If Control action can be used as a way of branching a pipeline's behaviour depending on the location or type of the input file.
