I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish.

After doing some digging I can see the main cause of the slow down is file transfers. It seems it's taking ~10 seconds to transfer each file. The files are only small (configuration files) so I can't understand why they would take so long.

This is an example of a file in my manifest:

file { "/etc/rsyncd.conf" :
    owner       => "root",
    group       => "root",
    mode        => 644,
    source  => "puppet:///files/rsyncd/rsyncd.conf"
}

Running puppet-profiler I see this:

10.21s - File[/etc/rsyncd.conf]

It also seems I cannot update more than one server at once using puppet. If I run two servers at the same time then puppet takes twice as long.

I have changed the puppet master from using webrick to mongrel, but this doesn't seem to help. This is making deploying changes painful. A simple config change can take an hour to roll out to all servers.