this post was submitted on 17 Jan 2024
47 points (89.8% liked)

Selfhosted

40437 readers
630 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
47
rsync speed goes down over time (mander.xyz)
submitted 10 months ago by [email protected] to c/selfhosted
11 comments fedilink hide all child comments
 

What could be reasons for my rsync, which is syncing two remote servers through ssh, to slow down over time like this? It keeps happening. How to check what is the bottleneck?

top 11 comments
sorted by: hot top controversial new old
[–] [email protected] 39 points 10 months ago

You aren't giving us enough information to even speculate the answer. Are these Enterprise grade servers in a datacenter? Are these home made servers with consumer or low grade hardware you're calling servers? Are they in the same datacenter or do they go out to the Internet? What exists between the hops on the network? Is the latency consistent? What is the quality of both sides of the connection? Fiber? Wi-Fi? Mobile? Satellite?

Does it drop too nothing or just settle into a constant slower speed? What have you tried to trouble shoot? Is it only rsync or do other tests between the hosts show the same behavior?

Give us more and you might get some help. If these hosts are Linux I would start with iperf to do a more scientific test. And report to us some more info.

[–] eager_eagle 30 points 10 months ago* (last edited 10 months ago) (1 children)

Bandwidth (disk and network) is just one metric. Could it be an increase in number of IOPS due to syncing several small files?

[–] Tangent5280 1 points 10 months ago

yeah this is what i thought too. proliferation of small files.

[–] mvirts 15 points 10 months ago (1 children)

It's always the disk cache

[–] eager_eagle 10 points 10 months ago (1 children)

At ~5GB per HOUR? I don't think so

[–] surewhynotlem 4 points 10 months ago

It's the floppy disk cache

[–] Unyieldingly 9 points 10 months ago

Use a VPN to check for a bottleneck, my ISP will cap my downloads from Steam to 10MB/s with a shitty VPN i get 25+MB/s.

[–] Amity_Noceda 8 points 10 months ago

Could be ISP throttling, at least that's my experience with cross-country data transfer

[–] [email protected] 1 points 10 months ago

If there is latency look at optimization around your tcp window scaling settings.

[–] sbonds 1 points 10 months ago

I've had some luck establishing the bottleneck using strace on both the sender side and receiver side. This will show if the sending rsync is waiting on local reads or remote writes and if the receiving rsync is waiting on network reads or local writes.

This helps find the specific resources to check.