Slow Server Speeds - An Interesting Problem

[wswd]

Hi guys,

I have a very interesting problem on one of our servers. Over the last few days, the network speeds have gotten ridiculously slow. Downloads (wget) that normally saturate the port are now at about 10mbps. Downloads from servers that are still close in proximity to ours (i.e. LA, Phoenix, etc.) still saturate the port.

Sounded to me like a routing or network issue, but that isn't the case. There are no dropped packets, no packet loss, etc. Our DC checked the router and switch, and everything checks out fine. Other servers on the same switch are doing great.

To further confuse matters, they put the server into rescue mode, and speeds were normal. This indicates something on the server itself that might be causing this. I'm still going to ask the datacenter to replace the ethernet cable and try a different switch/port, but in the meantime...

What the hell could be causing this? ha ha! I've exhausted pretty much every option short of reinstalling the OS, which I really don't want to do since there are clients on this server. I've run with/without firewall, flushed the iptables, turned iptables off, turned nat tracking off, reset the network card, you name it.

Anybody have any suggestions?

 

[Seraphim]

What OS?

Also is there a traphic grapher anywhere in the system that can show just your traffic? These days I leave a Cacti running that monitors incoming and outgoing bandwidth on my servers as reported by the nics so that I can spot abnormal spikes that could indicate trouble. Could even install it on the server itself if you don't have another location to monitor from, since it would still generate a graph of the network activity.

 

[wswd]

CentOS 5.8.

I do have bandwidth graphs from the port, but there is nothing unusual at all. Traffic levels are fairly low.

 

[mellow-h]

Default sysctl configuration may not help bursting you the full network potential sometimes. Have you tried tweaking your tcp memory using sysctl.conf file? As you say, your traffic levels are pretty low, this could be an issue. One other issue came in my mind is the IO wait. Do you see any significant IO wait while running the wget? If there is a high IO wait, that can potentially obstruct the network speed while running tools like wget.

 

[wswd]

No, nothing. Everything has been tweaked, no i/o wait. It's a fairly new and severely underloaded server. I've actually had 3 different admin teams look at this over the past week, plus my own, and nobody can find anything.

Unfortunately, it looks like a OS reinstall is in order.

 

[mellow-h]

One last pick, did you try disabling iptables?

 

[wswd]

One last pick, did you try disabling iptables?

Yep.