Today I spent almost 30 minutes trying to debug a problem where one of our Servers was not correctly synchronizing files to Amazon S3.
First thing I tried was manually doing a s3cmd put
of the files in question, and was immediately greeted by numerous connection reset by peer
error messages.
s3cmd is so smart, it even automatically retries for 5 times while throttling the upload bandwith - still to no avail.
When down to 5 kb/s upload speed I started getting broken pipe
error messages.
Well, obviously our server has more than 5 kb/s uplink so I suspected someone broke something on the network level, but everything else was working fine.
The final clue then came when I was running `s3cmd ls` on my bucket where I finally got a meaningful error message: the difference between the request time and the current time is too large
.
Huh? Yes! Doing a date
on the server revealed that the server was almost 20 minutes behind the current time, thus the SSL connection to S3 could not be established.
Turns out we had not enabled ntp and so the server clock kept drifting for a couple of months. The solution was straightforward:
sudo ntpdate ntp.ubuntu.com sudo apt-get install ntp
The first does an immediate update (only installing ntp will take some time to sync the clock back up, so doing a manual update fixes your problem immediately), and then it installs ntp to prevent this from happening in the future.