Slow times here as well, (3 seconds or so for most pages, but upwards of 10 seconds on occasion).
The ping time is not bad, 57ms to 84ms on my end, depending on the time of day.
Though I did notice that my 15KB avatar took nearly 2 seconds to load when I did not yet have it cached.
So maybe the server connection is fine, but there's something wrong with reading the stored information, such as all of the text, images/avatars, and theme data.