excessive latency with

Hello, help? I have an linksys adaptor and router, arris modem but am getting 
intermittent excessive latency reading with my voip,with vonage. I've 
replaced the router, connected the modem directly to the computer (this seems 
to work well) and everything okay. I'll reboot, power down everything, reset 
the router and modem and sometimes it will take care of the problem but 
withing a short periof of time my latency ms readings with go from 100 to 500 
to 2000 ms and I can't figure out why. Called Linksys, there content that I 
can get on the inerernet and feel it is not their problem . Is this a 
software issue. ? Anybody had this problem?? Thanks, John