The first thing to check is the memory allocation of your firmware. The default internal monitor is loaded from 0x0000-0x1075. When downloading firmware, make sure the code is not trampling over the monitor code. The document README.txt from location C:\Cypress\USB\Target\Monitor(once you have downloaded and installed CY3684) contains information regarding the memory placement of each of the monitor files.
The application note Setting Up, Using, and Troubleshooting the Keil(TM) Debugger Environment - AN42499 lists a number of other factors you can check to troubleshoot the debugger.