Since the instability only occurs in one frequency band, I suspect the signal generator first. Have you tried measuring the signal generator directly without the coupler (use a 20 dB pad if power is too high) to see if it is stable at 1785 MHz? Put the power meter as close to the generator output as possible to bypass all cables and connectors in-between.
If the signal generator power is stable without the coupler and cables, then next thing to suspect would be a bad cable or connector (or connection). It is possible for a pinched or partially broken cable (intermittent strand contact) to cause problems in a narrow band. Poor connector contacts can do the same. Connect your power meter at the end of the cable where the coupler normally goes, and use the attenuator if necessary, and check for instability. If it is stable, shake and twist the cables a little bit to see if it becomes unstable.
If the signal is still stable, then the only thing left is that the couple is bad. If it was hit with too much power at some time in the past, there could be a burn inside that is selectively unstable. One way a power coupler can be over-powered is to crank a lot of power into it with the output port either open (more likely) or shorted. Doing so causes the coupler structure to have to dissipate much more power than it is designed to handle.
Let me know the outcome of these test, if you try them, please.
- Kirt Blattenberger