Well, the simple answer is that you set the RBW
to be what makes sense for the region you are looking
at. For instance, if you are interested in the phase
noise at 1 KHz, you would set the RBW to maybe 100
Hz. That way the spectrum analyzer will "average"
the noise from 950 to 1050 Hz, and display that
answer at the offset of 1000 Hz. If you made the
RBW much wider, the spectrum analyzer would include
noise from much closer to the carrier, and since
the noise close to the carrier rises so quickly,
you would get an erroneous reading.
if you were out at 100 KHz offset, you could probably
use a RBW of 2 KHz of so and get a pretty accurate
VBW...you set that as small as you
can until it looks like you are "smoothing" out
important features, such as discrete spurious signals.
Practically, it is difficult to measure the
phase noise of a VCO close to the carrier, since
it is probably hopping around in center frequency,
so there is a big change from sweep to sweep. Tried
and true tricks: power the VCO and the VCO tune
line off of a battery supply (no ground loops),
shield the vco to keep out stray field pickup, put
a piece of cardboard over the VCO to keep away air
drafts, use an inside/outside DC block to connect
the VCO to the Spectrum Analyzer, use a 10 dB attenuator
to isolate the vco from the poor spectrum analyzer
input impedance. Sometimes I hang some mondo bypass
caps, like 200-4700uf, across the VCO power supply
and tune lines to tame it during such a test.
If it is still hopping around too much free-running,
you might have to phase lock it in a very narrow
badwidth to something more stable. You might want
to try a 10 Hz loop bandwidth, for instance.
Since VCO's typcially have poor phase noise,
you can also make a very accurate measurement of
their phase noise using a frequency discriminator
technique. In this method, the VCO can hop all over
the place free running, but you still get a good
indication of close-to-the-carrier phase noise.
If you want to be very precise, you have
to apply a small correction factor, due to the fact
that 1 KHz RBW does not in fact use a 1 KHz wide
bandwidth brick wall filter. The user manual will
give you the correction factor.
You, of course,
have to calibrate the data with each change in RBW
setting. For instance if you measure -100 dBc at
10 KHz offset in a 1KHz RBW, the actual phase noise
would be -100 dBc -(10 Log 1000) = -100 dBc -30
dB = -130 dbc/Hz.