Your results will make sense only
if you take into consideration the loss of the cable
in your output. Then you will get an output power
of approx. +5dBm, the IM3 products will maintain
the same -40dBc in relation to the fundamental and
then your OIP3 will be +25dBm. By the way why do
you use such a cable anyway. These output powewr
are very low and can't damage your Spectrum Analyzer.
In addition you can use an attenuator instead of
a long cable.
It does not make sense that
the P1dB is the same as OIP3. It must be lower.
IP3 is a theoretical point which is reached by a
linear extrapolation of the fundamental and 3rd
order prodcuts slopes (Pout vs. Pin). I assume that
your P1dB is in the range of +15dBm.
The third harmonic does not increase 3dB per
every dBm of input power. I asked around and
they seem to think that the power amplifier
is a cascaded device and the stages within the
PA actually has phase cancellation which does
not result in a 3dB increase of the third harmonic
per dBm increase of input power. Is this really
true? Through this explanation, I was made to
believe that the IP3 cascaded formula is not
accurate as it assumes distortion as being additive.
The IP3 formula relates only to the 2f2-f1,
2f1-f2 products; as these products fall into the
operation BW. Mathematically it can be proved that
the amplitude coefficients of these spectral components
have a relation of 2:1 to the fundamental signals:
If you increase the amplitude of f1 by 1dB then
the 2f1-f2 will increase by 3dB.
harmonics signals (3f1, 3f2) have a different amplitude
coefficient therefore their behavior is different.