Your problem is due to insufficient
sampling points in the measurement. Group delay
is measured as the slope of the phase vs frequency.
If there are too few sampled points, the instrument
is confused. This occurs because a typical phase
plot slopes down to -180 degrees and then restarts
at +180 degrees. Without several points within each
360 degree cycle the phase plot will sometimes slope
upward giving a negative delay. Also, long delays
have a very high phase slope, changing rapidly vs
To illustrate: a typical phase
plot will have many points near the 180 degree crossing.
( -175,-176,-177,-178,-179,-180,+179,+178...) a
phase plot with too few samples may go -120,+140,-10...
It becomes difficult to get the correct slope because
of the missed crossings at -180/+180 degrees. The
worst case may be 1 point or less in each 360 degree
cycle. In that case the delay is no longer a usefull
measurement. The result is a plot with incorrect
slope resulting in smaller than actual delay or
even negative delay.
To get a better reading,
increase the number of frequency sampling points
and and reduce the bandwidth. For long delays it
is often necessary to measure in narrow bandwidths