why do prop planes go off course in autopilot nav mode

Check if your nav/gps switch is set to gps.

Nav/Gps switch set to gps - check
Autopilot nav selected - check
Autopilot on - check

and the prop planes still creep off course.
when I reset the compass all the time the prop planes go back on course but don't stay on course

Hmmmm.

What do you mean you reset the compass? That is something I am not familiar with.
What is the wind doing, when you are flying? I don't think the autopilot compensates with rudder

From what it sounds like, your DG or HSI is precessing. It happens in real planes too. The sad part is that there is nothing you can do about it other than press the "D" key, by default, to reset it every once in a while. From my experience, it happens more often and the error is worse when your simulation rate is high and/or you are flying near the magnetic poles.

But his autopilot is flying, set to GPS. And that uses gps not the HSI.

Under AIRCRAFT, SETTINGS - REALISM, check to see if the "Gyro drift" box is checked.
When flying east to west or west to east in a real airplane the gyro compas must be reset for each five to ten degrees of longitude crossed (at mid-lattitudes where most of us fly). The gyro is "pointing" at celestial north while the earth is turning in a way that effectively moves the earth's pole relative to celestial north.
The "reset" re-aligns the gyro with the magnetic compass - but really with the earth's pole.
It's diffucult to make that clear but I hope I have helped. If you uncheck "Gyro drift" I think it may solve your problem.
Please report back.
Pete