Is there a maximum number of payments that the API can return for a "RequestAccountPayments" call?

Yes, a maximum of 4000 transactions will be returned per call.