I am thinking about investing and retiring in Panama. I am a US
citizen but my partner is from Japan and has permanent residency
status here in the US. He tells me he cannot go to Panama, is this
true?

Here's an interesting link about travel restrictions for persons with
"permenant residency" status.  It's as of 2003 so you'd want to talk
to a lawyer to confirm/update, but your partner's concern could be
related to the risk that your retirement plans would present the
danger that he would be deemed to no longer be permanently residing in
the U.S. and so lose his green card.

Galincog-ga forgot the link, but in the past green card holders had to
visit the States once a year to maintain their status.
This site is the place to find out.  It still seems so:
http://uscis.gov/graphics/howdoi/PermRes.htm
Myoarin