What did Playboy Enterprises do to gain all-rights to illustrations
created by Patrick Nagel?

I don't know if this answers your question but Playboy commissioned
Nagel to create illustrations for Playboy, so under copyright law
Playboy would own the copyrights to those illustrations it
commissioned.  Also, copyrights could be bought and sold so Playboy
could have bought the rights to other illustrations from Nagel's
estate.

In U.S. copyright law (at least from 1978 onward), the copyright in
the work does not automatically go to a person or business who
commissioned the work.

The very existence of a commission implies that Nagel was not an
employee of Playboy normally doing this sort of work for them.  That
means that for Playboy to own all rights, there had to be a written
agreement between Nagel and Playboy transferring these rights.  Such
an agreement (an assignment of rights) could be created even without a
commission.  Transfers of rights are sometimes recorded with the
Copyright Office.  You can search the records online at
www.copyright.gov.