who are the direct buyers or importers of mango fruit in japan? 

Here is a lead to get you started:

http://www.pakistanemb.itgo.com/FRUITS.htm

More specifically, here is the info on that site that may interest you:
Mr. M. Wada Mango imports.
M/s Nichimen Corporation, 
2-2,Nakanoshima 2-chome, Kita-ku,
Osaka, 
Tele: 81-6-223-4529
Fax : 81-6-223-4517

Another company I found is at:

http://www.st-koy.com/English/mangodeliveryeng.htm

They offer delivery of mango from the Philippines to customers in
Japan and may be contacted at the following phone number:
ST & Associates Management Co., Inc.
TEL:(63)-(0)2-817-8840/816-3575/816-3541/815-3316
FAX:(63)-(0)2-818-6997

They accept payments in both Philippine pesos and Japanese yen.