I purchased a ticket to nigeria from a travel agent called bon voyage
here in canada(winnipeg) for about 1900 and in the process of buying
it,i was informed of travel insurance but declined.Though i was told
there would be a penalty involved i declined thinking it's one of
those penalties you pay like 30 to even 100dollars for and because i
was so sure i'd be making the trip.
    unfortunately,i found i couldn't make the trip because of a visa
issue.I called the travel agent almost immediately i.e before the
scheduled time of the flight to have it rescheduled but was told it
would have to be cancelled and and a fine of about 500cdn imposed on
me. I felt that was unfair and went to their office to complain and
was told that i had already been informed of the PRICE of the
fine(this i can swear never happened) and there was nothing they could
do.The name of their wholesaler is world plus and on speaking to them
i was told i had to go to the same agency.
       My problem is I don't know who next to complain to or what i
should/can do.Thanks.

You bought a non-refundable, non-transferable ticket and unfortunatly,
needed a refund.  I would bet that there is something on your
paperwork that also explicitly states this...

Sorry but there is nothing you can do - this is how the travel industry works.

the thing is that i signed no document

You don't have to sign an agreement to be bound by it. By buying a non
refundable ticket, you are bound to its terms.
As an analogy, you never signed a document saying you would be bound
by the laws of Canada, but you are bound by them nonetheless.

thanks for the comments they've been helpful so far.The point to be
noted are I never innitiated a cancellation,I  called about an hour to
the flight to have it reschedulled because of a visa problem and they
said that changing anything before travel was not allowed only in
transit(extending my stay or missing a connection or so) and
hence,they would have to cancel and re-book and this would cost me
500cdn as a fine.

Samsonnigeria

It looks like from your last comment that the travel agency is in effect 
allowing you to change your schedule for a fee of $500. I think this is a
perfectly resonable deal and you should accept it without question. 
As the prior commentators have suggested if you have obtained a very cheap
ticket and you cannot travel for ANY REASON , the airline / travel agent are
quite within their rights  not to recompensate you for your loss.
For the amount involved (and Sorry to sound so preachy) you really 
should have bought travel insurance.
Regards
Mongolia

Indeed, travel insurance is one of the best bargains around.  When
travelling abroad, one should always purchase trip cancellation and/or
travel insurance.  The trip cancellation will refund your purchase
price in the above example, and the actual insurance will pay for
emergency evacuation, medical costs, repatriation, etc. while abroad
(costs that most domestic insurance plans do not cover).