Are royalties due on songs used in nightclubs/bars? What about
broadway and off-broadway shows? For example, i'm sure that there are
royalty contracts used if the appeal of a show is the use of the songs by a
particular artist as is the case for ABBA (Mamma Mia) or billy joel or
Walt Disney. But what about a show where the song is ancillary but
included in the performance? Background: This is research for a new
entertainment venue in the New York Tri-state area which will be a
nightclub / danceclub / off-broadway show hybrid and I'm having
difficulty figuring out what the dividing line is for when royalties
would be due and when permission needs to be sought. Many of the same
songs may be repeated performance after performance. Links to further
research would be appreciated.

Here are a few sites that should help you understand royalties:

http://www.filmmusicworld.com/faq/6.1.html

http://www.squarewheels.com/content2/ascap.html

http://entertainment.howstuffworks.com/music-royalties1.htm

http://cyber.law.harvard.edu/is02/readings/ascap.html

Myoarin

Royalties are always paid for songs used on Broadway, unless the songs
are in the public domain (which means they'd have to be pretty
old...generally from 1924 or older, althought he dates may vary
according to whether or not the original copyright was renewed by the
composers' estate).

In many cases, royalties are not paid for nightclub act songs.
Generally, composers don't expect individual artists to pony up for
singing their songs. However, the bigger the show, the more likely
they will expect to be paid. In addition, some composers are very
strict about getting their royalties. For example, Stephen Sondheim,
who has agents check out venues all over the U.S. (and perhaps beyond)
to ensure that his rights are fully maintained.

The best idea is always to pay the royalty.

Kriswrite

Although the case is limited to digital music sampling, Bridgeport
Music at http://pacer.ca6.uscourts.gov/cgi-bin/getopn.pl?OPINION=04a0297p.06
gives some helpful insights into relevant aspects of copyright law. 
This case is representative of current trends disfavoring a "de
minimis" analysis in favor of a more traditional analysis under fair
use.  In other words, just because you use a very small portion of a
copyrighted composition and your use is thus de minimis, that alone
will not save you if under traditional fair use analysis you are still
liable for infringement.

Depending on the song you wish to use, you will need to contact the
appropriate performing rights society, i.e., ASCAP, BMI, or SESAC, for
rights to publicly perform the copyrighted compositions.  They will
adjust the royalty based on how much of the copyrighted composition
you wish to "sample."  See http://www.ascap.com/licensing/about.html. 
BTW, ASCAP is very aggressive about seeking royalties for public
performance of songs in nightclubs and bars.