I heard that after the open ended finale of Star Trek Deep Space Nine,
the story continued in either a book or comic. Could you tell me what
the title of these books were?

Hello truck37-ga,

Thank-you for your question.

You are correct that a number of books continued the Deep Space 9
(DS9) after the television series stopped.  The books are often called
the "relaunch" series.

The books are:

* A Stitch in Time, by Andrew Robinson (May 2000)
* N-Vector, by K.W. Jeter (August 2000 - November 2000): a four issue comic book 
* The Lives of Dax (1999, reprinted 2003)
* Avatar, 2 books (S.D. Perry, May 2001)
* Section 31: Abyss, (Jeffery Lang and David Weddle, July 2001)
* Gateways 4: Demons of Air and Darkness (Keith R.A. DeCandido,
September 2001)    * Gateways 7: What Lay Beyond - "Horn And Ivory"
(Keith R.A. DeCandido, 2001)
* Mission Gamma series
  o Twilight (David R. George III, September 2002)
  o This Gray Spirit (Heather Jarman, September 2002)
  o Cathedral (Michael A. Martin and Andy Mangels, October 2002)
  o Lesser Evil (Robert Simpson, November 2002).
* Rising Son (S.D. Perry, January 2003)
* The Left Hand of Destiny (2 books, J.G. Hertzler and Jeffrey Lang,
May and June 2003)
* Unity (S.D. Perry, November 2003)
* Worlds of Deep Space Nine, vol 1: Cardassia and Andor (May 2004)
* Worlds of Deep Space Nine, vol 2: Trill and Bajor (February 2005)
* Worlds of Deep Space Nine, vol 3: The Dominion and Ferenginar (February 2005)
* Warpath (David Mack, April 2006)
* Fearful Symmetry (author unknown, April 2007)

Compiled with information from:
http://en.wikipedia.org/wiki/Deep_Space_Nine_relaunch
http://memory-alpha.org/en/wiki/Deep_Space_Nine_relaunch

I hope this answers your question but should you require any further
assistance please do not hesitate to ask for clarification.