When I playback the music videos that I've bought from itunes, they
play fine as long as I play them in the thumbnail size. But, when go
to full screen the videos are very "jerky". I have the latest versions
of itunes,quicktimes,windows updates. I have windows xp, 512 memory.

You need to troubleshoot: 

- try reinstalling iTunes, then run it again without any other
applications running. If video is seamless, try running with other
programs. The issue shouldn't be a memory problem, but you might have
malware running in the backround eating up a lot of system resources.

iTunes takes a lot of processor power when playing videos for
somereason, especially with mpeg4 and h.264.  I've had issues with my
computer running on  a Athlon 2600+.  What is your processor speed. I
wou also try closing as many un-needed windwows and programs in the
background while using iTunes for videos, that helped quite a bit for
me.

This is both CPU and Video Card dependent.

Make sure you have the latest video drivers installed.