Looking for a working configuration to get a Dell W1700 monitor
working on Fedora core3 (xorg.conf) at 1280x768. A modeline alone does
not seem to work (I still get 1024x768)

you dont need to add a modeline under the monitor.
i run at 1280X1024 and dont have a modeline for it under my monitor.
the important part is under screen, it should look roughly like this:

Section "Screen"
        Identifier "Screen0"
        Device     "Card0" #or the identifier for your video card
        Monitor    "Monitor0" #or the identifier for your monitor
        DefaultDepth 24 #pick your default color depth
        SubSection "Display"
                Viewport  0 0
                Depth     24
                Modes     "1280x1024" #specify your resolution
        EndSubSection
EndSection

you can have more than one subsection of the display for all supported
color depths.
if this doesnt help, maybe if you post the related parts of xorg.conf
i can be of more assistance.

--jeff