Is it illegal to put comics, that is publicly available from different
websites, all together on one website? As long as no money is being
made from that website, and all of the strips shown are freely
available already on the publisher's websites?

It would be breach of copyright unless you had obtained the copyright
owners permissions.

if the items are public domain, and you keep proof, then you'll be
just fine.  If an item is 'free to distribute', or gives away
'distribution rights', then you can use them as well.

If the item doesn't have these, even if it's free of charge, they
cannot be used without breaching copyright laws.  With intellectual
property (anything you'll find on the internet) there is an assumed
copyright if one is not written.

If all else fails, most intellectual property will come with a usage
license (or at least a readme file).  When in doubt, contact the
developer via email and ask - keep the response email on record.