Depending upon your artistic talent, would paper mache' do the trick?
Which type of animals do you house within your vivarium? Are they
likely to attack the wall? If so, consider a slurry of cement poured
over a rough surface. Concrete can be coloured. Alternatively,
consider buying slabs of slate. If you house fish in your vivarium,
how about recirculating the water up and over the rock face, to create
a waterfall? If you can segment the rock face, place lights behind the
wall. Picture the effect of running water passing over the lighted (or
is that lit?) gaps within the wall. A cheap mirror placed
appropriately will add another dimension. If you're lucky, a supplier
of slate may also have fossils for sale to add to the effect.
Please post a photo of your project when you are done.