Vanity publishing happens when a writer pays to be published. The term was first used in 1959 by Johnathon Clifford, who has campaigned against this nasty practice ever since. His website is well worth reading.
Here’s the official definition of vanity publishing, as accepted by the UK Advertising Standards Authority:
Vanity publishing, also self-styled (often incorrectly) as “subsidy”, “joint-venture”, “shared-responsibility”, or even “self” publishing, is a service whereby authors are charged to have their work published. Vanity publishers generally offer to publish a book for a specific fee or offer to include short stories, poems or other literary or artistic material in an anthology, which the authors are then invited to buy.
There’s a lot to remember there, but the important point is that any publisher which makes the majority of its income by selling books to its writers, rather than to bookshops or readers, is a vanity publisher.
Publishers which don’t charge their writers upfront fees, but instead encourage the writer to buy a few hundred copies of the book to sell for themselves, are still vanity publishers. The writers might pay their money out at the end of the process rather than the beginning (hence the name “reverse vanity publishing”), but they’re still paying to be published.
I’ll let Johnathon Clifford finish this off:
…any company that wants to charge you to publish your book is – by definition – a vanity publisher, whatever they may try to tell you to the contrary.
(Many thanks to the illustrious Sally Zigmond for the research for this piece.)