A vanity publisher is generally defined as a publisher which makes most of its income from the authors it publishes, rather than from selling the books it publishes on to new readers.
Because vanity publishers don’t have to worry about attracting readers to their books, they don’t have to compete against the thousands of books on bookshop shelves in order to make their sales: so they don’t have any real motivation to produce a high-quality commercial product. They rarely put much time, effort or money into editing, typesetting or designing their books, and so most vanity-published books are very sorry affairs—regardless of the quality of the writing between their covers.
Vanity publishers used to charge high up-front fees but as writers learned to avoid publishers which operated in this way, the vanity publishers adapted their behaviour in order to become harder to detect. Some charge no upfront fees and rely on their authors buying lots of copies of their own, often-overpriced, books; some ask for “author investments” in order to get the books into production; others offer editing or promotional services for a price—two things which mainstream publishers do at no charge to the writers they publish. Perhaps not surprisingly, most are reluctant to admit to being a vanity publisher and instead use terms like subsidy publisher, full-service publisher and even “traditional” publisher—a term which isn’t used by any of the reputable publishers I know of, and which was coined by the notorious Publish America, a notorious vanity press.