It is not unknown for fee-charging agents to charge their clients commission on the sales that they make to vanity presses. As vanity publishing costs the writer money, and literary agents are only paid commission on their clients’ income, it can be difficult to see how an agent could make any money doing this: but if you’re prepared to overlook good business practice and reasonable behaviour, it’s simple. Fee-charging agents don’t charge commission on the royalties that their clients earn from the sales of their books, but on the fees that their clients pay to the vanity presses.
It works like this. A fee-charging agent introduces a client to a vanity press, which then offers that client publication in return for a fee. The agent then demands that the writer pay a commission on that fee, not on any royalties they hope to make one day. The amounts involved depend on the terms of the writer/agent contract and the bullishness of the agent. So, if the vanity press charges the writer £3,500 and the agent then demands 15%, not only would the writer be stuck with a load of books that they were unlikely to ever be able to sell, but they would also be out-of-pocket to the tune of £4,025. And that’s in addition to the fees that they have already paid to their agent.
If you think that this is unlikely, or that the amount of money involved in such deals is negligible, then read Jim Fisher’s Ten Percent of Nothing: The Case of the Literary Agent from Hell. It’s an excellent profile of Dorothy Deering, a notorious fee-charging agent who ended up in jail.