Bearing in mind how many rights reside in every single book that’s written, should you refuse an offer if the publisher is demanding world rights in all formats? Not necessarily.
If the publisher has more chance of selling foreign rights than you do, you might well be better off granting them the rights to do so—even though it will probably mean that you earn less per deal: it’s typical for publishers to take 50% of the advance on rights deals they make themselves, and if your agent brokered the original then she’ll take her 15% commission out of your 50% share; whereas if your agent sells those rights for you directly, you’ll get all your advance minus your agent’s cut.
If you have a good agent, and she sold your book reasonably easily, she should be relatively confident that she will be able to sell your book into other markets: in which case, you might well be advised to hang onto as many of them as you can.
If your agent struggled to find your initial deal, however, and the publisher is not prepared to budge on the rights issue, then you might decide that it’s not worth jeopardising the offer you’ve been made in order to retain rights which your agent might also struggle to sell.
If you don’t have an agent you might want to look for one post-haste once you receive that first publishing offer: many agents will prioritise submissions from writers who have an offer in hand. But if you still can’t find an agent to represent you, and you’re unsure about how to proceed, find out what record the publisher has in selling foreign rights. If they don’t appear to have sold many you have a dilemma: you can’t be confident that they’ll be able to sell yours; but you are unlikely to be able to sell them yourself unless you have worldwide contacts in publishing (in which case, you probably won’t be reading this blog post!).
It all comes down to this: for a book to do as well as it can, rights should reside with whoever has the best chance of selling them appropriately. There’s no point hanging bitterly onto subsidiary rights you don’t have a hope in hell of ever selling; there’s no point selling rights to a publisher which is going to do nothing with them; and as ever, there’s no one solution which is right for us all.