Prefer, would rather, had better

Prefer and would rather can be used interchangeably. As you indicate, Omar, when we are talking about general preferences, prefer is followed by verb-ing, thus:
• I prefer listening to music to watching TV.
• I'd rather listen to music than watch TV.

'd rather
However, when we are talking about specifics, would rather is used as an alternative to would prefer to followed by an infinitive. Would rather is very common in spoken English and is often abbreviated to 'd rather. It is used in this form with all personal pronouns:
I'd / you'd / he'd / she'd / we'd / they'd rather…
Study these examples:
• Would you like to go out for dinner tonight? ~ No, I think I'd rather eat at home / I'd prefer to eat at home.
• Would you rather drink beer or wine with the curry ~ I'd rather drink beer. What about you?
• They'd rather have the strawberries by themselves, but I'd prefer to have them with cream.

Note that would rather is followed by a bare infinitive without to, whereas prefer requires to + infinitive. Would rather (but not would prefer to) is also followed by a past tense when we want to involve other people in the action, even though it has a present or future meaning. Study the following:
• Shall we go out for dinner tonight? ~ No, I'd rather we ate at home, if you don't mind.
• Shall I write to Harry and tell him that we've sold the car? ~ I'd rather you didn't.
• My mother would rather we caught the bus, rather than walk home after the party.

Rather than means instead of and can be used in combination with would prefer to and would rather. Study the following and note the intricacies of the verb forms:
• Rather than lose precious sleep discussing it now, I think we should go to bed and talk about it in the morning.
• My mother would prefer us to email each other once a week, rather than spend half an hour on the phone every night.
• My mother would rather we emailed each other once a week instead of spending half an hour on the phone every night. In fact, she insists on it. So we'd better do that, I suppose.

'd better
Note that 'd better, which is similar structurally to 'd rather, is used to suggest necessary action. In this case however, 'd is the abbreviated form of had, not would. Like 'd rather, 'd better is followed by the bare infinitive without to. Study the following:
• We'd better not be late for the Ambasador's party. It would be unforgivable to arrive late.
• You'd better phone him and tell him that you're not going.
• They'd better buy me a Christmas present or I shall never forgive them.
Note that it is sometimes slightly threatening in tone, as in the last example. Had better is always more urgent than should or ought to and has the same force as I would advise you strongly to…. or We must / we mustn't….