If both targets are illegal, Vengeful Rebirth doesn’t resolve. It will be put into your graveyard.

You can return a land card to your hand with Vengeful Rebirth. It just won’t deal any damage to the target permanent or player if you do.

If the target card in your graveyard becomes an illegal target but the target permanent or player doesn’t, Vengeful Rebirth won’t return that card to your hand and it won’t deal any damage (because you didn’t return a nonland card to your hand this way). Vengeful Rebirth will still be exiled.

If the target permanent or player becomes an illegal target but the target card in your graveyard doesn’t, Vengeful Rebirth will still return that card from your graveyard to your hand, but it won’t deal any damage. Vengeful Rebirth will then be exiled.