When someone asks you why you aren't hanging out with them after work, stick to your guns.
Simply tell them your time away from work is reserved for family and friends. If they try to make you feel guilty, stand your ground. They might be snippy for a while, but ultimately they have no choice but to respect your decisions.
If they don't and it becomes an issue that starts to affect your work environment and productivity, it might be something to take up with human resources.