Does the person you're in a relationship with ever frustrate you sometimes but you still love them?

I can definitely see your frustration. Who doesn't love alone time with their other half? At the same time, 4 hours with his family while he golfs doesn't sound so bad. Use those 4 hours to get to know his mother better...if you guys are going to be together for the long haul, this would be a great opportunity for you to start building a relationship with his family.