How to nicely say we don't want you to come?

My boyfriend has two sisters, one older and one younger. We used to have to split up our time between the two as they did not like the distance to each others houses and wouldn't come to ours as a midway. Lately, the sisters and their husbands have been spending more time together with or without my boyfriend and I. So now when my boyfriend and I want to spend time with the older sister and her husband, the younger and hers always seem to invite themselves along when they find out. And of course it's rude for any of us to say no... or so we think. Or maybe we should agree that it's rude on the younger sister's part to invite herself along. But this is driving us NUTS!!! Whenever the younger sister and her husband come along, they do nothing but play video games or ping pong (at least the guys) all night and this repetitiveness gets old FAST!!! I really don't want to disclude ourselves from seeing the older sister and her husband as they have a two year old daughter who we love to death and love seeing, so that would be almost like a punishment to her. That's another thing, when the younger sister and husband are around, it's as if it's a competition for them to keep her attention the entire time and not let others see her (if you know what I mean). This is getting crazy... and looks as though it will go on the rest of my life, should I marry my boyfriend.