Have you ever had to do this? Have you had to tell someone that you no longer wish to make room for them on your dance card? 

Sometimes these people have been with you for a very long time. It is not an easy step, but eventually YOU decide that this relationship no longer serves YOU. This, of course, is IT, this is the answer, this is how you make the decision.... does it serve YOU? Is it a one-sided affair? Are you working yourself into a frenzy trying to keep the status quo, and the other person is merely taking advantage? 

"No one can take advantage of you without your permission"

 Eleanor Roosevelt

This is a large part of taking care of yourself, not letting anyone disturb your peace. If someone is not 'there' for you, you are wasting your time sticking around. They have shown you who they are... believe them. They will only keep it up, if you allow it. If they are causing you pain in any way, you don't need to lie down and let them step on your face. 

Once you realize that the relationship has very little to do with you, that the other person is there strictly for their own agenda; then, the decision is quite easy. The actual confrontation may not be so. It is never easy to eliminate long standing connections. 

You will probably have doubts after the fact, but it is your life, you only have control over what/who you allow in.  One thing for certain, you will be happier without toxic people around you. Your happiness is of utmost importance to you and everyone around you.

Once you learn how to be happy, you won't tolerate being around people who make you feel anything less.