I have a few friends that have a quite toxic lifestyle. We love to hang around with them occasionally. They always put a mirror in front of us. Yes, they do affect our lifestyle, but not the way most would think.
Two key things here to consider - your mindset, and balance.
From a systems view perspective, any of the sides, poles is bad. You want to maintain balance and live your way.