Sometimes when you make the hard choice to cut people out of your life, there is nothing wrong with them. Many times they are good, decent folk. But you can sense it in your heart that they will hold you back some how.
Life does not need to be a zero sum game though. So I will say, being feared is a priority (willing to throw someone out of a group) and being loved (having those under you have a positive impression of you) is an important compliment to the former
Fear is like a clogged funnel. With said clog, the water (your time & energy) are redirected elsewhere. But once that clog (fear) is removed, the water is not only able to flow into that funnel, but all the endless possibilities that lie beyond it.
Any highly emotional reaction indicates they've found a pressure point. And any less savory elements of society would be tempted to put pressure. As such, often times it is wise to maintain a "poker face" and keep your cards close at hand.
The truth is what it is, but often times we don't want to hear it. We may know it's true, but we don't like it when people call us short/fat/bald. Therefore, oft times, the truth turns people against us, making needless enemies.
There are going to be followers that take advantage of you when you are weak, but when you have known people for a while and have been leading them competently, when you show a weakness, (not ineptitude) it humanizes you. Showing that flash of vulnerability has a disarming effect that is difficult to explain.
Well sometimes leading means putting your foot forward and showing you care first so that others follow suit.
With any group, there is going to be an objective, trying to move from point A to point B. But you don't reach that objective by going as hard as you can
Time hack and make leaps and bounds ahead of the traditional path you by doing what few are willing. It is an investment that will pay dividends years, even decades into the future.