In your life, you will meet sometimes with peoples who stands against you without any (Valid) reason. Not only that, till their death they will try to let you down as much as possible. No matter how many times they fail, they will continue their evil work.
What can you do with them? How long will you fight? Maybe 99 times you will survive, but what about that 1% chance of failure. I am not asking you to step back. I am suggesting you to make a backup plan. So that any bad time you won't feel you are nowhere. It will also give you some mental relieve.