then you're going to have a potential rot problem where the post enters the concrete.
If you use concrete to anchor the posts, it's best to have the concrete come above the ground a little so that the surface of the concrete doesn't stay wet.
If the concrete meets the post at, or worse below ground level, then you're going to have a lot of moisture entering and leaving the wood. That leads to a crack forming between the wood and the concrete as the wood expands and contracts. Water can then get into that crack, and you add freeze-thaw damage too.
Basically, there's two good ways to use concrete on a fence:
1) build it like a deck, where the concrete comes above the ground.
2) You're in a very loose, sandy soil where the post won't stand up on its own.
If those don't apply, concrete generally doesn't make the posts last any longer than just sticking a below-grade-rated pressure-treated post in the ground.
So back to the original problem and what to do now (This assumes your concrete is at or below grade):
The metal plate in your OP works, in that it provides a stop-gap solution that will last bit, if the problem is the post leaning and not the concrete leaning. It won't last nearly as long as a new post. It won't damage the concrete in any relevant way - for the post to lean the concrete has to already be damaged, or the post has to be broken or rotted significantly.
Your other option is to replace the post - but consider whether you should bother with new concrete. It costs less to just stick the pressure-treated post in the ground, and post-in-the-ground will generally last longer than using below-grade concrete.
You get the old post out by disconnecting the fence, and then digging around the post with a shovel. You can generally get it out by digging a bit, and then levering the post back and forth until it's loose. You don't need a puller or other fancy mechanism.
If you replace the post without concrete, you're going to need to add a bunch of dirt to fill the space that used to be filled with concrete. Make sure you pack that dirt in very firmly - don't add more than 2" at a time, and smash the hell out of the dirt as you backfill.