I probably started life a bit better off than he did, but not that much (aside from being white). My first year as a teacher I earned more than my father ever had as a farmer, and he raised 5 kids on his salary. Based on a wealth calculator, I'm currently in the top 94% of the US.
For most of my life I've made considerably less than 1,000 x my age in income. Until she was in her 50s, my spouse made roughly her age x 1,000. At that point she was terminated from her job due to cognitive issues, and has not made more than $20,000/year since then. For 13 years I earned in the very low 6 figures - then choose to leave that job for one that paid slightly more than my age x 1,000.
At the time Thomas was suggesting he might need to leave the court because it only paid $174,600, I was making $40,000/year - the most I had ever made. We had just refinanced our house down from 30 year mortgage to a 15 year mortgage, including my then recent law school debts, which we paid off early.
I consider myself embarrassingly wealthy. Much of it is luck - to be born white, to educated parents who were thrifty, at a time when education was less expensive, with parents committed to paying for a bachelor's degree for each of their children. But a fair amount of it is building on that base of luck. I've have two additional degree I paid for. My spouse and I are as thrifty as our parents were - the home we bought 35 years ago cost less than half of the mortgage we were approved for. We've never had cable TV. The youngest car we own (including our daughters) is a 2008. We've invested the maximum permitted in IRAs every year since they were introduced, and saved a good bit of our income behind that most years. We've never felt deprived (a blessing, given how many people barely get by) - but we have the luxury of being able to choose to live below our means and have deliberately chosen to do so.
So for Clarence Thomas to suggest that he is incapable of living on more than 4x what I was living on at the time (while saving enough money that I could very comfortably retire at 66) is appallingly out of touch with the every day reality of most of the people his decisions impact.