Also, using different OSs on different hardware to do different things.
Some of my best purchases have been from thrift shops. I'm still using one I bought for $80 years ago. It's running Linux, on an old Pentium 4 processor, so not cutting edge, but it supports GPU cards released much later. GPU computing is about all I use it for though; I can't find the memory to upgrade it to max RAM (it exists, but is a rara avis, so sellers jack up the price). Just bad timing -- larger DIMMS weren't released until the next memory standards were in all newer computers.
Several others I bought at auction online. If you have a distro of Linux you like, there's not much reason not to buy used computers. Oh, and I've picked up some put out in the hallway at work, and some our neighbors set on the curb, including an old LeNovo workstation which supports three full-length, full-height PCIe cards. Flushed the installed Windows, installed Linux. Have upgraded the twin CPUs twice (including from four- to six-core) and filled all twelve RAM sockets -- older server/workstation parts are abundant on eBay, and I've upgraded more as the prices have fallen.
I currently have two all-in-one iMacs (o o o o l d models) but won't buy any more Macs, new or used. Apple seems to have decided all hardware is use-once-and-dispose. Ironically, Mac was the first PC I learned to use, and my first four computers were new Macs, thanks to Apple's discounts to colleges and universities.
I absolutely DO. NOT. have to have the latest version of software, whether apps or OS, so my experience may be very different from what you're looking for. Take with a grain of salt, or so, but I'm pretty sure I've spent over $100 on a computer exactly once in this century.

ETA: University surplus property sales, often at auction. (Hat tip to hlth2be for the reminder!) Downside: they can be pretty-out-of-date models. Upside: I've bought a pair of fully functional computers (good for their day) for $5. Yes, per pair. If there's a college or university near you, find out what office handles these, and get a schedule or email updates.