Bigger problems are gerrymandering, too much money involved, and too much power of incumbency.
As a practical matter, far too few people have a real vote in choosing a representative.
Gerrymandering can make at least the winning party a nearly foregone conclusion, making the primary of the party that has a virtual lock on the district the only interesting election, if it's even contested, which it often isn't due to the power of incumbency and money issues.
Gerrymandering could be virtually eliminated by requiring districts to be "compact", I.e., in some way banning being part of a district yet wildly far from its center (there are several mathematical metrics for this). Not that we're ever likely to get a constitutional amendment for this enacted....
Too much money involved politics. Personally I think all political spending should be local. I.e., only from people who live in the district. How does it make sense for, say, some billionaire in Alabama to overwhelm an election in a district in washington state, for example?
An incumbent will always have an advantage, but something is wrong when the re-election rate is as high as it is. There's not enough accountability if the incumbent isn't worried about keeping their seat. Not sure how to fix this one, but solving the other two problems would help.
Increasing the size of the house would help, but *only* if the other problems are solved first. For instance, gerrymandering would be ever worse, as smaller districts give a partisan map designer more flexibility.