The simple answer is the cult of the car. It's Century long rise to dominance and maintaining of that dominance. All of which has been led and engineered by the money behind the auto industry. The government runs on what it's paid to vote for, and the auto industry (and the related oil industry) spend HUGE amounts to keep the automobile king. It's why even the bus systems and subways in most cities suck. Profit, profit, profit.
I'm gonna stop now, cause this whole topic is dangerously close to the no politics rule...
Its more than just the "Cult of the Car", its also the lifestyle that having those individual modes of transportation have fostered.
Typical day at my house: Get up. Wife is stay-at-home. Daughter 1 and Daughter 2 head out for work. Dad heads out for work. Daughter 1 - has a part-time job 8-noon, spends 2 hrs running across town to take a Kung Fu class, then goes to her second job from 2:30 to 6. After that she meets up for small group social with 2-3 girlfriends and heads home around 8pm.
Daughter 2 has coffee with her prayer group, then heads to her job around 10am. She works her shift, then heads home.
Dad goes to work, works, leaves work at 5:30ish. On the way home I make 2-3 stops at various stores to pick up ingredients for supper and/or other things we need.
Both daughters (especially the first one) have to haul multiple totes. I have my computer bags, lunch bag, and by the time I clear the last store a half-dozen bags of groceries.
Now: How is high-speed rail going to help me with this?
Worse, the times I have lived where there was public transportation available and I tried to use it, the cost was actually MORE than driving my car cost me, plus the commute time was a lot higher.
And in addition: Who are the ridership for these HSRs? As others have pointed out, local environments (hot, cold, rainy, etc) can make walking to and from a station problematic. Unless the HSR mates with a local transit system and there are cost-effective rider plans that make riding it economically sound, it won't fly.
And lastly: Size IS a factor, despite the fact that people who advocate HSR don't want to hear about it. Not so much in regards to urban sprawl but simply the fact that people will ALWAYS want to go places, visit family, visit friends, or just go see something that requires they have a car and good roads. So the individual maintains the automobile, and the State has to maintain the road networks and infrastructure to care for it, whatever happens with the HSR system.
This last is, IMHO, the real killer, because there simply is
no cost benefit to installing the HSR and no realization of efficiencies. People will still drive. People will still need the roads and maintain an automobile (or two ... or three) and the State will still need to maintain roads and bridges. Everything installed for the HSR is OVER AND ABOVE in cost ... and most Americans won't pay what it costs to ride a train instead of just driving their car to Grandma's house for Thanksgiving.
In the end, despite the way the argument is dismissed out-of-hand by so many, it really is the size - and the way we are so spread out but still want to go see each other all the time - that makes HSR a non-starter.
That, and the cost. I have yet to see a proposal for any HSR that doesn't do bad things to local tax structures and has a per-ride price tag that makes it no different than AMTRAK.