As a general answer, it’s supply and demand, like anything else.
When a coin is rare, supply is small by definition. This includes situations where a coin may be common in bad condition, but rare in excellent condition. The supply of the nicer coins is small, so they are much more valuable.
On the demand side, popularity has a lot to do with it. For example, US coins in general sell for more than coins of some other countries because there are more people collecting them. A rare coin from some obscure nation might sell for much less than a relatively common US coin, just because few people are interested in it.
Contrary to popular opinion, age has nothing to do with it. There are a lot of 2000-year-old coins that can be had for a few dollars each.
There are a few US coins (e.g. the 1804 silver dollar) that have sold for over a million dollars. The most expensive coin ever sold was the 1933 $20 gold piece for $7.5 million, of which there is only one legally held in private hands.