how do you know when one should be used instead of the other?

There's not much of a difference. There's a preference for が in relative clauses that contain other の's, and also when using の could create ambiguity between the possessive/descriptive の and the subordinate-clause-が-alternate の.

I have a feeling that a lot of this is simply idiomatic, so you may just have to learn where native speakers tend to use の -- google says that やりがいのある仕事 outnumbers やりがいがある仕事 by about 3:1.

I don't recall ever seeing any detailed explanation, in a grammar book or linguistics paper, about this difference.

I seem to recall a note about this in Read Real Japanese Fiction....

Quote:

Quote:だれかの忘れた子供用な靴

The first の in this clause could easily be replaced by が. This would, however, lead to a subtle difference in meaning. In the sentence as it is now, の creates a stronger bond between だれか and 靴; if の were replaced by が, there would be a stronger bond between だれか and 忘れた.

Not sure how accurate it is though.

That may be a different case because that's one of the times where の can be analyzed as either a possessive (Somebody's forgotten children shoes) or the replacement of が (Children shoes that somebody forgot). Native speakers might have a preference for one reading over the other, though.

In other cases like 目の大きい子供, the possessive reading is impossible (or at least extremely improbable), so there may be less of a difference between の and が in those cases.

The explanation I've heard is that of different emphasis. が emphasizes the preceding clause, while の is neutral.

I keep seeing this kinda thing come up where they use の where you'd expect something else. I'm slowly absorbing it and coming to terms with it that way