I would like to drill this point.
As you know, V1 (ichidan) verbs conjugate to られる for both their potential and passive forms. For example, 食べられる is either "can eat" (potential) or "be eaten" (passive). If you use your Rikaichan or Rikaisama, it will tell you "potential or passive."

So this hypothetical drill has sentences that have V1 in this potential or passive form, and you have to say whether the usage is potential or passive. Do you know of any drills like that?

The problem I am having is I always think it's potential when really it's potential only about 60% of the time.

They drill won;t work the way you want because in isolation native speakers get it 'wrong' too without context.

There's not sufficient information encoded to make a determination outside of context. And then with context, it's hard to get it wrong.

先生に食べられた. Even ignoring the silly idea, it's clear which case that it is.

Looking at someone: 食べられた？ Also clear which case that it is.

Of course, if horribly KY all bets are off.