That surely sounds counterintuitive, so it needs a lot of explanation. I'll add the caveat that it must only reward failures that were good-faith attempts with a meaningful chance of success, not just starting something and going AFK.
People sometimes talk about challenge in terms of death penalty, and cite early MMORPGs that had harsh death penalties. If every death means that you lose 5 hours of progress, and you play content with a difficulty such that you only die once per five hours, you're just treading water and not making progress. You'll quickly learn to seek out easier content. If you're playing content easy enough that you only die once per 50 hours, that's not challenging. So harsh death penalties are actually antithetical to challenging games.
So why reward failure? Let's suppose that you have choices of higher level content or lower level content. You get 50% more rewards for beating the higher level content, but nothing if you fail. However, you have only a 50% chance of success if you attempt the higher level content, but a 98% chance of success for the lower level content. Which content should you do? Some quick arithmetic reveals that you'd expect to get 31% better rewards from doing the lower level content where victory is nearly assured.
But even that assumes that they take the same amount of time, which they surely won't. Higher level mobs will take longer to kill, and will likely require players to spend longer recovering between battles. So if you go after the higher level content, you expect it to take longer to get those reduced rewards.
This can sometimes get so absurd that, instead of doing easy content that you'll beat 100% of the time, it's sometimes better to do easier content that you'll also beat 100% of the time, and only gives 2/3 as large of rewards, but you can do it in 1/2 of the time. And people wonder why so many games seem so easy.
The solution is to scale the rewards such that the most rewarding path involves doing content where victory is far from assured. I'm not saying that everything needs to be only a 50% chance of success, but I am saying that if doing content where you have a 99% chance of success gives markedly better rewards per unit time than content where you have only a 98% chance of success, your game is not going to be challenging.
This nearly requires offering substantial rewards for failure. Obviously, succeeding at content should be more rewarding than failing at the same content. And players ought not be encouraged to try things far above their level with no real hope of success. So even in failure, how close you came to success should matter tremendously to the loot you get.
But if a close failure where you had a 50% chance of success gives about the same rewards per unit time as an easier victory where you had a 99% chance of success, and succeeding at the harder content gives considerably better rewards yet, then we're on the right track. You can rescale things such that the optimal rewards involve players attempting things where they'll succeed 70% of the time or 80% or 90% or whatever, but keep it out of the high 90s if you want any semblance of challenge.
This, of course, ignores the problem of scaling challenge to group content, in which some group members are much stronger than others. But that's another topic for another time, and this post is long enough.
I posted this before the recent forum downtime. The transition seems to have eaten everything that happened for about two hours before the forums were locked, so that thread is apparently gone. As such, I'm posting it again. Feel free to repeat your comments made on the previous thread. Including the guy who came in to argue that even Care Bears think that anyone who disagrees with him should die and then got banned, and then had his ban eaten by the forum rollback.