Sure, but there are plenty of us who are making a living in other ways, and developing our own apps on the side over time like you are. And there are a lot of high school kids developing apps these days. You've got a lot of competition from all ages. I think currently there are something like 75,000 iOS developers.
As for the "quit my job" stories, I don't know that they depended on one app, but the stories I've read have always centered around one app that someone sold before quitting their job. As you point out, it would take time to develop others which may or may not sell, so that's a big risk. And making $20,000 in a couple weeks is not a great reason to quit your job. I've read of people doing that. The amount you sell one month doesn't necessarily correlate to the amount you'll sell during the next.
If you've been at all involved with software development, you also know that time is not your friend. The longer you take to get an idea to market, the more likely someone else will do something similar or the winds will change and other ideas will become more appealing to your target market. And, once you do get something out, you'd best have an update in the wings ready to go soon, or you risk being leap-frogged by a copycat who may have more money or cheaper resources.
The one thing I'd suggest as you work on your project is that you try to come up with design components that are a barrier to someone easily duplicating your work. For example, if your app is designed so that you can rapidly expand it, even without App Store submissions (new downloadable maps, etc.) that may allow you to keep ahead of the copycats long enough to find a sustainable following. Assume that someone will duplicate your work, because if you get noticed at all, that will happen. If your app really sucks (which I'm sure it won't), that's likely to happen as well. Sometimes the easiest thing is to take a bad idea and fix it, and there are plenty of developers looking to do that.
The other thing to keep in mind is that there are easier apps to develop that will make more money or have higher downloads if that's really what you care about. The "iJiggle Your Mom," "Boob Job," and "300 sex position" apps come to mind. Those are all pretty simplistic apps, but there's an endless supply of teenage boys who want them. I'm not saying we need any more of those, but there's an undeniable market there if you come up with something unique (or better executed) for it.
EDIT: The "200,000 iOS developers" I originally mentioned above appears to be way off. I thought I'd heard that somewhere after the WWDC, but I'm guessing it's more like 55,000-75,000. There are over 1/2 million apps on the App Store, though. The point is, it's hard to get noticed in all that, and Apple really doesn't make it any easier by featuring random apps or whatever. If you're not tops, your probably only going to be found by the iTunes search or thanks to your own marketing.
One other thing I wanted to mention. Consider localizing your app for different languages. It's not hard to do if you design things right, and I'm convinced it can make a big difference. I've got english-only apps being downloaded in Vietnam, Russia, China, and so on, and it stands to reason that if they were localized in those languages there'd be more people downloading them.
