Another factor that can make looking for such an extreme number of votes is that some fandoms simply do not generate that level of participation by readers.
Harry Potter, Naruto, or anything in Originals involving vampires, weres or both will have a large number of readers. But there are many wonderfully written works in smaller fandoms where getting a dozen votes is high praise indeed.
Your mileage may vary, but I prefer quality over popularity. Not that there aren't good fictions in those popular categories, but they're not as prevalent at the ratings would have you believe.