I can only speak to my own experience, which was being published via a small press with a targeted readership (LGBTQ). The original owners of the press are tremendous people, authors themselves, and I absolutely adore them. They sold the press, though, and the new owners were not as dedicated to readers or authors. The press is gone now, and I’m the process of having my stories republished with another press.
I have never self-published, so I’ll let those with more experience speak to that, but I can tell you why I went the small press route.
First, identify the publishers in your genre. Don’t pitch a slash romance to a het-oriented publisher. Keep in mind most publishers won’t touch pedophilia, incest, rape for titillation, necrophilia, bestiality, or stories that discriminate against a particular group. Some won’t want graphic sex. Most want happy-ever-after or happy-for-now endings.
One of the keys to being successfully published is a polished, edited, proofread manuscript. One of the biggest mistakes people make is thinking they can self-edit, or that their dear friend who beta-reads their fan fiction is going to make a terrific editor. What you need is a professional editor, especially when you’re new to publishing, and it can get expensive out of pocket. A good editor makes sure you don’t leave loose ends, that your chronology makes sense, and characters don’t change names mid-scene.
Same thing with the proofreader, who looks for the technical stuff like punctuation, verb tense agreement, and all that good grammar stuff. They aren’t as expensive as editors only because they don’t spend as much time with your manuscript.
There’s cover art, and the ISBN numbers, which aren’t very expensive if you buy them in bulk, but most of us aren’t going to purchase them that way. Ten ISBNs can cost $125, and you need a separate ISBN for ebook and paperback editions.
Many publishers will send books out for review, and maintain relationships with review sites and blogs for their genre, and some even take out banner ads. But most small presses expect you to help market your book actively.
Publishers provide those services for you, and in return, they keep a portion of the royalties from each sale. Given that the publisher’s just spent good money on the gamble my book will sell and return their investment, I’m very happy to share royalties, and tweet my ass off, blog, and even buy a banner ad myself.
Just read your contract carefully, and don’t be afraid to ask about anything that isn’t clear to you. Are you contracting for ebook rights only, or print rights? For how long with the contract run? Most are two to three years. Make sure you understand how to have your rights revert to you, if necessary. Investigate how royalties are paid, and how often. Third party sellers pay the publisher, who pays you, and that means those monies lag compared to a direct publisher website sale. How often do you get royalty reports? How are you paid: by check or Paypal or other means? Get in touch with other authors who publish with that small press, and ask how they like the publisher.