OK, the title is a little pretentious, maybe? But, in all fairness, for about the last 3-4 years I've been running RPGs with music. Not all of them, but most of them. And I have a few thoughts on how to do that successfully, so will pen them here.

Define "successfully"

So, what does that mean, using music in RPGs successfully? Obviously there will be different takes on this, however my definition of successful follow these main objectives:

Music should enhance the player immersion in the game. Keep in mind that the players will bring whatever attitude they've got, so like any games, don't expect them to come in with the exact same outlook and attitude as yourself. That said, choosing tracks that evoke the feeling or ambiance you are trying to convey can help steer the mood.
Music should enhance the GM immersion in the game. You are also a player, as the GM. I've found that the right track can enhance my immersion in the game as well. A well placed high-pace track can get me pumped for a combat or other high-stakes encounter. Also, plan how you will manage your tracks, because no one wants a GM who has to stop the game just to change tracks. This will pull everyone out of immersion, and have the opposite effect your are looking for!
Music should not be distracting to the players (or the GM). This is the most difficult aspect to get right. You don't want the music to drown out the players or their thoughts. It should enhance the mood, not be distracting. The easiest way to deal with a track that starts out well in helping create the mood, but quickly gets in the way, is simple reduce the volume until it's time to change to something else.
Music should never get annoying. This is perhaps the most important aspect, and closely tied in with the previous point. I keep it separate, however, because this is the next level of bad selection, where the song isn't just a little distracting, but outright annoying. This can easily happen when a song is too sharp, too loud, or just too repetitive.

A word on immersion

Many groups feel like they get plenty immersive, and have no need for music. That's great! If that works for you, and you have no desire to complicate your life, then keep on rocking on.

Additionally, I've heard of other GMs playing flavorful music (heavy metal, thematic world music, etc.) in the background while playing RPGs, and sometimes, sometimes, the music just hits it perfectly... a climax in the music that coincides with a climax moment in play. Sounds awesome, but also sounds mostly random. This is not what I'm talking about in this article. I'm concentrating on something a little more planned, and intentional.

As far immersion is concerned, I've found that music can definitely inspire the mood at the table. Looking for something mellow and expansive? Music that evokes wandering in the wilderness? A track that gives you that unsettled feeling of being in the dark underdark?

Example tracks

Here's an example of some tracks, and what I think of when I hear them:

Noble empires: Islands of Green by Benjamin Bartlett
Travelling by land: Trader's Life (from Fallout) by Mark Morgan
The dark underearth: Underground Troubles (from Fallout) by Mark Morgan
Humid forest full of life: Rain forest by Plate Mail Games
Tense explorations of ruins or crypt: Ruins from the Full Metal Jacket soundtrack

To me, the most important aspect of these tracks is that they don't easily get annoying. The cadence (or speed) and volume doesn't change dramatically, or at odd times. There is no pressure to change the story according to the music; instead the music plays a role in the background.

To that end, I normally play these tracks in a single-song loop, so they play over and over. That way I can control when I move to the next song, based on the pacing or scenes that the players are in, in the game. This makes it easy to control the tracks, but does require you to have tracks that do not get repetitive.

Background audio loops

There are places to get thematic background tracks for many situations, such as from Plate Mail Games by Wes Otis. In many cases these are perfect, as they are 10 minute tracks that loop seamlessly. Within that track there is a strong attempt to capture the mood and sounds, but without repetition, so that it avoids getting annoying. It helps that Wes is an expert sound guy with a depth of professional experience, and a huge collection of such tracks. You can find his tracks on Drive Thru RPG, if you haven't already purchased many in one of his prior kickstarters. He also has a new arrangement with Monte Cook games, with specific tacks for use with Numenera and The Strange.

You can sample the 10 minute loops for about 30 seconds a pop. And there are 100's of tracks.

There are some DYI tools out there as well, where you get to mix and match the type of sounds you want. For example, the Ambient Mixer provides a number of different settings, each of which contains a few appropriate sounds, that can then be customized for your needs.

The Ambient Mixer for "Studying at Hogwarts", with sounds for fireplace, background chatter, flipping through pages in a book, and more.

One example of an attempt to create a specifically tailored album for an RPG is The Last Parsec. This is a Savage Worlds space setting by Pinnacle, and as part of the Kickstarter for the game, they released a soundtrack, which you can now purchase separately. Personally, I don't find all the tracks that useful to me, as many have too much of an orchestral feel, but track 2, Beyond the Last Parsec, is a very evocative 5 minute track, and I've used it in a number of sci-fi and fantasy games.

Movies and soundtracks

In theory, another place to go is movie and video game soundtracks. In practice, it is very difficult to find tracks that don't change volume, cadence, or mood in a distracting manner.

Sometimes you find a gold mine, such as the Fallout soundtracks by composer Mark Morgan.

Most soundtracks are built to enhance the movie, however, and they therefore pick up pace and volume during key movie scenes, which may be inappropriate during an RPG where we don't have any planned timings. A soundtrack for a movie such as Lord of the Rings sounds like it would be a great place to get audio, however it's much more difficult to find songs that meet all of my required objectives.

Making concatenated tracks

Sometimes you can find great tracks, but they might be short. A short track isn't a problem in itself, but can more easily get repetitive and annoying. A good example is the No Escape soundtrack (by Marco Beltrami & Buck Sanders). There are some great high-paced tracks there (see tracks 6, 7, 9, 2, and 16), but at 1-2 minutes in length each, they would become very annoying very quickly. However, if you string a few of them together (for example using a free tool such as Audacity), you maintain enough variety to create a solid 9 minute loop, where the overall pace is similar, but the beat and style changes enough to keep it feeling fresh.

A slightly different, but similar approach, is to cut a track by removing an aspect of it that might not fit. You might have 6 minutes of brilliant, thematic music, and then some climax which just ruins it. Using a tool such as Audacity can make it easy to remove those parts. You too can be an audio engineer!

Finding thematic mood music in other genres

And then sometimes you just find good music. I tend to go with instrumental tracks, as lyrics can more easily distract. This can range from classic blues or jazz, to hip hop and slow techno beats, to various international "world" music tracks.

Here's some examples:

Intense space chase: Mistico by Faust and Shortee. These two are hip hop DJs, and so many tracks on their Digital Soul album are amazing.
Tribal combat: Gamal Gommaa by Sahara Saidi. This is a bellydancing track. If you get the full version, check out the temporary pace change that works magic from 2:35-3:35, which gives you that little breather you need to prevent overdose.
Creepy space medical bay: Hologrammic Dub (track 5) by DJ Spooky. There are many amazing tracks on the Songs of a Dead Dreamer album.

Playing well during game time

OK, you've collected a number of tracks. Now what?

During game time, I want to ensure that I can easily choose the track I want, on demand. This could be putting all the songs in one special playlist on my phone, or using a custom soundboard app on my phone or tablet to load specific tracks for use.

Have the tracks named appropriately. You can list them by name, if you're very familiar with the titles or artist. Alternatively, use a custom title if that helps you quickly find them, or attach them to your session or game planning. Titles such as "Orc combat scene", "Travelling between towns", and "The great forest" make it easy to hit the correct song. I normally go for a two-pronged approach, with titles such as "Orc combat (Gamal Gommaa by Sahara Saidi)", since I have a decent knowledge of the tracks, but also want that scene or game reminder of where they belong.

As an example, here is a partial view of a folder where I store tracks for a Star Frontiers space game I ran. You can see the tracks were renamed to be "in order" (roughly the order of scenes, however the players could easily skip whole parts of the ship), and always had a title, for example with track 2: "Distress Call", and also a list of songs, in this case "Hyperspace". In some cases there are multiple songs concatenated one after the other to play in a loop.

In conclusion, and what to watch out for

I won't say that everyone needs it, but I have found that in general, the audio I have used has garnered positive feedback. There are a few things to watch out for:

Mixing it up: I've found going back and forth from the simple "background audio" tracks which are very simple, to the more flavored (and sometimes high-paced) tracks, is very useful. It gives everyone little breaks from the intensity of the music. The longer role-played scenes usually benefit from mellower tracks. The high-intensity tracks can work for combat, assuming your combat doesn't last an hour. If you are playing games with long combats, try starting with a high-intensity track to set the mood, but then settle back into a background track for the long haul.
Volume controls: If you find the music is starting to get annoying, or a track is unusually loud or quiet, be ready to adjust the audio on the fly, without having to stop the game. Have those audio controls within reach so that you can make those changes immediately and seamlessly.
Audio track controls: Be ready to change tracks. Does your phone or tablet have a screenlock function that turns on automatically? This may make changing tracks time consuming and distracting. Adjust those before game time begins. Do you need to quickly be able to move to any one of many tracks? Look into a soundboard that plays music tracks, instead of a long playlist that requires you to scroll around looking for the right tune.
Get an X card, for music! The X-card is normally used to allow the table to skip a topic that some might find objectionable. Make a music card! If someone taps the card, reduce the volume or stop the track. Alternatively, just make sure that everyone is aware that the music is there to help, so if anyone finds it distracting to please speak up so that the game can be made better.
Experience and practice help immensely. Try it out. Make some mistakes. But always ask for feedback from the table afterwards, specifically about the audio, so you can get better.

One other thing: Having music inform the game

There is also another, possibly different take on all this, although I've only so far seen, and made it work, in one scenario: Just have the music play. Use the highs and lows of the music to actually inform the game and the story.

The example being Dread: Mad Max Fury Road (which I wrote about here, here, here, and here), a scenario I stole from Andy Munich, and have run a handful of times since. In this case, the entire soundtrack of Mad Max Fury Road plays in a loop, and as it does so, the GM uses it to guide the story. When the music gets mellow, the story slows down. When the music starts pumping, the narrative gets going, and shit goes down.

Would love to find similar instances where this works, but so far have come up empty... would love to hear from others if they have made something similar work.