Do you gain experience from them or not? If I fail to bust out someone with 8 guards would I gain or lose exp?

You gain a small amount of experience from a fail, it's the only way to train up if you're on a horrid low jb profile to begin with.

I believe you get a skill boost but no XP. I think I remember Squishy saying that he doesn't reward failures. 

Getting a bit of a skill boost while failing could be explained by saying that even if you fail 1000 times to bust someone out, you are getting a bit better at jailbreaking and gaining skill. You are not gaining XP, though.