Man I can relate 100 % to what you are saying. I've had the same feeling at jams myself. When I've jammed with friends that I've known for years it was just fun and I pretty much did what I wanted.
When I jammed with strangers, then it felt like what you described.
I definitely agree with what Wes said, you do need a little bit of organization especially if it's a bunch of people that don't play together often. If you play often with them then I think you can be a little yes structured since you'll probably be familiar with all the songs.
I think part of the reason is that you like the feeling of jamming better with close friends and I feel the same way. It's not the same with other people no matter how good you and/or they play. There's something missing on the personal level that you can only get with friends.
But if the music is important to you and I think it is you should use every oppurtunity to jam with other people as much as possible.
"It's all about stickin it to the man!"
It's a long way to the top if you want to rock n roll!