Is the game a lot tougher with only 4 characters? I don't really like having 6 in a party (that's why I loved M&M so much..) Does the game scale the difficulty based on the number of characters? Or is the game about impossible with less than 6?

I play with one character only, and although it was a little tough going in the very beginning, it's too easy now. Four sounds good. Just make sure you switch classes to get all the skills you need.