Normal is well, normal.
Hardcore is a permadeath mode (die without a scape and your character is gone).
Sandbox gives players complete freedom over their characters, so they can generate items and levels, but cannot do anything with any other player (good for testing builds).
The three modes cannot play with each other (although Normal/Hardcore can play together in Challenge and Battle), so this is why you've been getting an error. Unfortunately there's no way to really check apart from tabbing and checking the game type, although I personally write [HC] or [SB] if I'm on a mode that's not normal.