Well I don't know much about height maps since I don't make them myself, (too lazy) but I do know the basics. All I can tell you is that to smooth out edges, you can use blur and smudge tools. White is for the highest areas (hills/cliffs/ridges), black is for the lowest (lakes/oceans/valleys). Just use your creativity and I'm sure your maps will turn out great. :3 I'm assuming you have all the settings correct since you got them to work, so good job on that one.
As for the masks, I believe the colors are just Red, Green, and Blue.
Good luck with your maps to come! X3