For your terrain mask you need to make sure that you open up your height map (Settings on RGB colour) and use these colours:
R - 255, 0, 0. (The main terrain texture for the map)
G - 0, 255, 0. (The second terrain texture in your map)
B - 0, 0, 255. (The third terrain texture in your map)
For the terrain map you can use new layers, but you must flatten the image once saving it, don't merge it.