Went with the heightmap vote.
It's not like the making of a heightmap is the most annoying, it's just that a heightmap require so much work in order to make an interesting map in general. For sure, the different parts play it's role well when it comes to making a map(s) that really catches the players interest, but the heightmap is really the base/structure of the whole map and IS what makes a map creation stand out towards others, but it's also what you will go by when you actually place the rest of the objects into the map creating that "final" product.
People can play around with the objects all they want to really fetch the players interest, but they will never be able to get that living or creative feeling towards a map if they just use a flatgrass heightmap or plain bumps placed here and there. As a map progresses so do you in your creative thinking, and come up with new interesting things and ways how to use this little grayscale image as long as you force your mind into doing so.
I'd say the masks is probably the most annoying ones since it takes a lot of energy adjusting these, it could be a certain spot in the map that doesn't work out as you planned, or just that you did a sloppy work, and you must not only layer in the heightmap and set up your brushes opacitys correctly to get it right, but also restart the game every single time you made changes to it. If you work with the heightmap as a map progresses, so will you be forced to do with the masks, hence the annoyance.
Unless you have a map that is 100% place in the air, I'd say the heightmap is the most important part, and therefore what becomes the hardest, but of course, it could differ on how you see it. If you set the standards really high for the mesh placing it will take it's fair time.