{*} Want to be a Developer for FeralHeart? {*}
The texture mask should be what the ground looks like so it won't affect the land or water but just what the ground looks like.
I believe there should be an option somewhere to say it as higher bits which helps with smoothing. Inhad this problem myself in the past! I cant remember exactly what it's called or where the option is
Maybe try to make a smudged brush yourself, that is, try to gradually darken the places on the map where the water is to come into contact with the land. And if you're doing map mask, it's best not to use smudged brushes. It ruined my game once.