aeBrokenEdges is a very light gizmo with which you can smartly break the edges of your mattes, adapting to the luminosity of the plate. Here you can see a video explaining how to use it, you may find a more updated version on nukepedia or github.