I have some love/hate groundcovers myself. I think the biggest criteria for my love/hate ratio is the combination of how it grows (fast/slow) and where it is located. i.e. not all fast growers are hated if they are in the right place
I just pulled out about 100 sq ft of pachasandra because it just got everywhere. I will miss it in the winter. (pre-snowcover).
I have a huge patch of bunchberry (Cornus canadensis) that spreads very quickly, but it is in a part-shade woody tree-shrub area and it really seems to fit the woodland less-manicured feel. I also suspect it would be hard to eradicate if I wanted to. It has those thin roots that go everwhere and break easily. But I love it where it is.
I have creeping veronica Waterperry in a perennial bed and it stays put nicely. Love it. I also have Georgia Blue but it is much more aggressive. I have to pull it back severely twice a year. Dislike it (but not enough to remove it). I have a gold variegated one that barely grows. I moved it this year to a new place it may do better.
Lamium 'Pink Nancy' wanders and blends in nicely. It pulls out easily if it wanders too far. Love it.
Ajuga, love it in bloom (have Choc Chip and Crispa Metallica). Neutral otherwise. Need to have something interesting (and strong) nearby. I'm trying out Hosta Kabitan.
Thymes... love the daintiness... hate that they don't suppress weeds so well.
Geranium Biokova... I consider that more as a specimen perennial than a groundcover, but I can see it is somewhat of both. Love it.
Sedge, Treasure Island...grows slowly, love it because it can go anywhere safely, but unless it is cut back with other perennials, it looks messy far into spring until the new growth covers the old stuff. Cutting back a groundcover is not a fun thing. I used to when I had a little, but now its out of the question.
I enjoyed and saved this old thread about groundcovers in link below. I haven't read it in a while, but it might be interesting after reading this thread.
Q