Hadrian: Lord of Hope:

Hadrian Bothwell was never an excellent fit with a religious calling, so when his titled brother asks him to take over running the family estate, Hadrian gives up vicaring and returns to Cumberland. He's spent years thinking of Lady Avis Portmaine as the woman he should have fought for, but he finds Avis has become a recluse on the neighboring estate, socially shunned, and more unavailable than ever. If Hadrian wants to win the lady's heart, he must first win her freedom from the past that continues to torment her.