You might try Afterimage (Helen Humphreys), Fingersmith (Sarah Waters), Year of Wonders (Geraldine Brooks), or Master Georgie (Beryl Bainbridge). None of these are perfect fits, unfortunately.

I think Native Son has some of this dynamic, too - but I haven't read it yet. (I think Bigger Thomas is like a chauffeur or something.) Probably not very Downton Abbey in feeling though. I also haven't read The Mistress of Nothing, but I think it might work?