You can only speculate, they did not seem to have any family, did you note the 11 year old visitor, Bride Bridgman, is she a sister of your grandfather? Labour was also cheap, so it would have been easy to employ a lot of people. Have you looked into any of them being related, perhaps through Mrs Fitzgerald, I wonder what her maiden name was, as James was not related to the head of the family, no other notation would be made.
Did James continue to bake?
I do a timeline on some of my ancestors, which means I do a chart in date order, showing dates, where & what he did, when & where children were born, marriages & was he alive when they got married, perhaps you have done this, it helps you flesh out the person & their life.
Enjoy the journey.. Carmel