That house is certainly consistent with the building as enumerator site 1 Cloonenagh townland on the 1911 census - same general size, and number of windows in the front, the roof was probably updated to tile at some stage, but the basics match. On the aerial view you can see further structure behind the cottage, likely also a later addition. The house is just inside Cloonenagh townland but right at the border...
I’m pretty certain that the photo you have matches the building at this location, but from the current clues I dont think it’s possible to be 100% certain that this is the same as that Duffy household on the 1911 census - how certain are you that the photo is of your Duffy house ?
The fact that this house is located in the area where there’s short border (~200 metres) between the townlands of Cuslough Demesne and Cloonenagh might account for the switching back and forth… if you walked out the front door into the middle of the road from the house in the google photo you would probably cross the border between the two townlands! i.e. Cloonenagh to Cuslough
Creagh road is on the western edge of the town of Ballinrobe, heads roughly towards Cuslough Demesne..
It should be possible to follow the household using the Valuation Office records, i.e Cancelled Land Books, these carry forward into the 20th Century from Griffith’s Primary Valuation and contain similar information, noting changes to occupier and immediate lessor, new buildings etc, there are also maps to go with these changes. These Valuation records are available in their research room in Abbey Street, Dublin city, they are not available online.