The 1850 townland address for James & Brigid Sweeney is Cloonroughan which was a McGarry stronghold.
In 1826, Brigid McGarry witnessed the wedding of Mary McGarry & John Connor of Cloonroughan. In 1832, she godparents for Andrew McGarry & Mary Mee (confirming this Andrew was her brother). In 1838, she godparents for Michael Sweeny & Margaret Beirne (confirming that by then, Michael was her "brother-in-law").
In the 1835 Tithe Applotment Records for "Cloonrogan", an Andrew McGarry is listed (suggesting their address at this townland was through Brigid McGarry) with 13 Irish acres.
Andrew McGarry & Mary Mee were baptising here from 1832 and 1837; Andrew McGarry & Winifred Gormly were baptising there from 1807 to 1826 (no Andrew Jr. nor Brigid on the list of baptisms, but gaps in records are not unusual in these parts).
In 1839, a Roger Sweeney & Brigid Fihily (Feely) of Cloonroughan baptise a son, Thomas, and James Sweeny stands as godfather. This places the Sweeneys in the townland of Cloonroughan (with not enough land to be named on tax records).
In the 1857 Griffith's Valuation, Andrew McGarry at farm #1 Cloonroughan has 99 acres and a house valued at 15 shillings, which is rather surprising. (Such a substantial grazier as this, was unlikely to be resident in a house of 15 shillings, unless there was a clearing during the Famine, and even then). It is likely that Brigid and James lived in one of the unoccupied houses at #1 Cloonroughan in 1850.
Anyway, I hope this gives you enough leads to find your way home!