Diane,
Few RC parishes in the Fermanagh area kept records before the 1820s. (The churches were functioning, they just didn’t keep any written records). Irvinestown’s start in 1846. The parish immediately to the north is Magheraculmoney with records from 1836. Immediately to the south is Enniskillen, with records from 1836. They all start around that time.
Have you checked the Irvinestown records for any Rehills post 1846, as I suggested in 2016? Probably worth doing. The name is so rare that any you find seem likely to be your family.
But to answer your question about searching further back, you will struggle to make much progress. There are so few records to rely on. Most Irish research comes to a standstill in the early 1800s for this reason. One resource that does exist is part of the 1821 census for Co. Fermanagh including Derryvullan (the civil parish that Irvinestown is in). There are no Rehills/Rahills/Reils listed in the parish, which is odd. http://www.census.nationalarchives.ie/search/ Perhaps your family arrive din the parish after 1821?
I searched the 1835 tithe applotment records for the parish and there were no Rehills (or variants) farming in the area then. So your ancestors will likely have been agricultural labourers or weavers. They tended to move about to follow available work and can be difficult to trace.
Likewise there are no Rehills in Griffiths Valuation of the parish in 1862.
Statutory birth, death & marriage registration began in 1864. I searched those indexes from 1864 to 1901 for the Irvinestown area and didn't find a single Rehill, suggesting they were long gone from the parish by 1864.
There are Rehills elsewhere in Fermanagh. There’s 6 or 7 families in the 1901 census but they are all at the other end of the county.
Fermanagh Gold site specializes in records for that county:
http://www.fermanagh-gold.com/api/v1/search
Possibly DNA testing may be a way of matching with others who have additional information about where the family originate. Family Tree DNA reportedly has more people with Ulster roots than any other company. That obviously increases the chances of finding a match. You might want to try them or, if you have already tested, you can transfer your results to them for no fee.
The North of Ireland Family History Society is running an Ulster DNA project and can offer FTDNA testing kits at a reduced price. http://www.nifhs.org (Go to DNA project on the website).