Difficult to answer this for every situation. No two cases were the same. Mixed marriages did occur, and were quite common in the counties of Ulster where the numbers of other denominations matched, or even occasionally exceeded, the numbers of the RC population. Such marriages gave the couple and their families some things to think about, both in terms of whether one of them should change denomination and with regard to what denomination(s) to bring the children up in. And no doubt quite a few families didn?t approve of their offspring?s choice. But they did happen, and in not unsignificant numbers. (The records of all the main denominations regularly record adult baptisms, and that was usually a precursor to a marriage that would have been regarded as mixed).
Illegitimate children were usually kept by the mother or her parents. If that was not possible, they were sometimes farmed out to a married sister. Failing that, then an informal adoption arrangement was a possibility, with the child being handed over to a couple with no children or some other couple willing to take it. The baptism records of all the main denominations also contain a steady stream of foundlings, left on the church porch. So that was evidently another well trodden path.
You often see children born well before their parents had married on the 1901 & 1911 censuses, so I think it?s fair to say that in some cases such illegitimate children were included in the married couples family. Formal adoption was only introduced in Ireland in the 1950s (1927 in what is now Northern Ireland). So prior to that there was no formal adoption. You simply looked after the child, and gave it your surname.
I don?t think there were many orphanages in the early 1800s, but obviously as they increased in numbers that was another option (often under the auspices of the church). The creation of workhouses under the Poor Law system from the 1840s onwards was another solution.