Is it permissible for her sister’s husband to stay overnight with them in their house?

Your sister’s husband is regarded as a stranger
(non-mahram) to you, so it is not permissible for you to be alone together
or for you to sit with him without a mahram being present, or to go out of
the house with him, or to uncover your face in front of him. He is like
any other non-mahram man. 

It says in Fataawa al-Lajnah al-Daa’imah (17/420): 

The sister’s husband is
not a woman’s mahram, and he is regarded as a stranger (non-mahram) to her.
It is not permissible for her to uncover her face in front of him, or to
shake hands with him, or to be alone with him, or to travel with him. He is
like all other non-mahram men. But if she sits with him in the presence of
one of her mahrams and observing proper hijab, there is nothing wrong with
that. End quote. 

The following question was mentioned in Fataawa al-Lajnah
(17/160): 

My brother’s daughter and I are married to two brothers, and
I am his cousin. We have lived together in one house for a long time, and I
sit with my husband’s brother wearing hijab that is not complete, in the
sense that my face is uncovered, but I do not make a wanton display of
myself (tabarruj). What is the ruling on this? 

The committee replied: 

A woman is obliged to cover her face in front of the husband
of her female cousin and the husband of her sister, and the husband of her
brother’s daughter, because they are strangers (non-mahrams) to her. It is
not permissible for her to uncover her face in front of them, as is the case
with all other non-mahram men, because the face is one of the most important
‘awrahs that must be covered in front of men, because it is the site of
fitnah (temptation) and looking. End quote. 

His simply staying over at your house is not haraam in and of
itself, if all precautions are taken and there is no danger of anything
happening that Allaah has forbidden. 

But, since the house in which you live is small as you
describe, then your sister and her husband should look for another house and
move there, if they are living with you on a permanent basis, or reduce the
length of their visits to you, if they are just visiting you. During those
times you must not be negligent with regard to observing hijab in front of
him, even if that is difficult for you, because this is the command of
Allaah and He knows best what is fit for His slaves. 

And Allaah knows best. 

See also questions no.
6408, 13261,
13728,
40618.