Is it permissible for her sister’s husband to stay overnight with them in their house?

Your sister’s husband is regarded as a stranger (non-mahram) to you, so it is not permissible for you to be alone together or for you to sit with him without a mahram being present, or to go out of the house with him, or to uncover your face in front of him. He is like any other non-mahram man. 

It says in Fataawa al-Lajnah al-Daa’imah (17/420): 

The sister’s husband is not a woman’s mahram, and he is regarded as a stranger (non-mahram) to her. It is not permissible for her to uncover her face in front of him, or to shake hands with him, or to be alone with him, or to travel with him. He is like all other non-mahram men. But if she sits with him in the presence of one of her mahrams and observing proper hijab, there is nothing wrong with that. End quote. 

The following question was mentioned in Fataawa al-Lajnah (17/160): 

My brother’s daughter and I are married to two brothers, and I am his cousin. We have lived together in one house for a long time, and I sit with my husband’s brother wearing hijab that is not complete, in the sense that my face is uncovered, but I do not make a wanton display of myself (tabarruj). What is the ruling on this? 

The committee replied: 

A woman is obliged to cover her face in front of the husband of her female cousin and the husband of her sister, and the husband of her brother’s daughter, because they are strangers (non-mahrams) to her. It is not permissible for her to uncover her face in front of them, as is the case with all other non-mahram men, because the face is one of the most important ‘awrahs that must be covered in front of men, because it is the site of fitnah (temptation) and looking. End quote. 

His simply staying over at your house is not haraam in and of itself, if all precautions are taken and there is no danger of anything happening that Allaah has forbidden. 

But, since the house in which you live is small as you describe, then your sister and her husband should look for another house and move there, if they are living with you on a permanent basis, or reduce the length of their visits to you, if they are just visiting you. During those times you must not be negligent with regard to observing hijab in front of him, even if that is difficult for you, because this is the command of Allaah and He knows best what is fit for His slaves. 

And Allaah knows best. 

See also questions no. 6408, 13261, 13728, 40618.