He died and left behind his sister and his brother’s children

If
a person dies and did not have any children, and also did not have a father
or a grandfather, then this case is called kalaalah, which refers to the one
who dies and does not leave behind any parent or child. 

Kalaalah is mentioned in the Qur’an, in the verse in which Allah, may He be
exalted, says (interpretation of the meaning):

“They ask you
for a legal verdict. Say: ‘Allah directs (thus) about al-Kalaalah (those who
leave neither descendants nor ascendants as heirs). If it is a man that
dies, leaving a sister, but no child, she shall have half the inheritance.
If (such a deceased was) a woman, who left no child, her brother takes her
inheritance. If there are two sisters, they shall have two-thirds of the
inheritance; if there are brothers and sisters, the male will have twice the
share of the female. (Thus) does Allah makes clear to you (His Law) lest you
go astray. And Allah is the All-Knower of everything”

[an-Nisa’ 4:176]. 

And Allah, may He be exalted, says (interpretation of the meaning):

“If the man or woman whose inheritance is in question has
left neither ascendants nor descendants, but has left a brother or a sister,
each one of the two gets a sixth; but if more than two, they share in a
third; after payment of legacies he (or she) may have bequeathed or debts,
so that no loss is caused (to anyone). This is a Commandment from Allah; and
Allah is Ever All-Knowing, Most Forbearing”

[an-Nisa’ 4:12]. 

Shaykh Muhammad al-Ameen ash-Shinqeeti (may Allah have mercy on him) said:
The correct view is that what is meant by kalaalah is leaving no ascendants
or descendants, as an-Naazim said: 

They ask you about al-kalaalah… it definitely refers to lack of descendants,

No
surviving parent or offspring, so there are no sons, (no parents) and no
grandparents. 

End quote from Adwa’ al-Bayaan (1/228) 

Based on that, if that person who died did not have any father or son, and
the rest of the heirs are as mentioned, then there inheritance is to be
divided in the following manner: 

The full sister gets half, as the share allocated to her by sharee‘ah,
because Allah, may He be exalted, says (interpretation of the meaning):

“They ask you
for a legal verdict. Say: ‘Allah directs (thus) about al-Kalaalah (those who
leave neither descendants nor ascendants as heirs). If it is a man that
dies, leaving a sister, but no child, she shall have half the inheritance…”

[an-Nisa’ 4:176]. 

The two sons of the full brother get the rest of the estate, because the
Prophet (blessings and peace of Allah be upon him) said: “Give the shares of
inheritance to those who are entitled to them, and whatever is left goes to
the closest male relative.”

Narrated by al-Bukhaari (6746) and Muslim (1615) 

The two daughters of the full brother (the two nieces) do not inherit
anything of the estate. 

The division of the estate mentioned above is based on the assumption that
the sister mentioned in the question is the full sister of the deceased or
his sister through the father; the same is assumed in the case of her
brother. 

For more information, please see the answer to question no.

160948

And Allah knows best.