He committed homosexual acts with someone, and now his brother wants to marry the sister of that person. What is the ruling?

Praise
be to Allah.

Firstly: 

Homosexuality is a grave major sin, because of which Allah destroyed the
people of Loot and demolished their cities on top of them, turning them
upside down, because of what their deeds involved of deviation and going
against sound human nature. 

Allah, may He be exalted, says (interpretation of the meaning): 

“And
(remember) Lout (Lot), when he said to his people: ‘”Do you commit the worst
sin such as none preceding you has committed in the Alameen (mankind and
jinns)?

‘Verily, you practise your lusts on men instead of women.
Nay, but you are a people transgressing beyond bounds (by committing great
sins).’

A nd the answer of his people was only that they said: “Drive
them out of your town, these are indeed men who want to be pure (from
sins)!’

Then We saved him and his family, except his wife; she was of
those who remained behind (in the torment).

And We rained down on them a rain (of stones). Then see what
was the end of the Mujrimoon (criminals, polytheists, sinners, etc.)”

[al-A ‘raaf 7:80-84].

“So As-Saeehah (torment – awful cry, etc.) overtook them at
the time of sunrise;

And We turned (the towns of Sodom in Palestine) upside down
and rained down on them stones of baked clay.

Surely! In this are signs, for those who see (or understand
or learn the lessons from the Signs of Allah).

And verily! They (the cities) were right on the highroad
(from Makkah to Syria i.e. the place where the Dead Sea is now)”

[al-Hijr 15:72-76]. 

And there are other similar passages. 

At-Tirmidhi (1456), Abu Dawood (4462) and Ibn Maajah (2561) narrated that
Ibn ‘Abbaas (may Allah be pleased with him) said: The Messenger of Allah
(blessings and peace of Allah be upon him) said: “Whoever you find doing the
action of the people of Loot, execute the one who does it and the one to
whom it is done.”

Clased as saheeh by al-Albaani in Saheeh at-Tirmidhi. 

Ahmad (2195) narrated from Ibn ‘Abbaas (may Allah be pleased with him) that
the Prophet of Allah (blessings and peace of Allah be upon him) said: “May
Allah curse those who do the action of the people of Loot, may Allah curse
those who do the action of the people of Loot,” three times.

Classed as hasan by Shu‘ayb al-Arna’oot in Tahqeeq al- Musnad. 

Secondly: 

It
is permissible for the brother of the one who committed homosexual acts to
marry the sister of the one with whom he committed those acts, because he
has nothing to do with his brother’s sins and, likewise, the sister has
nothing to do with her brother’s sins. So there is no way that this could
make the marriage prohibited. 

And Allah knows best.