When there is a preposition before a noun, it is always the preposition which governs the noun case. So, "у" always requires genitive.
Prepositions should not be translated directly from one language to another one. But there can be some hints to understand them.
The preposition "у" is more or less like English "at": Стол стоит у стены. - The table is (stands) at the wall. "Стена" is in genitive because of "у": у стены.
We often use this preposition to express possession as well: У меня есть книга. (At me there is a book. = I have a book). У моей сестры есть сумка. (At my sister there is a bag. = My sister has a bag).
The same preposition is also used to convey the idea "at someone's place" (although the word "place" is dummy and it is not used in Russian translation): Я был у друга. - I was at my friend. (= At my friend's place). У нас не курят. - At us (= at our place) (they) do not smoke. = Please do not smoke at our place. Какая у вас погода? - What is the weather at you? (= at your place, in your country, in your city etc.).
1. Please notice: день рождения.
2. В гостях у Тани is literally "in guests at Tanya". As I explained it above, yes, it is "at her home". "В гостях" is a fixed expression "when being someone's guest".
3. Вчера был день рождения Тани, сестры Олега. = Yesterday was birthday (lit. Day of Birth) of Tanya, of sister of Oleg. Here "сестра" is used to clarify who Tanya is, that is why "сестра" repeats the same case as "Таня": the birthday of Tanya (i.e. of Oleg's sister).
And "сестра" just cannot be in nominative as you suggest, since it is not a subject of the sentence
Foreign names ending in -о, -е, -и, -у, -ю, -э cannot be declined. Theoretically, in "книги Марио", the name "Марио" IS in genitive. But all the cases of this name just coincide: Марио, Марио, Марио, Марио, Марио, Марио.
As I mentioned it above, it is a set expression. Actually, there are two forms: "в гостях" (location) and "в гости" (direction).
"В гостях" (literally, "in guests") = not at home, at someone's place, being a guest there. A person at who you are staying (i.e. the host) is specified with the preposition "у". Or, it can be used without specifying a host: Не забывай, что ты в гостях! = Do not forget you are in guests (not at home)! It is a common saying to remind someone that a guest should respect the host's rules.
"В гости" (literally, "to the guests") means direction. The person to who you go is specified using "к" (+ Dative): Я иду в гости к другу. = I am going to guests to my friend (i.e. I am going to see my friend, to his place). It can also be used without specifying the host: "Мы идём в гости" = We are going to pay visit (to someone).