If you're turning it on while there's a cartridge in the slot, then it means it's not reading the game, and is acting as if there's nothing inserted (whereas if there's no game inserted, it'll sit at the Game Boy screen since there's nothing to read). Most likely in this case it's just a matter of cleaning the contacts on the Game Boy Advance cartridge, preferably with high-grade rubbing alcohol (I'd recommend nothing lower than 97%).
Have you tried staring it up with only one game, or have you tried a few?