Hi,
There is a problem either with the connection of the video cable between the motherboard and the display at either end, the cable itself (does the problem come and go if you move the lid back and forth?) or the screen itself.
Here’s a link to the [https://downloads.dell.com/manuals/all-products/esuprt_laptop/esuprt_alienware_laptops/alienware-m17x-r3_service%20manual_en-us.pdf|service manual] for your laptop.
Scroll to p.71 to view the necessary pre-requisite steps and then the procedure to remove / replace the display. This will help you to first gain access to where the display cable is plugged into the motherboard to check that it is seated properly. If it appears to be OK then you’ll have to continue with the procedure and remove the display cable and the display to investigate further.
Here is a link to just one [https://www.laptopscreen.com/English/model/Dell/ALIENWARE~M17X~R3/|supplier] of a replacement screen. It is not a recommendation to use them. It is shown just to give an idea of the cost of the part.
There are other suppliers online that may suit you better. Just search for ''Dell Alienware MX17xR3 lcd screen'' to get results.
Hopefully this is of some help.