First you may want to look at this Vid: 2017 27-inch iMac with Retina 5K Display Unboxing and Teardown jump to the 10:00 time point which shows the PCIe SSD Apple is using. It's very different from the M.2 SSD's.

Apple's PCIe NVMe SSD's use a custom device driver so while you might be able to get the adapter and NVMe SSD to physically fit it won't be recognized by the OS.

While I haven't tried anything in this new iMac I can tell you I've tried the newer MacBook Pro retina series which has a 2 lane NVMe SSD and it didn't work.