First you may want to look at this Vid: [https://www.youtube.com/watch?v=YSgcJH5bdWM|2017 27-inch iMac with Retina 5K Display Unboxing and Teardown] jump to the 10:00 time point which shows the PCIe SSD Apple is using. It's very different from the M.2 SSD's.
Apple's PCIe NVMe SSD's use a custom device driver so while you might be able to get the adapter and NVMe SSD to physically fit it won't be recognized by the OS.
While I haven't tried anything in this new iMac I can tell you I've tried the newer MacBook Pro retina series which has a 2 lane NVMe SSD and it didn't work.