If you know the application which can open RBXM file then run it and see if there is a File->Open main menu option in the application.
If a RBXM file is not opened in an application and you instead get a 'Windows can't open a file' error message you should try looking for an application which can open the file in question.Find a RBXM file in question in Windows OS File Explorer and double click on it to launch the corresponding application.
