When you try to upload a customizations file (xml or zipped) into CRM 4.0 RTM and get the message: "Either the file couldn't be uploaded or this is not a valid customizations file" it probably signals that there is something wrong with your XML; you shouldn't get this message so it probably means that the xml was manually constructed or tweaked which is unsupported (for sections other than sitemap and isv.config). To troubleshoot, the first thing that you should try is validating the file yourself using a simple xml tool. I recommend you xml notepad. Once you open your xml file with xml notepad you just have to go to "View>Schemas" and point to the following 3 files (do a quick search under your CRM Web folder): customizations.xsd, isv.config.xsd, and sitemaptype.xsd. Any errors with the schema should be easily identifiable and, in most cases, very easy to fix manually.
Also, make sure that the file that you are trying to upload isn't exceeding the maximum upload size of your IIS and CRM server; if you were trying to upload a huge xml file,just zip it and try to upload the zip file instead. For deeper troubleshooting, dev errors and trace files are the best alternatives. The "Maintaining and Operating Guide" included as part of the implementation guide has more details.
You may be wondering, why all the complication? CRM should be telling me exactly what went wrong right from within the UI... I fully agree with you. We did a bunch of enhancement around import/export of customizations but friendly xsd validation is one area that we didn't invest so much; mainly because, in theory, nobody should have invalid xml files...in practice, it may happen for various reasons. Well, I hope this helps you troubleshoot any issues with import/export faster.