Evaluating SharePoint Partner Solutions and Applications built on SharePoint
With the SharePoint Conference there are a number of new partners in the SharePoint space with more coming every day. How do you keep them straight? How do you know if they are going to treat you well? I've been thinking I need to put together a litmus test for SharePoint partners. The SharePoint team doesn't have the resources to do certification on the product or have "logo'ed" partners, but there are ways of telling how well their solutions will do under scrutiny. I've come up with a simple set of questions you can use to find out how well the partner follows the "rules."
You'll also find these questions aren't just for partners, but good for your business and dev teams. Push them to give you the answers. If they try to follow these Five simple guidelines or have good answers to all of them we'll all be better off. Budget constraints are not a reason to skirt these guidelines. These are real no brainers, but I figure spelling this out will help you have the right conversations. If you think I'm wrong let me know. With all these, I hope you're following the guidance for managing this all in a dev, test, or build environment. You don't want to find out that the production environment is being impacted by these. You can nearly validate most of them in the test environment before they make it production. Don't just take their word for it... that's what I'm saying.
Instructions for keep partner solutions or your dev teams inline
1. Do they directly update or modify the database? You should find out what the solution is doing and why they are doing it before going with any solution that directly accesses and modifies the database. One of the riskiest of solutions is based on an assumed database schema. Given there are partners that are out there that have found no other way to do what they are doing, but the better you can understand these things and as well encourage MS to expose these things via the object models, web services or other APIs the better off all of us will be. Think about supportability as well. CSS isn't going to be excited to work on your environment if they know database modifications have been made. Modified stored procedures are a no, no.
2. What is the residue or footprint if you remove it? Is it a solution or feature? Can that solution be removed cleanly? Some of the nastiest solutions I've ran into have very messy solutions where when you run the remove programs or in more recent cases remove the solution through either the UI or stsadm, the pages puke. I mean they throw chunks. Sorry for the sad picture, but yes this means they hemmorage errors and won't render anything. If they say, yes there's residue, the follow up should be how do I clean up this residue? If they don't have a good answer then run. I wish the product had better answers, but today it means building responsible code or getting the devs to build hunter tools to track and remove residue.
Demand Solutions and Features or MSIs... You do want your DLLs in the form of solutions that can be easily deployed, removed and managed. I tell customers they should demand that their developers do it this way. You should expect the same or better from a partner solution. The partner should be thinking ahead to what happens when the evaluation is over. If the evaluation is over and it destroys your environment and doesn't remove cleanly you're both in trouble. Some web part solutions will never have a good answer to this, and when that's the case it's important that you acknowledge a plan for how to deal with these. Is there a tool that will be purchased or provided to help out the operations and support team? What is the configuration management plan or release plan and how do you decide how to remove it?
3. Do they have an upgrade path? How will this be upgraded when it's ready to be upgraded? If the partner themselves has only been around for a few months, they may have some suggestions for what they might do. New partners should be given a chance, but you should ask for maintenance agreements through upgrade. Even if its your dev team within your company or a scarier a consulting company who is simply working on a "short term project" to address a "highly critical" requirement, they or you or your business must allocate budget for the future. Dev projects are not just pay once. With the 3 year cycle of SharePoint, you'll be revisiting these solutions and add-ons and you'll want to have a contingency plan for supportability and upgrade, or what does #2 look like when the functionality is now in the product? They get paid the bucks to come in and what do they do... they think short term and not about upgradability and supportability. That's why you'll see custom site defs and list defs. Easy answers, but not good answers. More and more you'll find application based SharePoint deployments *need* a dev to complete the upgrade as a result of custom code including customizations that either were modifications to out of box pages or new site and list defs. Those are the hardest ones to address.
4. What is the support and sustainability for the app or solution? How often is something rolled out and you have support for the solution for a couple or three months and then the dev leaves the company, the consultants roll off at fiscal or budget runs out? Hey everythings good right? We haven't had any issues up to now. Why would we have issues in the future? I'll give you 5 reasons... 1) service packs 2) hot fixes 3) upgrades 4) server rebuilds 5) unexpected errors. As soon as you have to troubleshoot that crazy web part that use to be soooo cool that is now slowing down the system or is now resulting in can't load assembly or can't locate, or whatever. You're going to be hating life. Where is that dev? Do you have a 24 hr support number with a support contract with a maintenance agreement? Where did you store that PID key? That's why it's so important that you must do configuration management. Do you have a book or a log outside of your server farm where you keep a log of what's been rolled out? Does that governance plan have contact numbers both within your team including support hours and SLAs for the dev teams? Heard of OLAs? Those are operations agreements you might have with other teams. Those devs may have a pager that they rotate for times like this. If they don't maybe they should or at least get with the times and have a text message system.
5. Does it scale? How does it scale? The saddest stories I've heard are where SharePoint gets blamed for someone's custom navigation control. Why it's always navigation I don't know. Some of the coolest master pages aren't the largest. Sure there are some decent sized ones, but the really good ones are those who take into account caching methods and don't hit the database for each item and know how to optimize those round trips. There is already a big enough perf hit to get security trimmed UI, the bread crumbs, role based UI and targeting. We don't need much extra. What this one means is who is perf testing this solution that's being rolled out? If there's a chance it could slow the system what are the benchmarks for it? You should ask to see benchmarks with OOB vs. the solution. If it's your dev team, you should get them to use Visual Studio Test tools against it with it off and with it on. It's important you understand your environment and what the slowest component or bottleneck is. Not that you want it to be the out of the box components, but you do. Once you want to do more than that, you may go with minimal master pages for the serious designers, or you go with page optimization techniques for anonymous Internet sites.
If all this seems overwhelming then try to start by keeping it simple. The KISS principle pays off early and frequently. It's a lot harder to start over, or to try to play catchup after some custom assembly has been freed into the wild and no one knows where it's being used. The more you know what the out of the box can provide, the better off you'll be when providing guidance to the business on how best to take advantage of the solution. Less is more. There are some awesome solutions out there and some awesome partners that support them. Just make sure you weed out those that are just trying to make a buck and don't have your best interest in mind. You want to make sure whatever choices you make you acknowledge that these are lasting relationships. I'm saying this is that new brother in law you'll see every Christmas from now on. That's really what rolling out service packs and upgrades feels like, it's a level of trust that is going to feel uncomfortable. Oh yeah, forgot I had that solution, but man I feel good that it's Uncle Jesse, he's on it, and he's got my back. :)
There are some similar posts and reference articles that may help with this topic:
