The proof of concept lab (POC) is an important tool for both vendors and customers. For customers, a POC is a great way for you to evaluate the vendor promises and make sure that the “marketing” matches your business. For vendors, it’s a chance to show off your products capabilities strengths and to help differentiate you from your competitor. Unfortunately, POCs tend to be half-assed and disorganized. Proof of concept labs are complicated beasts. There is no such thing as a quick POC. The more rushed the POC the more likely it is to fail – and a failed POC is a huge burden of time and money on both vendor and the customer. Success is determined by the inputs and interaction from both the customer and the vendor.
I have been able to participate in several POCs, big and small, and I would like to share with you the lessons I have learned – both good and bad.
Customer Tips
The first thing you as the customer needs is a TEST PLAN. No test plan equals a failed POC or indeterminate results. The test plan defines the tasks that you wish to see demonstrated. Start by listing the requirements on a piece of paper or in a text document. These are your requirements and it is those requirements that will determine success. Vendors may add on differentiators during the POC, but your list defines the core of what must be tested.
Next, sketch a topology that you think would best represent the requirements. You can make it look like your a subsection of your existing network or a new design completely. This is what the vendors will need to build to. You can overlay routing protocols and addressing to remove any ambiguity, but that is optional.
Once you have a list of requirements and a topology, then you can start developing the test plan itself. If you need help developing the plan then ask your equipment vendors. They can get you old POCs or test plan templates. Take the test plans you received from the vendors and create your own. This is especially important if you are evaluating several vendors for the same project. You cannot determine a clear winner if each vendor executes different test plans.
Vendor Tips
- It takes 1 week of prep time for every 2 days of actual customer demonstrations. Also, I would say that a minimum of 2 weeks of prep time is required for any POC.
- The test plan MUST be completed and approved by the customer BEFORE onsite POC preparation begins.
- Interoperability POCs are a bad idea as an initial POC. Interoperability testing is a good thing, but the customer must provide the resource to operate the “other” vendors equipment. Having your SE’s operate another vendors equipment is a recipe for disaster.
- You need to schedule POC resources (eg equipment, CSEs) well in advance of the POC date. Equipment must be allocated, cabled up and the right code version loaded before you get on site. Having a Corporate Systems Engineer (CSE) available to execute the test plan provides the best chance for a successful POC. CSEs run POCs everyday.
- Set customer expectations early to head off unnecessary disappointments down the road.
- You will only execute what is in the test plan – no more, no less. There is increased risk of failure if you deviate from the test plan. Since networks are complicated and POC are reviewed under a microscope any mid stream changes increases the probability that something will go wrong. I have had customers ask, “Why are we dropping ONE packet.”
