Lack of right performance tools and bad product performance observability
Often times, when a performance problem is encountered, it is caused by a lack of tools needed to detect the problem in the first place- or a lack of tools to troubleshoot suspicious performance numbers. Without the correct tools, the performance issue has a chance to slip by without being noticed and then it can cause bigger problems at a later time.
Furthermore, in many cases, the product observability is so bad that it is impossible to figure out the cause of the problem without sending an “instrumented” version of the product.
Lack of sufficient control over the product is also an important factor. Many times the solution to the performance problem at a customer site can only be found by changing a few, unfortunately-hard-coded values.