Back in the 90s we wrote a Windows graphics driver so that we could demonstrate a new high speed bus for the PC at Comdex. We got the driver written, got 4 flashy apps that would nicely demonstrate the performance of the new bus, and everything ran at Comedx without a hitch.
Then a few weeks later a marketing guy comes to me and asks if he can give the software to a graphics card vendor as a reference. I say of course (because getting vendors onto this bus was the whole point of the demo) but I tell him that he has to make it perfectly clear to them - in writing - that this was only debugged as far as getting 4 apps to run on it. And he promises to do so and in a few weeks I start getting calls and emails about bugs anyway
4 running apps was/is nowhere near enough to test complicated infrastructure. There are many APIs, optional parameters, different data types, room for confusion in the specifications of complex data types… even something as simple as some of the source data being or not being word-aligned can cause different code to run.
I few years later I worked on a similar project and every night we installed the newest build on 290 different hardware configurations, and then they all ran applications driven by scripts all night long and we came in the next morning to see what happened. PCs and OSs are complicated - you don't know that thing X is at fault simply because similar thing Y runs.
Then a few weeks later a marketing guy comes to me and asks if he can give the software to a graphics card vendor as a reference. I say of course (because getting vendors onto this bus was the whole point of the demo) but I tell him that he has to make it perfectly clear to them - in writing - that this was only debugged as far as getting 4 apps to run on it. And he promises to do so and in a few weeks I start getting calls and emails about bugs anyway
4 running apps was/is nowhere near enough to test complicated infrastructure. There are many APIs, optional parameters, different data types, room for confusion in the specifications of complex data types… even something as simple as some of the source data being or not being word-aligned can cause different code to run.
I few years later I worked on a similar project and every night we installed the newest build on 290 different hardware configurations, and then they all ran applications driven by scripts all night long and we came in the next morning to see what happened. PCs and OSs are complicated - you don't know that thing X is at fault simply because similar thing Y runs.
