Regression Testing and Migration Paths.
A recent PSoC3 Problem Determination episode elicited a Cypress Forum suggestion to migrate from Creator 2 to Creator 3 in an effort to find the cause. This step was undertaken but with a large amount of trepidation as to what to do if Creator 3 resolved the problem.
A significant amount of development, which included extensive testing, was expended using Creator 2. My project is one of an embedded design for Data Acquisition and Control of some fairly expensive machinery; one where expensive damage to the machinery could occur if an error went undetected as a result of a migration. The damage could well exceed the profit that would accrue as a result of a customer purchasing my embedded system.
Being a small shop, I did not develop extensive regression testing procedures. I employed as much data independency as the Keil C compiler would support so that incremental changes to my C code would be isolated from other code. Granted this is not a fool-proof process.
During the testing of Creator 3, I took a few minutes to compare selected Creator 2 generated source files with Creator 3 versions. I particularly noted that the PS0C3 files of CyLib.c and ADC.c Delta Sig appear to be significantly re-written. I rely heavily on the ADC Delta Sig. Thus I am faced with having to re-do a significant testing effort if I migrate.
So far, Creator 3 testing has not solved the problem. But if it were to do so, would Cypress provide a corrective “patch” for Creator 2?
Creator strongly advises that Component upgrades be made when migrating to a new release. Do older component versions get regression-tested with a new release so that migration can be a slow process by a user without introducing transition problems as a result?
AFAIK older component versions get not touched with newer versions of Creator, so they should behave the same (and generate the same code).
Naturally there might be changes to how placement of components work (because the CyFitter process might optimize differently) , but these are differences that might even occur within the same CReator version (by just making small changes to the project).
And then there is the compiler - that might be the biggest difference when changing Creator version (apart from updating components). This might also be the reason to modify older components - namely when their code for some reason won't compile anymore (which I doubt).
So if you are conservative: create a copy of your project, use that for upgrading to Creator 3 and see whether that fixes your problem (just upgrade all components). If it does, start over again and upgrade one component after each other, until you know which one was responsible for the problem. Then you can start over again (trhats what backups are for 🙂 and upgrade just this component. Then you can do regression testing, in the affected areas.
When a component update is installed you get asked when opening an existing project whether you want to update your components, you can even select which and which not. So it is easy to keep the old component versions within the project, you are not forced to use the new components for old projects.
Not to be a fly in the ointment but sounds like you are Space Age critical (almost)
in your application, and the only answer for that is to build in resilency and design
for test. Down to the memory/register code level. Build a "signature" test series for the
application where the final signature must match exactly, or the cause determined.
I do not envy the effort this takes, but if lives or very high cost systems at stake
there is simply no other choice. The signature is a forcing function, then a test
of all registers, memory affected.
I hope you are selling your design for $ 1T a copy......:) to get the resources needed
for an effort like this.
One additional note: yes, Cypress might backport bugfixes to older versions if its necessary to get a component working (I thing the EEPROM component got one IIRC). But this will be the exception.
Thanks for your replies.
No my project is not space age critical nor are lives involved. But my concerns come about from a previous experience.
One company I was working for used a Statistical Analysis System and at a user conference it became apparent that the developer of the product was very aggressive with improving their product. This caused much consternation for a large number of the attendees. Questions that I posed here are an echo of those questions at the conference. The use of said product was somewhat "mission critical" at many shops. Improvement of a product is a normal event, but there is a need for a user to incorporate changes at the user's pace.
My questions were posed to ascertain that progress can be incremental and at the user's pace.
Your replies give me some direction and confidence that I will not be forced into a hasty state.