Sunday, August 05, 2007

Testing?

In a comment to my previous post, Kristi writes:


Dr. Sannier,
Isn’t it usually customary when converting such a large legacy system to a new one that the new system shadows the old system for a period of time to work out bugs before it goes live? As far as I know, this was not done with the HRMS to PeopleSoft conversion. Why is this?



Deploying a system of this magnitude without a plan for testing it would be reckless indeed, and deserving of censure. But that has certainly not been the case with ASU's OASIS project.

We could not have been nearly as as successful as we have been over the last 18 months if we had not followed a rigorous testing plan. Payroll is the most recent in a series of successful system deployments, systems that include Admissions, Financial Aid, Student Registration, Benefits, Talent Acquisition -- all major applications which have been in steady, high volume use by the university since their respective launches between January and June of this year.

Here’s a quick breakdown of the testing that the new Payroll system underwent in 2007.

In January, the OASIS team initiated the record conversion process to move data from the legacy system to HCM. Unit testing of individual modules and data conversion tests were begun and continued to be run until the go-live in July. From February through April, over 50 test payrolls were run on a regular basis, to identify flaws in the new system and ensure their correction. By March, the majority of payroll records had been successfully converted, but continual evaluation and improvement of that data continued through to go-live.

In May and June, a series of test payrolls were run in parallel with the Legacy system, to help the team identify any remaining errors. Over 150 separate audit procedures were created and run after each conversion and test payroll to verify accuracy and identify issues. Beyond the testing for delivered software, all modifications and interfaces ASU created itself included their own set of unit, regression, integration and parallel testing.

Despite this extensive testing, which included parallel runs with the legacy HRMS system, some problems remain to be fixed once the system comes live. Testing and quality control can only do so much. Like the punch list on a new house, some flaws must be repaired all together, at the end. It's tempting to believe that any system can be tested to perfection, but complex systems like these, involving hundreds of thousands if not millions of records, are moving targets. Testing can tell you a lot about the robustness of the operational components, but it can't reveal every flaw - a fact that provides little consolation when the error directly affects you.

That’s why it’s important to remember that the key to our success so far -- in each of the OASIS rollouts -- has been the combination of extensive testing with the adapability and flexibility of ASU's staff. It is staff members who are making this system work: by quickly learning new procedures, by having patience and confidence while issues are resolved, by helping to identify remaining gaps, and by working together to solve any challenges we face.

It will be due to staff cooperation and diligence that, after the dust has settled, ASU will regard OASIS as a collective success, one that reflects positively on the New American University. It will be because all of us will have worked together that we've been able to rapidly and cost-effectively deploy new application systems to help ASU achieve its goals. Those goals - access, excellence, impact - a quality education for the public good – are surely worthy of our best efforts.

I assure you that everyone involved with the deployment of the payroll project -- from managers, to coders and testers, to those entering data or helping staff to ensure their pay is correct -- all of them are working hard to fix any outstanding issues and make things right for anyone who has been adversely affected by the recent changes.

If you are experiencing an issue that the regular OASIS channels aren't fixing quickly enough, please email me at adrian.sannier@asu.edu. I will personally ensure that the problem is addressed quickly.

8 comments:

Anonymous,  August 5, 2007 at 10:02 AM  

Can you define over 50 test payrolls and regular basis? I'm assuming that you have logs of the test showing exactly how many at each interval with observation. How much manpower did you put into testing? Did you have a qualified QA analyst overseeing the quality procedure? From my viewpoint of what you described, the testing from February through April does not seem very rigorous.

Then you describe for two months you did test runs comparing accuracy between the new and old system. Did you find any errors which produced any results such as the system giving incorrect data of 0 hours worked? Did you double check to make sure that the results were accurate as in the new system is bi-weekly, or did you change the old system to produce results which were bi-weekly as well? Again, from my viewpoint, the testing did not seem very rigorous.

Did you have this same problem with the paychecks when you were doing testing? If you did, did you calculate the percentage of how many people were going to be affected? Wasn't there a better contingency plan instead of having the employees having to go to the Arizona State Credit Union paying a 4.99 percent interest on a loan?

If you did have this same problem during testing, did you calculate how much money would be spent on fixing the problems while the system was in place instead of fixing the problem before the system was implemented?

You may say that you did everything that I questioned, but unfortunately I wouldn't know as there is little information publicly. If there were any information that you can publicly provide such as the project management portfolio or exact testing and findings everyone would have more data and perhaps understand what had happened instead of people calling for investigations.

I still would like to provide my skills to the UTO organization, as it seems to me a very challenging career, unfortunately it seems as even though jobs were posted, those jobs were dropped or were outsourced. Even if there was a newly created position of even helping the UTO as an intern for minimum wage I'd probably still take it as to see if I could lend my efforts to help ASU.

concerned,  August 6, 2007 at 4:57 AM  

Hi Dr. Sannier,

This is a bit off topic, but this is a great opportunity for bringing up the current IT layoffs and outsourcing events since the light is being casted on this issue ever so brightly.

Dr. Sannier, since you are being so open about this payroll issue, could you also discuss the reasons, motivations and the idea of how to transfer or cut these jobs, in part, due to the IT reduction in force with CLAS, your own Department and the future departments that will be cut or eliminated if you convince the Deans of their respective colleges to do so?

So far, this has not been discussed by you, Mr. Crow or any heads responsible, even though I am sure you are very aware of the chatter and resentment by those who feel you may be handling this transition with a heavy hand.

Any insight you may give on a different blog subject would most be appreciated.

-concerned

X,  August 7, 2007 at 9:33 AM  

While I worked at ASU, it was extremely frustrating, and now it is just plain baffling to see that the measure of success for system deployments is "steady, high volume use". It is common knowledge that if you have a word with anyone who has ever tried to make use of one of these systems in the "series of successful system deployments, systems that include Admissions, Financial Aid, Student Registration, Benefits, Talent Acquisition", you will likely hear a slew of expletives describing a completely unbelievable experience. I, personally, have had mind-boggling experiences with the Talent Acquisition system, and have heard tell of valiant attempts to register for classes or cash out benefits upon being RIF'd... all to no avail. I would suggest that some other measures of success be incorporated into your assessments, and that project plans allow for ample time for testing, QA, bug-fixing, and post-mortem analysis. These components of project planning are sorely lacking under the current arrangement at UTO.

Y,  August 7, 2007 at 1:56 PM  

What amazes me is that we (among many other large institutions) have paid millions upon millions of dollars (I've heard up to $22 million) for a system that:

- Disables the use of the back button on one's web browser (who uses that old thing anyway?)
- Has led to lawsuits at some of the institutions wherein it was implemented (see this $70 million doozy: http://seattletimes.nwsource.com/html/localnews/2002449249_computer23m.html )
- Loads content in frames (a technology NOBODY uses anymore for SO many reasons)
- Loads OVER 80 KILOBYTES in header code in most documents (most companies try to REDUCE bandwidth costs with new systems)
- Is a usability nightmare (have you tried registering for courses lately? Or applying for a job at ASU?)
- Is slow in every browser except Internet Explorer for Windows (which has been in steady decline among users worldwide)

Can ANYONE at UTO defend against these points? Politics and paychecks aside, this thing is a technological nightmare. Seriously, I would LOOOOOVE to see a written defense of PeopleSoft. The cherry on top would be to get AT LEAST 20 signatures of people involved with the implementation to say that they're proud to be a part of the OASIS team.

I have heard nothing but complaints about PeopleSoft. I have been working in IT for a decade now. Please see the following disheartening articles to see what other people and companies think:

http://chronicle.com/weekly/v50/i31/31a03402.htm
http://www.computerworld.com/action/article.do?command=printArticleBasic&articleId=9001396
http://software.silicon.com/applications/0,39024653,39125171,00.htm
http://www.fastcompany.com/magazine/100/beauty-of-simplicity-sidebar.html ("PeopleSoft software. The product most likely to induce a bout of Tourette's syndrome in the office.")

To me, PeopleSoft is like the movie Battlefield Earth; no one likes it, but somehow it got enough green lights to get produced. Furthermore, someone convinced a movie studio to spend millions and millions of dollars producing it. It's the kind of thing that's just so unbelievably bad that you can only HOPE that someday, someone will be intelligent enough to throw the whole thing out and start with something good.

Adrian, how can you defend this piece of software? Do you think it is well-written and easy to use? How can you use a WordPress blog, promote Drupal, and use a Mac while at the same time, promote and endorse such a fantastically horrible product as PeopleSoft. It goes against every principle and philosophy of lasting, open technologies. And how can we expect that a system like this is going to last so many years?

Honestly, whether you like it or not, you're the name and face behind the system. When people think of PeopleSoft, they're going to think of Adrian Sannier. Is this the legacy you want to leave with ASU? Do you want people, years from now, to talk about this implementation and attach your name to it?

If you only answer one question from this entire comment, would you, could you please explain how PeopleSoft will save the day and how we'll look back at this years from now and think, "The beginning was rough, but I'm glad we have this system"?

Y,  August 7, 2007 at 2:02 PM  

What amazes me is that we (among many other large institutions) have paid millions upon millions of dollars (I've heard up to $22 million) for a system that:

- Disables the use of the back button on one's web browser (who uses that old thing anyway?)
- Has led to lawsuits at some of the institutions wherein it was implemented (see this $70 million doozy: http://seattletimes.nwsource.com/html/localnews/2002449249_computer23m.html )
- Loads content in frames (a technology NOBODY uses anymore for SO many reasons)
- Loads OVER 80 KILOBYTES in header code in most documents (most companies try to REDUCE bandwidth costs with new systems)
- Is a usability nightmare (have you tried registering for courses lately? Or applying for a job at ASU?)
- Is slow in every browser except Internet Explorer for Windows (which has been in steady decline among users worldwide)

Can ANYONE at UTO defend against these points? Politics and paychecks aside, this thing is a technological nightmare. Seriously, I would LOOOOOVE to see a written defense of PeopleSoft. The cherry on top would be to get AT LEAST 20 signatures of people involved with the implementation to say that they're proud to be a part of the OASIS team.

I have heard nothing but complaints about PeopleSoft. I have been working in IT for a decade now. Please see the following disheartening articles to see what other people and companies think:

http://chronicle.com/weekly/v50/i31/31a03402.htm
http://www.computerworld.com/action/article.do?command=printArticleBasic&articleId=9001396
http://software.silicon.com/applications/0,39024653,39125171,00.htm
http://www.fastcompany.com/magazine/100/beauty-of-simplicity-sidebar.html ("PeopleSoft software. The product most likely to induce a bout of Tourette's syndrome in the office.")

To me, PeopleSoft is like the movie Battlefield Earth; no one likes it, but somehow it got enough green lights to get produced. Furthermore, someone convinced a movie studio to spend millions and millions of dollars producing it. It's the kind of thing that's just so unbelievably bad that you can only HOPE that someday, someone will be intelligent enough to throw the whole thing out and start with something good.

Adrian, how can you defend this piece of software? Do you think it is well-written and easy to use? How can you use a WordPress blog, promote Drupal, and use a Mac while at the same time, promote and endorse such a fantastically horrible product as PeopleSoft? It goes against every principle and philosophy of lasting, open technologies. And how can we expect that a system like this is going to last so many years?

Honestly, whether you like it or not, you're the name and face behind the system. When people think of PeopleSoft, they're going to think of Adrian Sannier. Is this the legacy you want to leave with ASU? Do you want people, years from now, to talk about this implementation and attach your name to it?

If you only answer one question from this entire comment, would you, could you please explain how PeopleSoft will save the day and how we'll look back at this years from now and think, "The beginning was rough, but I'm glad we have this system"?

Michael,  August 7, 2007 at 3:36 PM  

A really funny thing came to my attention last month (now that I think of it, it wasn't that funny as my paycheck got fouled up twice). The problem is best described by the "We're sorry to have fouled up your paycheck" e-mail below:

"If you are paid on the academic schedule (during the 9 months of your contract) and are working during either summer session, there are some changes regarding benefits deductions that you should be aware of due to the recent HCM/Payroll conversion.

Previously, a portion of the insurance premiums you paid during the contract year were used to pay for premiums during the summer months when you were not receiving pay. If you agreed to work the summer sessions, no additional deductions were taken during the summer. With the new HCM system we are not able to accommodate this scenario this summer.

The HCM system will take premiums from your pay for the remainder of the summer. If you already had the premiums deducted, this overpayment will be reimbursed in August. We are exploring technical solutions to avoid this situation happening again next summer. However, there are a couple of other alternatives available to us. If you convert to Fiscal pay (paid over 12 months) this will not be an issue since pay and deductions occur each pay period. The other alternative is for HR to work with the Provost's office to determine which faculty will be working the summer '08 session as early as possible. If we know in January that a faculty member will be working the summer, we can "disable" the precollect function and normal deductions will be taken each payperiod through the spring semester and then during the summer.

I know that this new system has brought about a number of changes in our processes and we appreciate your flexibility and support in enabling us to implement these across the university."

Please allow me to translate:

"The new system doesn't actually work in an academic environment and we have no idea how to make it work but we can maybe sorta fudge it with your help, especially if you can predict the future."

Was this part of the extensive testing or did nobody notice that there were faculty at ASU?

By the way, in my case two other paycheck categories were also wrong...

concerned,  August 13, 2007 at 8:04 AM  

Hi Dr. Sannier,

Could you please provide myself and everyone else who is curious about the move to outsourcing and consolidation of IT deparmtents to a central IT, if you are going to answer my question in order to open a conversation about these events? It would be much appreciated. It feels as though you may be dodging the issue.

Thank you,
Concerned

Anonymous,  August 22, 2007 at 7:52 AM  

Hello,
I am a relative of an ASU employee and I happen to be a SAP software test manager, Not only does every scenario have to be tested completely, we are also required to have a comprehensive back out plan. This plan is to be used to revert to the "old" system in the event of major issues. It sounds like either a back out plan was not in place, or you were willing to accept the risk and cost of not having one. In the public sector this kind of software deployment would not be acceptable.