Friday, September 28, 2007


It seems that last Tuesday’s Wall Street Journal article about ASU’s ERP implementation has stirred considerable discussion in various venues around the country. Much is being extrapolated from some fairly vague details in the article. While we appreciate the Journal’s take on our progress, short articles like this have limited space for specifics. And focusing on controversy, while providing a provocative read, doesn’t create the best backdrop for discussing the broader issues.

In the interest of spurring a solid debate on the virtues and drawbacks of ASU’s approach, I'm using this venue to answer some of the questions that the Journal article either missed or answered incompletely. Hopefully the context added here will contribute to public debate.

Why Did ASU Undertake the Project?
In December 2003, three distinguished university CIOs were asked to review ASU's aging administrative systems. Their exceptionally frank report made it clear that:

  1. ASU's administrative systems were on the verge of collapse;

  2. ASU's administrative computing strategy was fundamentally flawed, focused more on conserving resources than on replacing the aging systems - a replacement needed to mitigate the serious risk of total systems failure;

  3. ASU's lack of an effective replacement strategy and the "can't do" attitude of its IT staff made any contemplated replacement plan extremely risky;

  4. In the opinion of the reviewers, replacing the legacy systems at ASU, one of the nation's largest universities, would require 5-10 YEARS (and an investment of some $70-$100M).

(For more on this, see How Deep Was the Hole?)

What was Unique about ASU's Approach?
At ASU we advanced a unique approach to cost-effectively deploy our Oracle/PeopleSoft SIS and HR/Payroll system. The approach had three key components:

  • First, we used a strategy we called Regents Vanilla to quickly win and maintain strong cross-university executive support. A key component of the Regents Vanilla approach was our commitment to base the system deployment on an existing implementation, rather than the usual approach of building the system from scratch;

  • Second, we used an approach we call Implement, Adapt, Grow. This approach allowed us to contain scope creep and included a commitment to stick to a well-planned schedule of rolling component deployments, and to bring in additional resources to correct defects exposed by our extensive system testing, rather than delay deployment. "Implement, Adapt, Grow" includes an implicit acknowledgment that, despite the high level of testing that we actually conducted, no amount of testing would produce a perfect deployment, leading to our commitment to rapidly respond to defects in the system in the first few weeks of deployment. This is the reality of every major new system deployment in every organization. We may have just been more honest than some to admit it up front, but the perspective gave us the flexibility to rapidly find and resolve problems;

  • Third, we used a Strategic Technology Alliance to reduce our time to market. Our alliance strategy included decisions to externally host the application and to choose a single partner (CedarCrestone) for both implementation and hosting.

How Big Was This Project?
At 65,000 students, ASU is one of the nation’s largest research universities, located in the rapidly growing Metropolitan Phoenix area. In February 2006, ASU began a project to replace seven of its core administrative systems — Admissions, Financial Aid, Student Records, Registration, Talent Acquisition and Payroll. Eighteen months later, at a development cost of about $15M, the replacement systems are fully operational. Six of the seven major components - the exception being payroll - went into service with negligible interruptions to University business (apart from the enormous work load placed on the dedicated cross-university team that labored so hard to make things go as smoothly as possible).

Was the ASU Approach Successful?
It's only after an institution begins to reap the benefits of its new systems that the whole story comes into focus. However, some facts not covered by the WSJ article point toward success:

  • ASU enrollment grew during the period of deployment.

  • ASU successfully registered approximately 65,000 students for the Fall ’07 semester with very few visible glitches. At the peak of the drop/add frenzy, ASU ran nearly 6,000 concurrent sessions with decent and consistent response time - and no crashes. Not many SIS systems can make this claim on their first go-live (or second, or sometimes third).

  • Financial aid was distributed accurately in a timely manner.

  • From concept to final cut-over, the project was completed in 18 months for about $15M. For an institution of ASU’s size and complexity, previous project forecasts put the costs at many multiples of $15M and at least a 5 year implementation time frame.

But Payroll Was a Disaster, Right?
ASU regrets every payroll difficulty that we caused our employees and we are working very hard to rebuild trust in our new payroll system. Clearly, if we had it to do over again, there are things we would change. No question. And once the dust clears, I’ll be happy to share our conclusions about what we did right and what we did wrong.

The basic facts are these: we underpaid around 3,000 of our 12,000 employees in the first three payrolls. Half of those underpayments were for $100 or less. 84% of the underpayments were managed within 4 days of each payroll. But that still left several hundred employees with short checks that were not corrected quickly enough. To make matters worse, the majority of the people affected were hourly employees, who tend to be least able to financially weather missing or late paychecks. As the Journal reported, it was not so much that the software failed us, than that our employee support environment wasn't up to the task.

That said, the great majority of our employees were paid correctly and accurately. By the third payroll, the error rate for the new payroll system (4%) was 33% lower than the error rate of the system it replaced (6%). We successfully ran Open Benefits Enrollment for all employees with barely a ripple. The web time clock - which proved to be a bad cultural fit for our organization - was replaced with a more friendly system by the third payroll, and that is running smoothly.

Right now ASU is working through the process of adapting to the new system. We're fixing things that need improvement; we're implementing new ideas and new interfaces. It's safe to say we are at the low point of enthusiasm for the system, which is not atypical of the adoption cycle for systems of this magnitude. But our community is working with us patiently to advance and improve. Most importantly, we are safely on the other side of the river. All of our investments are going toward the future. There is no risk of having to retreat to the legacy systems, no risk of being unable to run the business on the new system. We look forward to interacting with the Higher Ed IT community over the next several months to learn the lessons, both positive and negative, that this implementation can teach us all.


ASU Student September 28, 2007 at 2:22 PM  

"...ASU’s administrative systems were on the verge of collapse; ASU’s administrative computing strategy was fundamentally flawed, focused more on conserving resources than on replacing the aging systems - a replacement needed to mitigate the serious risk of total systems failure..."

Having worked for a large corporation before, I have experienced first-hand the enormous frustration of not having the proper technology in place to get the job done.

I am elated to be attending a University that seems to be on the right track with the use of proper technology. Of course there were going to be hiccups with the roll out of a major new system. But can you even IMAGINE the backlash from employees and students if the old systems had crashed without the new ones in place?!?

While I have never experienced the stress of having to take out a personal loan to cover a missing paycheck, I have a feeling it would have been even more disastrous to do nothing and allow the old systems to rot on the vine.

Shaun Usman,  September 28, 2007 at 3:27 PM  

"The web time clock - which proved to be a bad cultural fit for our organization - was replaced with a more friendly system by the third payroll, and that is running smoothly."

I wanted to comment on this -- Dr Crow had held a 'Coffee and Conversations' event in the MU in, if I recall right, June. There, someone had brought up and asked about the, at that point, rumors of an upcoming web time clock system. Dr Crow answered quite clearly that he was not in favor of any such system and that in his experience, systems like that never work. I just think it is interesting how soon ASU proved his point...

Nancy D.,  October 2, 2007 at 4:01 AM  

Did anyone notice... even the Wall Street Journal has a hard time keeping their error rates low. In the second to the last paragraph "The most recent payroll had a 4% error rate, which is BELLOW (oops!) the 6% error rate the old system traditionally had. Some kind of cosmic karma perhaps.

Matt R,  October 18, 2007 at 6:04 AM  

As a student, my acocunts were severely affected by the transition. After repeated attempts to contact administrators (about two and half months) my accounts were eventually corrected. Because of this , I was not able to buy the books I needed for class until almost one month into class. Why have these problems not been addressed and are there lawsuits pending?

Dr. Robert J. Sellani December 4, 2007 at 1:33 AM  

Hello. I have read all the postings regarding the problems encountered with the payroll implementation and have some questions.

Does the payroll function report to Accounting or Information Technology or Human Resources, (without a direct line to the CFO)?

Assuming the function reports to the CFO, was there anyone from the Accounting function responsible for the implementation? Where were the Accountants during this implementation? The accounting function, particularly if there is an Internal Audit function, will need to have documented how the transition from the old system to the new system took place. External auditors will be looking for that documentation to assure proper controls were in place as the migration took place.

Was any attempt made to run parallel or to test the output with specific transactions as they would be done by the users? Error testing to see it an employee could enter data out of the normal range? I beleive the answer was there was no parallel testing done, due to the implementation philosophy of Implement, Adapt, and Grow. In my former world, we called that the Big-Bang theory of systems implementation and it is not without risks!

Also, it seems the main culprit was the web-based time clock. Was this problem due to a lack of acceptance of the new technology for data entry by hourly employees? Was there any attempt to validate the system prior to implementation?

In your opinion, was there sufficient training regarding the web-based time clock prior to systems implementation to achieve buy-in from the employees?

I am a bit confused with the payroll system that was implemented. Was this a PeopleSoft Payroll product or home grown from NAU?

From an Accounting perspective, a 4-6% error rate is very unacceptable, particularly where money is involved. As a former Accounting Director for a Fortune 50 Division, I can't imagine payroll having any error rate! It is one thing to get inventory valuation wrong but getting someone's paycheck wrong is asking for big trouble.

After reviewing all of what has been written, I think he implementation seems to have gone pretty well, given the complexity. Every once in awhile, there is a "gotcha" in a major implementation and it looks like this one hit a very sensitive area. I would appreciate your comments as I am writing a paper regarding this subject. Thanks! Bob.

Kelly B March 4, 2008 at 2:47 AM  

As someone that works in IT at another university, I would be interested in an update now 6 months later.

Kelly B March 4, 2008 at 2:50 AM  

One other comment..I agree a 4-6% error rate seems high in payroll. Also, you will only ever hear from people that were underpaid and so who knows if the number is higher but those overpaid go unreported.

mr ADSL March 15, 2008 at 12:28 PM  

I agree with some others here. WOndering how the process is going later on. We have a error rate on payrolling of 1.6-1.8%. Including 0.5% from errors that are favorable for recipient.

Aoleon May 1, 2008 at 6:32 PM  

I am glad that they have corrected these payroll issues.