Tuesday, August 09, 2005

"Closing the Gap"

Got another scorpion last Friday. Well, in fairness, my son Andy got him. My wife spotted it in one of the moving boxes, and did the only sensible thing. She promptly closed it and waited till I got home. The box was full of that paper that movers seem to use so very much of. Anyway, while I was gingerly pulling individual sheets out of this box one at a time, my son was tearing into it willy-nilly, clearly not appreciating the gravity of the situation. When we were done, the box was empty, and still no scorpion. So we started folding the paper to throw it away and, of course - THE SCORPION FELL OUT ON THE FLOOR. It was hiding you see, as scorpions do. Same color as the paper. Squish. Current score: Sanniers 2, Scorpions 0.

Apart from scorpion hunting, a big chunk of last week was devoted to learning about ASU's Student Information System (SIS). Taking this opportunity to showcase my ignorance, here is my 30,000 foot view of the SIS.

  • SIS is a homegrown system that is a little over 25 years old.

  • It is based on IDMS, which owner Computer Associates describes as a Network and Relational DBMS for Ultra-High Performance

  • It runs on an IBM mainframe.

  • It handles the high volumes of transactions associated with student record keeping efficiently and reliably.

  • Over the years, it has been successfully customized to represent many of the nuances of ASU business practices.

However, there is growing concern among those responsible for maintaining and enhancing SIS that it is rapidly approaching the end of its useful life. This concern is based primarily on the following reasoning:

  • While the existing system does, for the most part, support ASU's current mode of operation, the challenges of transforming ASU into a New American University will require extensive, continuous change to the existing systems.

  • Because of its IDMS roots, its age, and its modification history, SIS has become increasingly difficult to modify and adapt and now borders on unmaintainable.

  • The system's implementation language, COBOL, is a dying tongue. The talent necessary to maintain and enhance the system is a precious, limited, and dwindling resource.

In short, in the words of Mr. Scott, "I canna change the laws of physics Captain. She canna take any more!".

So those who know the system best say we need a new starship -- a new SIS system, based on vendor supplied technology, such as those provided by Oracle, PeopleSoft, SCT, or SAP to name a few. Such a system would replace the IDMS database with a more modern, more flexible, relational database. Depending on the availability of resources, the system could be extended to replace not only SIS, but also the Financial and HR systems. The hope is that the new system's more modern design would provide the flexibility needed to support the ongoing needs of the New American University. The more integrated the systems, the more straightforward it should be to support the flow of information back that will allow for the amazon.com-ification of all of ASU's information systems. The result...a better working and learning environment for ASU students, faculty and staff; better access for the community at large.

Which would be great. I mean, who wouldn't want a new starship? Unfortunately, there are also some downsides.

  • The cost of new starships is, well, astronomical. Without splitting hairs, a new system will cost on the order of several tens of millions of dollars, depending on who's counting, and on what you're counting.

  • New starships take a long time to make. On the order of years. Several. Certainly not less than 2. Usually more, again depending on who's counting, and on what you think of as "done".

  • While in the end, new starships are better, stronger, and faster than old starships, at first, new starships are worse, weaker, and go slower. They also feel strange to the users and they don't work all the time. This is known in the biz as the ERP learning curve. (By all accounts, implementing an ERP takes time, and there is a steep learning curve. "You have to grow live, instead of going live," said Gregg Jacob, the information systems and services manager for Tuolumne County, Calif. "It's a heuristic process. You can't expect to get it right the first time.")

  • Building new starships doesn't always go well. Example1 Example2 Example3 Example4

Given all these risks, it's easy to say, "Ahhh, Mr. Scott could always find another warp or two in his drives come crunch time. Those techies always say they can't so it'll look cooler when they can." And that's certainly true sometimes - techies want to be appreciated just like everybody else does :).

But, sometimes there's nothing left. Sooner or later, the ship simply, really, no fooling CANNA TAKE ANYMORE. That happened recently at a Big 10 University who will go unnamed here. The old system just gave up the ghost. Couldn't get the students registered. Had to get out the pencils and paper.

That's the wrong time to be buying a new car (I mean starship). You can't get the best deal when your current one just won't go anymore and you still have to get to work. You don't get the best price and you don't get the best car.

This while process is risky. Its risky to get a new system and its scary not to get one.

So what do I think? Well, some of my friends think SIS must be replaced as soon as possible. Some of my friends think that we need to find a less expensive alternative that still takes us in the direction we want to go. Me...Well at the moment I support my friends.

Most of the conversation about the issue thus far has been among IT savvy folks, the ones charged with making the wheels turn. Many are convinced that a new system is the only option. Up till now though, as near as one week's asking around can tell me, there hasn't been a lot of discussion by the leadership - the Dean's, the VP's, etc. - about whether this is the year to pull the trigger or not. There is always pain in an organization this big, so the question is: Has the pain reached the level where the leadership will commit serious money to a long term solution? Because that's money that won't be available for other really important things. I think that dialog needs to happen soon.

We are fortunate to have Max Davis-Johnson on board, a veteran of NAU's PeopleSoft implementation. Max and company helped develop a replacement for an SIS system with a common heritage to ASU's. They made it work there too, and pretty quickly as I understand it. Max is among those leading the charge for a new system, under the battle cry "Close the Gap". He's a vet of this particular fight, so I think his position has to be respected.

That being said, what's missing for me at this moment is the other voice, the one saying, "Captain, I'm not sure we need a whole new system. By attaching the gaflommitz intake to the rheostatic injubinator, we can isolate the existing SIS system, keep it running, and still achieve all, or almost all, the improvements the New American University will need for the next several years."

"But that's never been tried before. We'll all be killed..."

I think you know how the rest of the story goes. Our missing hero does in fact connect the gaflommitz intake to the rheostatic injubinator and then -- depending on which version of the story you're writing -- the universe is saved or ... we all die.

I'm not at all sure about that gaflommitz intake, but I do wish there was a technical champion for an incremental approach. Not because I'm convinced that she would be right. She might well be wrong. But the dialog between the champions of the two approaches would help everyone make better decisions. Not only that but she also might spark new ideas on how to make do with the existing system while we "grow into" a new one. Even if the best thing is to craft a whole new system, there'll be a period of several years during the reengineering process where we'll have to live with what we have (or "run what we brung" as we used to say in the software biz). I'd just feel better about the whole debate if the technical champion for the status quo would make herself known...and I don't think I'm the only one.

Of course, I could be all wet. I have been before. I've really only been at this a week, and part of that time's been spent worrying about...I mean hunting...scorpions. But if a champion for the old SIS is out there, now is the time to come to the aid of your New American University.


pacifictoy August 10, 2005 at 8:22 AM  

Scary scary scary...
The cost of the new starship is astronomical compared to the cost of a holodeck (the C6 at Iowa State University. http://www.vrac.iastate.edu).

Good luck to the captain.
"Make it so!"

schnee August 11, 2005 at 5:26 AM  

Sounds like a fun time to be dealing with tech at ASU! The gaflommitz intake could work, but who wants something the size of a gaflommitz intake sticking out of the window of your ship as you limp across the stars?! I wonder if this contradicts the thought that enough duct tape can fix anything...

Maybe you can find some "pets" which feed on scorpions and let them go to town! :) I hear some cats like to hunt the little devils...

Nancy,  August 11, 2005 at 5:35 AM  

1. If you want to find scorpions before they find you, you can use a blacklight at night to look for them around cracks, fencing, walls, ceilings, and under anything laying on the ground. They fluoresce. (http://ag.arizona.edu/pubs/insects/az1223/#Blacklighting) Chickens like to devour them, but owning chickens has some downsides.

2. I applaud your rapid understanding of the SIS dilemma. I'm not sure that I could champion the gaflommitz approach, however - as fun and heroic as that might be - because the key problem with the current system is irremedial: we are losing, slowly but irrevocably, access to staff who understand the old system's technology (COBOL, IDMS, mainframe utilities, JCL, ISPF, Panvalet). Our own staff retires, grows ill, even dies; we are watching this happen, literally, month by month. What do you do when Scotty is gone?

Nancy Lee

Adrian,  August 11, 2005 at 8:21 AM  

And Scotty is gone isn't he?...

Wondering about the "encapsulation" option? Because we aren't the only people in the IDMS boat. (Southwest Airlines, British Telecom, Miami-Dade).

Aren't there bolt ons to IDMS, available from CA and elsewhere, that add things like web-services or ODBC drivers or whatever (these are one step above the gaflommitz intake I think :) ) that can be used to modernize interaction with the database and obviate the need for as deep a reliance on dying tongues?

Thanks, by the way, for being the first ASU commenter...Hope you are the first of many...

I guess this is as good a time as any to set out the blog's comment moderation policy....

Please comment away... Before a comment will appear on the site, it will go to my moderation queue...I will remove SPAM without notice. If I'm otherwise concerned that a comment is inappropriate to the forum, I will reply directly to the poster, voicing my concerns, prior to approving the comment for display.

I welcome all sincere commenters, particularly those from anywhere in the ASU family. Dissenting opinions are explicitly encouraged. Just keep it collegial.

Derwin August 11, 2005 at 8:58 AM  

I just had to respond to your current blog entry on “Closing the Gap”.

First, I reply as a resident of the scorpion capital of the US. My wife and I live on a 2 ½ acre hobby ranch/orchard in the Lehi section of North Mesa, the original Mormon settlement in the Valley of the Sun. I am told that our orchard was first planted in 1896, and has remained very rural up to this day, with nothing but the Salt River Indian Reservation to the North on the other side of the 202. Consequently, we have much of the native wildlife, including skunks, coyotes, lizards and snakes of all kinds, and especially the scorpions. We typically find about one a month roaming through the house. Usually we find more during the warmer times of the year. There are a few species of scorpions in the valley. Unfortunately for us, we are infested with the “Bark” scorpion, the only species of “medicinal” interest out of the forty-some varieties in this country. It has been a while since there has been a recorded death in the US from a scorpion sting, thanks in a large part due to the efforts of a recently retired ASU professor who used to produce the anti-venom. So suggestion number one: invest in a portable black light; you can get them at a Spencer’s gifts at your local mall for around 10 bucks. I have on occasion performed a sweep of the house, both inside and out, and find it extremely easy to find a scorpion when I want to as they glow a nice bright green under the black light.

Second, I reply as a 20 year veteran of ASU IT, now primarily responsible for the architecture and development of our enterprise identity, authentication, and authorization system known as “EDNA”. It is rather difficult to provide enterprise solutions to the ASU community. I often have explained to my real world corporate associates the difficulties of enterprise perspective in a large educational institution. We the enlightened find more comfort in a “city-state” cooperative much like the Classical Greek model of political organization. Of course, it does have a tendency to create focus and allegiance to our own parochial organizations rather than the institution as a whole, and on occasion we see our own versions of the Peloponnesian wars. So the 2nd suggestion is moderated by the political realities of this great institution. Alan Kay was fond of pointing out how any problem within computer science can be tackled by creating a new abstraction layer. My wife, who also works for IT, has been frustrated for over 3 years because no one wants to take the responsibility for creating a new abstraction layer, an enterprise object model for ASU. The question for ASU is where does enterprise object modeling responsibility reside? The industry as a whole has been slow to remodel IT organizational structure to better reflect the realities of new development paradigms. It is no longer acceptable to have Data Administration responsible for the enterprise data model and IT responsible for enterprise applications. So where should enterprise object modeling reside? Central IT, Data Administration, or CTO? What about CTA … chief technology architect?

It is so much easier to buy enterprise-out-of-the-box, but regardless of whether we buy or build, an object model is necessary to better understand what we need.

Adrian,  August 11, 2005 at 10:19 AM  

Thanks for the response. Haven’t had a blacklight for a few years(didn't realize Spencer Gifts was still open), but I will follow your advice. Also glad to hear that I won’t be killed by the inevitable scoprion bite. Sam DiGangi tells me they really hurt though.

I appreciated your comments on the difficulty of deploying enterprise solutions:

I think your city-state analogy is especially apt (not to mention dead cool). It captures the intricate political and cultural landscape very nicely.
Your second comment also feels right on to me. Over the years, the various city-states, acting independently and jointly, have crafted a interconnected alliance of information systems that enable ASU’s activities. But I suspect that, in true city-state fashion, this alliance was not conceived against any comprehensive plan.

While various people probably hold big pieces of the puzzle in their heads, I haven’t yet found a top down view of the enterprise system as a whole. . I think that you are exactly right when you say that an enterprise object model of the system as is, and as we would like it to be, is critical to whatever modernization/reengineering/improvement process we embark on. And I don’t think we need to pick a vendor or a technology stack before we get that process started.

Where should that modeling function reside? Once again, some of my friends would say one thing, some would say another. At the moment, I’m with my friends on this one too. But these organizational questions will have to be answered as part of the technology strategy I am charged with helping coordinate. So I will have to come off the fence pretty soon.

Thanks for a thoughtful and well considered response. I know that EDNA is one of the University’s IT jewels and I congratulate you on your contributions to it.

Nancy,  August 11, 2005 at 10:57 AM  

In terms of database encapsulation: it’s possible to access an IDMS (CODASYL) database relationally and thereby, among other things, use a JDBC driver along with some other software glue to get to it from a Web front end. We've done a little of that here in IT at ASU. Other less direct ways of bridging to, and duplicating from, our legacy database are currently employed quite successfully here. The format of our legacy data can be overcome from a programming point of view, though it may become challenging to find DBAs who can maintain its structure and upgrade when needed.

But data format is only part of the problem. Our business rules are embedded in data structures and code more or less equally, and interdependently. The number of people who have the training to access these encoded rules is quite limited, yet the rules are precious by virtue of their importance to the University’s day-to-day functioning, of the labor involved in investing them in code over the last 25 years, and of the labor it would take to recreate them from scratch. (“From scratch” means holding interviews with administrative department representatives and extensive business analysis.)

The conclusions to our questions always seem to boil down to resources, or perhaps, cost vs. risk. If I spend “x” amount, how much risk “y” can I avert? If we had the funds to spend on a whole starship, and we wanted it to fit us moderately well, we should probably look for a vendor system that’s working well for a similar university. If we wanted to redefine starships and make ours the head of the fleet… well, I don’t know of a university that’s done it successfully in recent years, and I worry about the reasons for that. (Otherwise I’d think, yeah, let’s collaborate consortium-style with some other schools, a la JA-SIG, and invent a non-proprietary shared-code way of doing things.)

Even if we decide to invest in some bolt-ons while implementing a purchased system, which is probably the least we can afford to do, risk-wise, we should still attempt to extract (from existing legacy staff, code and data – though extracting from staff sounds a little Brothers Grimm) the business rules and objects, to reside in some non-proprietary easily-accessible form. Then, regardless of the particulars of system implementations, the rules and necessary data would be independently preserved. Changes to implementations should always flow from the model downward. System criteria for vendors and internal developers would be based on the model. Validity of processes, if developed independently of IT, could be evaluated against the model. Changes to rules in the model would have to agreed upon by “stake-holders” and published, so that, come the day that we need another new spaceship, the blueprint is ready.


Jon August 18, 2005 at 7:39 AM  

To make the process even more interesting, if ASU is to develope into the "New American University", it must also CHANGE how it currently does business. Without this change, the mindset of our members (Staff, Faculty, Administrators), and the requirements of the new SIS system, will still reflect the old processes and restrictions, which will handicap our growth.


kakdh,  August 24, 2005 at 3:22 AM  

I was first stung by a scorpion in bed when I was 16 years old. It stung me twice in the middle of the back. My back was like jello for the next couple of days and I laid around with ice on my back. I know from personal observation that scorpions have a bad habit of crawling on the ceiling and dropping when they get tired. I suppose that is only one possible way they could end up in a bed. I've been stung three other times as an adult, once on the foot (I was walking around the house barefoot at the time), once on the wrist and one more time in bed on the calf. The one on the calf was the mildest since the scorpion was not full grown. I started hunting scorpions at night with a hand-held black light a couple of years ago. I squash them with a heavily gloved hand or a garden trowel. It has really made a dent in their numbers. I have heard a trigger operated torch is even better since it can reach into crevices. I believe the best months to hunt scorpions are May and June. I also cut down on their food supply by placing boric acid in the seams of the patio and carport slab as well as in all entry ways to the house. That stuff will not kill a bug quickly, but does an incredible job because they don't know it is doing them harm to crawl through it. Here is a good website on boric acid and bug control (including scorpions): http://alsnetbiz.com/homeimprovement/boric_acid.html.
I have also begun to use Diatomaceous Earth for bug control. I understand that it acts much more quickly than boric acid and even acts as a barrier: http://www.ghorganics.com/DiatomaceousEarth.html. If you should get stung don't panic. The pain starts out about like a wasp sting. It just lasts longer and the numbness is spreads to a greater distance. I once saw a young man stung by a scorpion who seemed to be hardly bothered by it at all (I wish I was so lucky)!