A Catastrophic failure exposes IBM weaknesses.

[Guest.Visitor]

Brian, I expect that Dale doesn't like to be challenged. But, isn't that what life is all about? Constantly challenging each other? Life would be pretty boring if we all just nodded our heads and agreed 100% of the time. chuck Opinions expressed are not necessarily those of my employer. "Brian Singleton" wrote in message news:6ae6c5bf.8@WebX.WawyahGHajS... | I kinda like Chuck's posts, most of them anyway. ;-) | | If you don't want to hear from someone but still want to hang out, you can use a newsreader with a kill filter to access the forums. | | I read every post here and I've never seen a reason to killfile *anyone*. We're all civil. There's a healthy discourse going on here, and we should strive to keep it that way. | | Brian

[David Abramowitz]

I hope that Dale reconsiders his decision. I'd like to think that one can disagree, even be diametrically opposed and keep it on a higher plane. In this particular instance there is a contributor who is in trouble. If assistance can be offered, that's great. The assistance can come in the form of a suggestion of something that has not been tried, or pointing out an error of something that hasw been tried. Either way, a goal has been reached. If anyone including myself comes across as a supercillious self righteous bastard, it does not give anyone else license to say so. With careful wording, you can get something off your chest without having to remove your shirt. Dave

[Guest.Visitor]

David said: "you can get something off your chest without having to remove your shirt." I'm not sure I needed that visualization! It isn't a pretty sight, especially mine. ;-) chuck Opinions expressed are not necessarily those of my employer.

[T.Stockwell]

Chuck, It's been clear for some time that IBM's focus on service has been shifting for some time. What ever happened to "The customer is always right!"? However, your experience with IBM service echoes similar problems that customers of all hardware platforms have been experiencing. IBM taught a lot of AS/400 customers to treat their maintenance contracts like insurance policies, when in fact, in today's service market, all a maintenance contract provides you with is authority to access service. It's a retainer for CEs and SEs, to be available to help. We don't buy the service, or any sort of quality of service, but only the authority to make the request for service. Fortunately, most of IBM service people are "real people" who are sympathetic to the customer, but getting to their skills will continue to be problematic and expensive. Getting past the gatekeeper will continue to be a difficult problem. Congratulations on getting the system back on line. I thought your description of the process was excellent. Yeah, the frustration shows, but its a normal part of a recovery after a catastrophic event. (I speak from personal experience, having inherited multiple failures of HP minis and IBM equipment -- two at wineries in the Napa Valley after a flood and one at an apparel manufacturer after the SF earthquake. You get to a point where you just gotta start throwing hundred dollar bills at the problem, while you feel like everybody's got you at their mercy. Meanwhile, you'll never know what really caused the failure, because the service personnel will always blame whatever device or module was out of warrantee. They're really good at covering their tracks.) Meanwhile, it may be an excellent time to approach your management with other options for disaster prevention beyond the basic tools that IBM provides with OS/400. There are quite a few vendors who can help your team identify strategies that could bridge the gap beyond traditional backup and recovery. What really killed you was contingency planning, or perhaps holes in your contingency plan. If I were in your shoes, I'd ask the managers of each department to estimate the costs the downtime created for the company, attempt to place a dollar amount on the cost, and then use that figure to readdress options for contingency planning in the future. The window of opportunity is short to do this, because management's attention span is usually pretty short. Nonetheless, some recommendations that extends beyond IT's service role -- new services, new disaster avoidance or contingency planning products, etc. -- will provide your management with an assessment and a potential pathway in the future. Each of us who have gone through a major catastrophe feel somehow personally responsible for getting the system back on its feet; management expects us to take responsibility, and to deliver. On the other hand, working around the clock is probably not a part of your job description, and your management knows this. (Great adrenaline rush, but not real healthy over the long haul.) So, at the moment, after the successful recovery from this catastrophe, you're in a golden moment of opportunity to approach your management team and position them to shoulder some of that responsibility too, by funding your team with better tools to prevent future catastrophes. IBM hardware is very good, but now clearly not bulletproof. Service is still pretty good, if you can get past the wicked angels that are trying to keep you from bothering the SEs and CEs. Sell your management on this reality -- with your recent experience as a tool -- and you might be surprised at how much better your team's efforts are received in the future. Ask your BP to make recommendations of contingency tools: he'll certainly help if he can see some dollars in it through the IBM BP channel. Finally, everyone who has witnessed someone else's meltdown will second guess your decision process, the facts, your "professionalism", and your account of what really happened. What's more important is that you got the system up, by hook or by crook, and that your people are now back at work. However, no one who's been through multiple catastrophes will doubt the fact that you really earned your bacon during this experience. Good for you. Tom Stockwell

[Guest.Visitor]

I remember back in 1993 when I went through a similar problem trying to get help from IBM and we were not covered under support. My point is that this is not something new from IBM. If you are not supported then sucks to be you. My two cents on this thread is that as stable as one platform may appear to be, it never replaces High Availability. Now you are talking more $$$$ and it is a hard sell just like Tom says. If you can place actual dollars lost per minute/hour of downtime you just might be able to prove the need for High Availability. On a side note, when I saw the space shuttle go down yet again I thought to myself "With all the hours of research and maintenance put in and if that piece of craft is not bulletproof then just how reliable is any man-made equipment." I am surprized in even my vehicle to be so reliable at this point. Remember -- High Availability -- Scott

[Guest.Visitor]

I remember back in 1993 when I went through a similar problem trying to get help from IBM and we were not covered under support. My point is that this is not something new from IBM. If you are not supported then sucks to be you. My two cents on this thread is that as stable as one platform may appear to be, it never replaces High Availability. Now you are talking more $$$$ and it is a hard sell just like Tom says. If you can place actual dollars lost per minute/hour of downtime you just might be able to prove the need for High Availability. On a side note, when I saw the space shuttle go down yet again I thought to myself "With all the hours of research and maintenance put in and if that piece of craft is not bulletproof then just how reliable is any man-made equipment." I am surprized in even my vehicle to be so reliable at this point. Remember -- High Availability -- Scott

[Guest.Visitor]

Scott, Good point on the space shuttle. We've come to believe our own hype when we talk about the AS/400. Just 24 hours before our disaster I had said to someone, "our AS/400 has never been down due to hardware failure. In fact, I've never been around an AS/400 that's failed." Now I feel kind of silly. Maybe I jinxed it. Kinda like talking about a pitcher having a no hitter in the 7th inning. High availability has it's value. I can actually put a dollar figure on this outage due to lost productivity. However, it may not justify the cost of high availability. Our ecommerce site, which does over $1 million per month, runs on Microsoft ISS and Win 2k server. So, it wasn't dramatically effected by the AS/400 outage. And, dare I say it, the IIS web site has never been down. Damn, I'd better be prepared for another long weekend now! ;-) chuck Opinions expressed are not necessarily those of my employer. "Scott Walker" wrote in message news:6ae6c5bf.14@WebX.WawyahGHajS... | I remember back in 1993 when I went through a similar problem trying to get help from IBM and we were not covered under support. My point is that this is not something new from IBM. If you are not supported then sucks to be you. | | My two cents on this thread is that as stable as one platform may appear to be, it never replaces High Availability. Now you are talking more $$$$ and it is a hard sell just like Tom says. | | If you can place actual dollars lost per minute/hour of downtime you just might be able to prove the need for High Availability. | | On a side note, when I saw the space shuttle go down yet again I thought to myself "With all the hours of research and maintenance put in and if that piece of craft is not bulletproof then just how reliable is any man-made equipment." I am surprized in even my vehicle to be so reliable at this point. Remember -- High Availability -- | | Scott

[Guest.Visitor]

Tom, Let me make one thing clear. All of the service personnel I dealt with were very professional, and knew their stuff. I think that the iSeries has become so complicated that it's hard to quickly isolate a fault. However, they were all troopers and worked hard to get me backup. The unfortunate part is that they are not enough any more. You can never call and get directly through. Even an outage requires a call back. Here's a contrasting service tale: 2 weeks ago our Microsoft Exchange server was inundated with messages from the Fizzer virus. No one here got the fizzer virus but, somehow, in the process our Microsoft Exchange server contracted the "Code Red" virus. We didn't know this at first, focusing only on the fizzer. We eventually contacted Microsoft support, which is charged at a flat rate of $255, and were immediately put through to a level 3 tech support guy. This guy knew Exchange inside and out like the back of his hand. Anyway, to make a long story short, he ended up spending over 8 hours on the phone with my computer support staff correcting the problem. All for only $255. It used to be Microsoft was the joke of the service industry and IBM was the gold standard. It appears now that the roles have been reversed. You're right that now is the best time to get better protection. I will strike while the iron is hot. My BP offers a plan where all transactions are mirrored on a system out of state. When a disaster occurs we just change a DNS entry in our DNS server and that other system takes over without missing a heartbeat. I'll be looking into that pronto! chuck Opinions expressed are not necessarily those of my employer. "Thomas M. Stockwell" wrote in message news:6ae6c5bf.13@WebX.WawyahGHajS... | Chuck, | It's been clear for some time that IBM's focus on service has been shifting for some time. What ever happened to "The customer is always right!"? However, your experience with IBM service echoes similar problems that customers of all hardware platforms have been experiencing. | | IBM taught a lot of AS/400 customers to treat their maintenance contracts like insurance policies, when in fact, in today's service market, all a maintenance contract provides you with is authority to access service. It's a retainer for CEs and SEs, to be available to help. We don't buy the service, or any sort of quality of service, but only the authority to make the request for service. Fortunately, most of IBM service people are "real people" who are sympathetic to the customer, but getting to their skills will continue to be problematic and expensive. Getting past the gatekeeper will continue to be a difficult problem. | | Congratulations on getting the system back on line. I thought your description of the process was excellent. Yeah, the frustration shows, but its a normal part of a recovery after a catastrophic event. (I speak from personal experience, having inherited multiple failures of HP minis and IBM equipment -- two at wineries in the Napa Valley after a flood and one at an apparel manufacturer after the SF earthquake. You get to a point where you just gotta start throwing hundred dollar bills at the problem, while you feel like everybody's got you at their mercy. Meanwhile, you'll never know what really caused the failure, because the service personnel will always blame whatever device or module was out of warrantee. They're really good at covering their tracks.) | | Meanwhile, it may be an excellent time to approach your management with other options for disaster prevention beyond the basic tools that IBM provides with OS/400. There are quite a few vendors who can help your team identify strategies that could bridge the gap beyond traditional backup and recovery. What really killed you was contingency planning, or perhaps holes in your contingency plan. If I were in your shoes, I'd ask the managers of each department to estimate the costs the downtime created for the company, attempt to place a dollar amount on the cost, and then use that figure to readdress options for contingency planning in the future. The window of opportunity is short to do this, because management's attention span is usually pretty short. Nonetheless, some recommendations that extends beyond IT's service role -- new services, new disaster avoidance or contingency planning products, etc. -- will provide your management with an assessment and a potential pathway in the future. | | Each of us who have gone through a major catastrophe feel somehow personally responsible for getting the system back on its feet; management expects us to take responsibility, and to deliver. On the other hand, working around the clock is probably not a part of your job description, and your management knows this. (Great adrenaline rush, but not real healthy over the long haul.) So, at the moment, after the successful recovery from this catastrophe, you're in a golden moment of opportunity to approach your management team and position them to shoulder some of that responsibility too, by funding your team with better tools to prevent future catastrophes. | | IBM hardware is very good, but now clearly not bulletproof. Service is still pretty good, if you can get past the wicked angels that are trying to keep you from bothering the SEs and CEs. Sell your management on this reality -- with your recent experience as a tool -- and you might be surprised at how much better your team's efforts are received in the future. Ask your BP to make recommendations of contingency tools: he'll certainly help if he can see some dollars in it through the IBM BP channel. | | Finally, everyone who has witnessed someone else's meltdown will second guess your decision process, the facts, your "professionalism", and your account of what really happened. What's more important is that you got the system up, by hook or by crook, and that your people are now back at work. However, no one who's been through multiple catastrophes will doubt the fact that you really earned your bacon during this experience. Good for you. | | Tom Stockwell

[Guest.Visitor]

Dale W. Brunner wrote: > Finally, I'm not "grossly unprofessional". I just think you have > soiled these pages with unsubstantiated, and unsupported claims about > your problems. If you have problems, why do you always seem to blame > the service, that you obviously have not entered into a contract > with? Have you ever considered the fact that your hardware problems > are only .05% of all the problems ever posted in this forum? Why > haven't the rest of us experienced the same calamities you have? No > one else seems to have given you the "amen brother". To counter your assertion that noone has given him an amen brother, I'll paste my response from the newsgroup Chuck also posted this story in: What a horrible story Chuck. I recently experienced the degradation in IBM hardware as well. We had an uncontrolled power down to our 730. IPL'd when power was restored - yellow light and SRC code. Call IBM, tell them it's a priority call, the box is down. Normally, this gets you right to a human - not this time, I get music on hold and an offer to leave a voice mail and an offer to call me back! Totally unacceptable response to a machine down condition. After a second IPL, because I was tired of waiting on hold, the machine was on it's way back up by the time I got a human in support to talk to me. Bill

[Guest.Visitor]

IBM Support - decreasing over the years isn't the largest problem. The largest problem is that the customers are not given an understanding of what is covered and what is not. 2 Scenarios within the past few years: 1) While consulting in the midrange market, I was called to reinstall the OS for a hospital. When arrived at the client, determined that a CE had been in and replaced a fan and hard-drive(s). When he attempted to install the OS, he ran into issues and called support. When he called support, he was told to "pack up and leave" as the customer had hardware support ONLY and the OS is not included. After approximately 12 hours, the OS was reinstalled and backups were being used to recover the data. 2) Just this year, we have dropped IBM hardware support for a less expensive hardware support vendor. VP of the department had talked to the IBM rep about the switchover. Within a couple of weeks, we had a CE coming in to remove Service Director Licensed Programming that informed us that we can longer contact IBM via the ECSLINE as we do not have hardware support. Hence, we can no longer order PTFs via the ECSLINE, but have to call and order the CDs. Although PTFs are considered software (along with the OS), the ECSLINE being considered hardware cannot be used for contact to IBM. We would have not been so angry if the knowledge of what implications would have been shared from the IBM representative during the phone conversation.