IDS Forum

RE: Informix 11.10FC2W5 hangs & crashes

Posted By: Tim Ertl
Date: Tuesday, 30 September 2008, at 4:44 p.m.

In Response To: Re: Informix 11.10FC2W5 hangs & crashes (Mark Jamison)

Hi Mark, I did not know these problems were not related. We ran on the old
server (9.3) for 8 years+/- then we upgrade to 11.10 and we run fine for 6
weeks. Then we had a crash that was determined to be a memory corruption
(anything can happen problem) followed by a HANG a little later. My best
guess was that both were one problem. 20x20 hind sight we had two problems
at almost the same time. What are the odds in that? Especially with
Informix. We applied W5 and things went bizzerk! 1 more crash and 4 more
hangs in two days.

We are currently stable for the past 27 hours (+/-) but everything is
running 5 to 10 times slower. That is understandable for the change IBM
made. I think it must have been a work around rather than a fix. I dunno and
was not able to find out. IBM added a BTSCANNER num=2, threshold=200,
rangesize=500, alice=6. threshold 5,000 is the default and increasing 10x or
100x the manual says should improve performance. So when IBM made it 200
(yes 200) we felt the impact with a tremendous slow down.

Apparently W5 created a new problem when it fixed an old one. IBM has a fix
coming. They have identified my problem with some magic they have and a
patch will be out soon (I HOPE, our performance is bad right now).

I was looking for others with the same problem and I found one other person
but they had not resolved yet either.

Now I think we have a fix coming so I am happy.

Tim Ertl
413-442-9000 x6211

-----Original Message-----
From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of Mark
Jamison
Sent: Tuesday, September 30, 2008 4:00 PM
To: ids@iiug.org
Subject: Re: Informix 11.10FC2W5 hangs & crashes [13539]

Hi Tim,

The reason alice, and/or numbr of BTREE scanners have been added would only
be
in response to a hang, not the crashes.

Sounds like you certainly should have 2 PMR's open, as the hangs mostly
likely
cause the crashes, and the crashes don't cause the hangs.

Getting back to the hangs though. Are they continuing to occur after the
changes to BTSCANNER?
The reason I ask, is that until 11.50 the default for BTSCANNER is 1 thread
and using Leaf scanning. On a busy system, this combination could, although
by
no means is guaranteed, to cause system slowness and blocked checkpoints.
With
leaf scanning, it is very easy for a single BTSCANNER thread to get behind
in
cleaning and never be able to catch up, the worst case scenario with a
BTSCANNER thread in that condition with leaf scanning doesn't allow one
index
to finish cleaning during the time allotted for an active hot list, and
meanwhile the BTSCANNER thread is holding resources needed for a checkpoint
to
complete. Increasing the number of BTSCANNER threads, would help mitigate
this
problem, and enabling Alice mode would likely eliminate the hangs. The
reason
why is that Alice is much faster than leaf scanning. By much faster, I mean
that on your average 1 MB Index, a leaf scan can do as many as 680 I/O's,
while Alice will do 4.

Don't suppose you have an onstat -C all from your last hang? For that
matter,
do you happen to have an onstat -C all now?

Again BTSCANNER was only likely addressing the hangs.

----- Original Message ----
From: Tim Ertl <tim@lmrgroup.com>
To: ids@iiug.org
Sent: Monday, September 29, 2008 5:00:49 PM
Subject: Informix 11.10FC2W5 hangs & crashes [13520]

We have about 6 or 7 MAJOR failures of Infmormix 11.10FC2W5 in the past
few days. I have a pmr going and a tech support person working on it for
several days now.

I am hoping someone else might have seen this problem and fixed it them
selves. I am all Googled out.

We have had Assertions and HANGS on Blocked Check Points. Sometimes we
can stay up for only 3 hours other times we stay up for a whole day.
This morning alone we Hang at 6am and then again at 9. Now we are quite
till 17:30. We have restricted usage to Medical Emergencies Only to
prevent any unnecessary usage. We move some stuff over to M/S Sql
(something some people around here looked for excuses to do).

We have been running 11.10FC2 since Aug 16th no problems. Then all of a
sudden we have an assertion at 3 oclock last Thursday monring. Since
then we have had about 6/7 Informix Failures.
IBM Support had us upgrade to W5 and since we did that we have had 4
more problems. IBM has added and onmode -C alice 6 for some reason I do
not understand. We can not continue like this.

ALL of these problems start off with about an hour of the informix log
saying Txns blocked:1 and checkpoint durations greater than 0. We used
to have zero all the time for these. During the hour before lock up we
have slow periods of up to 5 minutes then all of a sudden things free up
and we go back to normal, on and off hangs. Then after about an hour of
this the system freezes on blocked check point.

Physical log is at 1% used and there are 7 logical logs and only one is
in use. The others are backed up.

I have tons of onstat -a's. Has anyone had some experience with blocked
check points that might be able to shed some light on this.

Why would informix go belly up all of a sudden like this?

Running on a Sun Sparc T5120 32gb memory, 64 virtual processors (cores
or what ever they call them). Ton's of disk space now. We upgraded Aug
16th and things have been great, then crash!

Thanks for any suggestions.
We may need to hire special help!

,,,Tim

****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.

****************************************************************************
***
Forum Note: Use "Reply" to post a response in the discussion forum.

Messages In This Thread

• Informix 11.10FC2W5 hangs & crashes
  Tim Ertl -- Monday, 29 September 2008, at 6:00 p.m.
  • Re: Informix 11.10FC2W5 hangs & crashes
    Obnoxio The Clown -- Monday, 29 September 2008, at 7:07 p.m.
    • RE: Informix 11.10FC2W5 hangs & crashes
      Tim Ertl -- Monday, 29 September 2008, at 8:46 p.m.
      • Re: RE: Informix 11.10FC2W5 hangs & crashes
        ZHAN YU LIN -- Monday, 29 September 2008, at 9:39 p.m.
      • RE: Informix 11.10FC2W5 hangs & crashes
        Stuart McCann -- Monday, 29 September 2008, at 9:53 p.m.
      • AW: Informix 11.10FC2W5 hangs & crashes
        Andreas.KUTSCHE@spar.at -- Tuesday, 30 September 2008, at 3:22 a.m.
  • RE: Informix 11.10FC2W5 hangs & crashes
    Savio Pinto (s) -- Tuesday, 30 September 2008, at 3:37 p.m.
    • RE: Informix 11.10FC2W5 hangs & crashes
      Tim Ertl -- Tuesday, 30 September 2008, at 3:44 p.m.
  • Re: Informix 11.10FC2W5 hangs & crashes
    Mark Jamison -- Tuesday, 30 September 2008, at 3:59 p.m.
  • Re: Informix 11.10FC2W5 hangs & crashes
    Mark Jamison -- Tuesday, 30 September 2008, at 4:00 p.m.
    • RE: Informix 11.10FC2W5 hangs & crashes
      Tim Ertl -- Tuesday, 30 September 2008, at 4:44 p.m.
      • RE: Informix 11.10FC2W5 hangs & crashes
        LARRY SORENSEN -- Thursday, 2 October 2008, at 12:34 p.m.
        • RE: Informix 11.10FC2W5 hangs & crashes
          Schmitz, Rob B [EQ] -- Thursday, 16 October 2008, at 10:54 a.m.

IDS Forum is maintained by Administrator with WebBBS 5.12.