LINUX - Memory

[Ashenfury]

Man I am in mortal combat with the deadly ORA-4030 error. This error means that the Operating System is not relinquishing memory to the database that is being requested by the database. All memory requests are supposed to be within the SGA+PGA limits (1600MB SGA and 2GB PGA).

This is a 32bit CentOS Operating System. I have set the Kernel Parameters to Oracle recommended values (in /etc/sysctl.conf) and I've set all pertinent shell settings to unlimited values (in /etc/security/limits.conf).

Every time I run a full backup the damn thing fails with ORA-4030. I've read about changing the SGA attach address in Linux but I've done that once and I never want to do it again. Does anyone have any other ideas? Changing OS and Database is not an option you smartasses :P

[Malstrom]

This was written about 32-bit windows, but likely applies to most 32-bit systems.

If you do not have sufficient address space to allocate PGA memory, you can run into ORA-04030 errors or system commands can fail with OS error 1450 or 997. The following solutions are suggested:

Run a smaller workload and use less users.

Run with the /3GB option and use Windows Enterprise Edition. Note that the /3GB option has the potential to deprive the operating system of necessary kernel resources, which can lead to Oracle instance failures. See Note 297498.1: Resolving Instance Evictions on Windows Platforms Due to OS 10055 Errors (OS-1540) on My Oracle Support (formerly OracleMetaLink) at:

https://support.oracle.com

Apart from not using the /3GB option, you can tune the amount of RAM that is available to the kernel by using the /USERVA parameter. Visit the following URL for details:

http://support.microsoft.com/default.as ... -us;810371

Determining the correct value is a trial and error process, but 2048 has worked for some users.

Oracle can support Physical Addressing Extensions/Address Windowing Extensions. This is another way in which you can get the Oracle instance to use large amounts of RAM (64G on 32-bit Windows).

Use VLM with a small window so that address space is allocated for PGA.

Reduce the per thread stack space usage using orastack.

This issue is tracked with Oracle bugs 4552171, 4552150, and 4552097.

Some of those fixes obviously will not work for you, but I would not be surprised if your problems are related to the 32 bit nature of the system. FWIW there are few applications that benefit more from 64-bit (or greater) than databases.

Also note that when you say

I have set the Kernel Parameters to Oracle recommended values (in /etc/sysctl.conf)

. Make sure these are recommended values for a 32 bit system.

Also note, that despite the fact that it says on this chart:

http://www.centos.org/product.html

that you can have 16GB of memory, obviously on a 32-bit system you need to do some special magic (memory address extension of some sort) to access the memory. For very finely tuned memory architectures like the PGA and SGA this may or may not work as advertised.

For example, you are giving 3.6GB for the database, which is within 32 bit addressing, but there is also going to be memory taken up by the OS, and this might conflict with your maximum "natural" memory addressing.

I might try reducing the PGA+SGA down to 2 GB and see if that gets rid of the error, and then scale it up a bit at a time to see what your system can tolerate before generating the error again.

Apart from that, I would not be able to help you much without actually jumping on your system,which I am not willing to do. Sorry. This stuff is mostly a shot in the dark.

Mal

[Ashenfury]

Yup, the Kernel limits are set for 32bit and I've not only followed Oracle documentation on 32bit Linux memory settings but I also have an Oracle Support ticket opened and we're nearly finished clearing the database of all memory problems.

So with that I know that it's the OS (almost positive). I couldn't let an outsider into the system even if I wanted to, don't worry :)

EDIT: Yes, 64bit is certainly the way to go but it's an old system... bah!

[Malstrom]

FWIW I always found Oracle a pain to work with as well. As I had mentioned in an earlier thread, if the company makes almost as much money training people to use their systems as they do selling the systems themselves, then it's in their best interest to obfuscate things as much as possible to get people to pay for training.

Hit your boss up for some training at Oracle University. It's overpriced but if someone else is paying for it it can be fun. Last time I was there was for security courses and the only other people in the class were 2 german military hackers. Good times. Good times.

Mal

[Ashenfury]

I've taken 6 Oracle University classes and none of them taught me how to fix Linux memory. It was worth a shot in the dark to see if somebody had dealt with the issue. As it stands I'm patching the database to 11.2 and then applying a one-off patch. The engineer is fairly certain this may fix the issue but I'm not so confident.

[Duskheron]

*Pulls out 12 year old OCP certificate*
*mutters*
*Puts certificate back in drawer*

God, I am so glad I got out of government contracting.

**goes back to drawing silly pictures of my Chiss Imp Agent**



edit: of course, I do, kinda, sometimes, not often, but sometimes... really miss the money :P

[Grainger]

You're pushing 3.6G on a 32 bit system. That's nasty, nasty, nasty. The default user/kernel split will be 3G/1G. I'm assuming that you've tweaked that to at least try to squeeze the kernel into 0.4GB.

Even so: http://download.oracle.com/docs/cd/B283 ... pi_vlm.htm is about the best I got. Unless you've got hugemem or VLM stuffs running 1.7G is the largest SGA size you can have.

[Malstrom]

Grainger +1

[Ashenfury]

Valid observation. I pushed it up and beyond to try to get around this 4030. It was actually over 1.6 to begin with so I'll try pulling it back and see if the backup completes. If the split is 1/3 then shouldn't 3gb be the largest SGA? I've read that before in documentation and that was my assumption. We're trying to phase out these 32bits but there are already a lot in the field.

[Canaie]

What was the reason that you couldn't run 64bit CentOS? The cpu on the box doesn't support it?

[Malstrom]

If the split is 1/3 then shouldn't 3gb be the largest SGA? I've read that before in documentation and that was my assumption. We're trying to phase out these 32bits but there are already a lot in the field.

From the docs Grainger posted above:

"In a typical 32-bit Linux kernel, one can create an SGA of up to 1.7GB size. Using a Linux Hugemem kernel enables the creation of an SGA of upto 3.6GB size. To go beyond 3.6GB on a 32-bit kernel, you must use the VLM feature."


It's not necessarily a "but I have this much RAM" thing.

Mal

[Ashenfury]

Right. Well here's the kicker: During a reboot right after posting that last message it failed POST with a motherboard error. Dispatching technicians now :)

[Ashenfury]

What was the reason that you couldn't run 64bit CentOS? The cpu on the box doesn't support it?

Because it's an old system. We adopted 64bit a couple of years ago. These are mission critical high availability servers that never go down. Turning off one of these servers can deny dial tone to wireline customers and drop calls for cellular ones. Once these systems are in place, they stay that way until removed or upgraded.

Funny that you mention 64bit CentOS because that's what we like to deploy (v5.3). It behaves just like a Red Hat Enterprise build which is what we used for 10 years. Sadly this customer won't upgrade this system for several more years.

To really answer your question though: Yes the hardware does support it. At the time we just weren't using 64bit yet. None of our proprietary drivers were compiled for 64bit either and we had no clue how easy it was to do so.

[Malstrom]

Right. Well here's the kicker: During a reboot right after posting that last message it failed POST with a motherboard error. Dispatching technicians now :)

Fantastic! Now that the machine is down (and not your fault) - time to upgrade to 64 bit!

Mal

[Ashenfury]

I've got my boss agreeing to an OS rebuild if we find any corruption. An OS rebuild would mean a database rebuild and these guys have a year of searchable data imported so we are attempting to avoid that. Let's privately cross fingers for corruption... LOL!

EDIT: Actually I just thought about it and I have backups of the database. It would still be undesirable downtime.