For those of us who review the Oracle installation documentation, we know that Oracle suggests that you configure a substantial amount of swap space. If you have between 1 and 2G RAM, you need to configure 1.5x RAM for swap space. For 2 to 8G RAM, swap space has to equal RAM. For RAM more than 8G, swap needs to be ¾ RAM.
No doubt your system administrator (in Unix) will tell you that that is more swap than you need. He may tell you that you only need one or two gig of swap when you have plenty of memory.
I can tell you from personal experience, that your system administrator is wrong if he tells you that. I have been there, and done that as the saying goes. Make sure you configure the full amount of swap recommended by Oracle, more if possible. Here’s why:
In a Unix operating system, there are two methods of allocating swap, eager swap and lazy swap. Under lazy swap, swap space is allocated only when needed, that is when an active process needs additional memory and all real memory is used up. If there is insufficient swap space available, the Operating system will create more swap space by killing idle processes. Obviously this is a dangerous and undesirable situation. A process using eager swap will allocate swap as it allocates real memory. If your oracle database has 6G RAM allocated in the SGA, it will allocate both 6G RAM and 6G swap. If there is insufficient swap space, it won’t kill other processes to make space. It will lock up the program attempting to allocate space, and eventually any other processes on the system that need swap space will also hang. In Solaris, I have seen such errors as ‘unable to fork a process’ or Solaris error 12, not enough space. In oracle, you will get messages like this: TNS-12500, TNS-12540, TNS-12560, TNS-00510. Usually your server will lock up shortly after the ‘not enough space’ or ‘unable to fork a process’ message.
The example above actually happened. It was my fault. I had given the Unix system administrator the system requirements, including the amount of swap required. However, I had not followed up to make sure the settings were actually in effect. As it turns out, the Unix administrator had only allocated about 1/3 of the system RAM for swap, relying on his knowledge and understanding of swap. This was my fault because as an Oracle DBA, it was my job to verify everything before you hand the installation over to the users. That includes the system level settings. If you can’t see the settings (but as a DBA you should be able to see the settings), you have to rely on your system administrators word. But I could see the settings, I just didn’t follow up and verify them.
Some Oracle DBA’s I have talked to have allocated twice as much swap space as system RAM, following the older Oracle guidelines. After doing some researched I have discovered that Unix can be configured used to hold an image of the operating system processes for diagnostic purposes in the event of a system crash. Thus, my recommendation is that the minimum swap space should be equal to the available RAM, and configuring twice the amount of RAM is the maximum amount that should be needed. Despite what the official Oracle documentation says.
July 30, 2008 at 7:02 pm |
Hello Andrew – We have similar behavior as you have described above. For several weeks we saw memory fragmentation after upgrading the netbackup client on the server. The issues with the server started appearing right after this upgrade. We decided to remove the client and after that was done, the fragmentation went away. But 2 weeks after that change was put in place, the errors started reappearing in the listener logs. The “top” utility shows that the server is healthy – enough memory, enough swap, cpu idle etc. There are over 900 connections into the database and we have tuned the database memory parameters and have kept up with the tuning of PGA/UGA/SGA/Sorts etc. I stumbled upon this article and thought that I should directly ask you. I would appreciate if you could help. How do I know what Eager or Lazy swap are we using? Also, any other details on this particular problem and its diagnosis and resolution would be mighty helpful as this is our primary POS database. Thank you and regards, Kuldeep
July 30, 2008 at 8:04 pm |
If you have sufficient swap configured, then it is unlikely that your problem is a lack of swap space. How much RAM is on your server, and how much swap is configured? It sounds to me more like you are running out of user memory, in which case you need to consider using MTS instead of dedicated connections.
July 30, 2008 at 9:06 pm |
[oracle@xxxxxxx$ swapon -s
Filename Type Size Used Priority
/dev/xxxxxx/xxxxxx partition 2097096 53204 -1
That is what it is configured to. We have 6GB RAM on a 32 bit OS. The SGA is at 900MB [approx.]. We did some analysis on user memory.
SQL> set heading off
SQL> set echo off
SQL> set feedback off
SQL> select ‘Total current memory for all sessions: ‘ || sum(value) || ‘ bytes’
2 from v$sesstat a, v$statname b
3 where name = ’session uga memory’
4 and a.statistic# = b.statistic#
5 /
Total current memory for all sessions: 92265108 bytes
SQL> select ‘Total maximam memory for all sessions: ‘ || sum(value) || ‘ bytes’
2 from v$sesstat a, v$statname b
3 where name = ’session uga memory max’
4 and a.statistic# = b.statistic#
5 /
Total maximam memory for all sessions: 127797696 bytes
SQL> select ‘Total current memory for all sessions: ‘ || sum(value) || ‘ bytes’
2 from v$sesstat a, v$statname b
3 where name = ’session pga memory’
4 and a.statistic# = b.statistic#
5 /
Total current memory for all sessions: 183325780 bytes
SQL> select ‘Total maximam memory for all sessions: ‘ || sum(value) || ‘ bytes’
2 from v$sesstat a, v$statname b
3 where name = ’session pga memory max’
4 and a.statistic# = b.statistic#
5 /
Total maximam memory for all sessions: 183782376 bytes
Thanks,
Kuldeep
July 30, 2008 at 9:17 pm |
What is your Oracle version? If it is the 8.1.7.4, then you should have 12G Swap (twice RAM). If it is Oracle 10, then you need 6G swap, in either case it looks like you have insufficient allocated. I dont know how to tell which type you are using, but the simplest thing to do is to create the appropriate amount of swap space and run from there. If the problem does not happen, you are done, if it does happen again, look at the MTS settings.
July 30, 2008 at 10:56 pm |
Realistically, to be able to handle the number of connections you are talking about, it would be a good idea to go to a 64 bit OS, and use about 32G RAM.
July 31, 2008 at 1:20 am |
> it would be a good idea to go to a 64 bit OS, and use about 32G RAM
Why do you suggest that? The SGA is reported to be 900MB and the “session pga memory max” is reported to be 175MB making for just over 1GB of memory usage as I see it here. Why the need to jump from 6GB of RAM on the system to 32GB RAM when Oracle looks to be using about 1GB of the existing RAM?
Thanks,
Mark
July 31, 2008 at 3:08 am |
What is the Oracle version? Do you understand that there is a lot more to Oracle memory usage than PGA Memory? PGA is only the memory that the individual user processes need to run. The remainder of the memory on the server is used in the Oracle server processes and memory segments. For a 900 user system, it would not be at all unusual to have 100+G RAM devoted to the Oracle instance. It might be a good idea to take a class on Oracle concepts, or hire a consultant to run a tuning exercise on your Oracle instance.
Oracle will not run in much less than 256M (that is, SGA_TARGET=256M). Oracle will not use more memory than it is granted by the PGA_TARGET and the SGA_MAX_SIZE or SGA_TARGET, the starting point for determining the appropriate amount of memory for your instance is the awr or statspack report.
July 31, 2008 at 4:18 am |
> What is the Oracle version?
It looks like it is 8.1.7.4 based on the original posting.
> Do you understand that there is a lot more to Oracle memory usage than PGA Memory?
Yes. My question was why the recommendation to up the server memory to 32GB based upon the provided information? What in the provided information indicates that such action is necessary?
Also, if there was insufficient swap space (i.e. the swap was exhausted/filled) would there not be a message in the system logs to indicate that event?
Thanks,
Mark
July 31, 2008 at 12:33 pm |
In my experience, no there is no message about swap being full. Possibly because the space is actually getting marked as used, not being physically used for swapping? You would need to talk to a Unix System administrator on that one.
The memory issue is really just an experienced guess. A 900 user system would not typically perform well in the amount memory that they have allocated. 32G is a good place to start on tuning the instance memory. However, they are on 32 bit Oracle, so they would also have to switch to 64 bit to do a lot of good.
July 31, 2008 at 12:54 pm |
Andrew/Mark – Thank you for helping out here. The information above might be relevant to our current problem. We will work with the DBA staff and the system administrators. I will post what we found or otherwise…
Thanks,
Kuldeep
August 12, 2008 at 11:01 pm |
Andrew – As promised here is the update –
1. Our Senior DBA has implemented MTS after careful consideration and we are seeing major improvement in memory/session management. The system is still under observation.
August 12, 2008 at 11:19 pm |
Excellent, glad to hear that your problem has been resolved.