For those of us who review the Oracle installation documentation, we know that Oracle suggests that you configure a substantial amount of swap space. If you have between 1 and 2G RAM, you need to configure 1.5x RAM for swap space. For 2 to 8G RAM, swap space has to equal RAM. For RAM more than 8G, swap needs to be ¾ RAM.

No doubt your system administrator (in Unix) will tell you that that is more swap than you need. He may tell you that you only need one or two gig of swap when you have plenty of memory.

I can tell you from personal experience, that your system administrator is wrong if he tells you that. I have been there, and done that as the saying goes. Make sure you configure the full amount of swap recommended by Oracle, more if possible. Here’s why:

In a Unix operating system, there are two methods of allocating swap, eager swap and lazy swap. Under lazy swap, swap space is allocated only when needed, that is when an active process needs additional memory and all real memory is used up. If there is insufficient swap space available, the Operating system will create more swap space by killing idle processes. Obviously this is a dangerous and undesirable situation. A process using eager swap will allocate swap as it allocates real memory. If your oracle database has 6G RAM allocated in the SGA, it will allocate both 6G RAM and 6G swap. If there is insufficient swap space, it won’t kill other processes to make space. It will lock up the program attempting to allocate space, and eventually any other processes on the system that need swap space will also hang. In Solaris, I have seen such errors as ‘unable to fork a process’ or Solaris error 12, not enough space. In oracle, you will get messages like this: TNS-12500, TNS-12540, TNS-12560, TNS-00510. Usually your server will lock up shortly after the ‘not enough space’ or ‘unable to fork a process’ message.

The example above actually happened. It was my fault. I had given the Unix system administrator the system requirements, including the amount of swap required. However, I had not followed up to make sure the settings were actually in effect. As it turns out, the Unix administrator had only allocated about 1/3 of the system RAM for swap, relying on his knowledge and understanding of swap. This was my fault because as an Oracle DBA, it was my job to verify everything before you hand the installation over to the users. That includes the system level settings. If you can’t see the settings (but as a DBA you should be able to see the settings), you have to rely on your system administrators word. But I could see the settings, I just didn’t follow up and verify them.

Some Oracle DBA’s I have talked to have allocated twice as much swap space as system RAM, following the older Oracle guidelines. After doing some researched I have discovered that Unix can be configured used to hold an image of the operating system processes for diagnostic purposes in the event of a system crash. Thus, my recommendation is that the minimum swap space should be equal to the available RAM, and configuring twice the amount of RAM is the maximum amount that should be needed. Despite what the official Oracle documentation says.

With the onset of Sarbanes-Oxley, one of the new requirements that DBA’s have to meet is to create instance specific Oracle listeners that are not on the default listener port of 1521. This is really a fairly straightforward process, but one that a DBA may not be familiar with if his site standard is to always use the default.

Most DBA’s create the new listener with the word listener in the name, and an identifying string either preceding or following the word listener. For example, LISTENER_KRB11G or KRB11G_LISTENER. The entry for creating the listener would look like this:

LISTENER_KRB11G =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = krbrhel)(PORT = 1522))
)
)
SID_LIST_LISTENER_KRB11G=
( SID_LIST =
( SID_DESC =
( GLOBAL_DBNAME = krb11g )
( ORACLE_HOME = /u01/app/oracle/product/11.1.0/db_1 )
( SID_NAME = krb11g )
)
)

The command to start the listener is (remember, on Unix the listener name is case sensitive):

lsnrctl start LISTENER_KRB11G

To check the status:

lsnrctl status LISTENER_KRB11G

To set the listener password (also part of Sarbanes-Oxley), see this excellent post on the subject:

http://www.dba-oracle.com/t_listener_password_security_lsnrctl.htm

Below are some questions you might want to ask a potential DBA you are considering hiring. Rampant Press has a very good book with technical questions on this subject,

http://www.rampant-books.com/book_2008_1_oracle_job_interview.htm

but these aren’t nearly so technical. These questions are designed to find out how the DBA thinks, and how well he will fit in with the others in your organization. There are no 100% correct answers to this, but there incorrect answers.

1. Describe your work experience.

2. Describe your activities during a typical day in your current job?

3. What do you do when priorities change quickly? Please give an example.

If she answers ‘finish my current task’ before looking at the requirements of the new one, she has given a wrong answer.

4. It is 2:22 am Monday. A database has crashed and you are the on call DBA what steps will you follow to bring the database back on line?
An incorrect answer would be turn over and go back to sleep. Correct answers might be log onto the server box and read the alert log. Check for activity on the server, etc.

5. What is the biggest error in judgement you have made in your current job? Why did you make it? How did you correct it? What did you learn from it?

If he says he has never made a big error in judgment, he is lying.

6. When given a special project to work on (patch all the dbs with a critical patch for example), how do you approach the project.

A reverse planning method would be appropriate.

7. As an on-call DBA what are the most common issues you have had?

A full archive log destination is pretty common. Failed backup scripts also.

8. Describe a time when you came up with a creative solution to a database performance problem.

9. A developer comes to you with a complex query that does not perform well. What steps do you take to help her optimize this query.

Get an explain plan, look at the indexes and use of wildcards. There are a lot of good answers for this.

10. An application in production that has always performed well in the past has suddenly started timing out. During your research, you discover that a query that has performed well in the past has suddenly started running slowly. What steps do you take to resolve this problem?

See if the explain plan has changed. See if stats are good on the table. Check for invalid indexes.

11. You are observing the database, and discover that there are several ad-hoc queries being run through sql-navigator in the production database. There is no rule against this, but you suspect that the number and size of the queries is slowing down the response time in production. What do you do?

This is really a trick question. The answer you are looking for is talk to the people who are running the queries, or their supervisors, explain the problem, and see if they can reduce or eliminate the activity. Diplomatically.

Nice Trick for Reading Text Files Into Oracle

Have you ever had the need to read a text file into Oracle? And parse out pieces of information when you do it? Yes, you can do it with SQL Loader, but Oracle external tables offer much more flexibility. The example below shows you how to load an Oracle alert log into an Oracle table using an external table, but you can use this method with any text file.

First, define your directory and grant the required privileges:

create directory log_dir as ‘/u01/app/oracle/admin/orcl/bdump’;
grant read on directory log_dir to dba;
grant write on directory log_dir to dba;

Next, we create our table:

create table alert_log
(txt_line varchar2(512))
organization external
(type ORACLE_LOADER
default directory log_dir
access parameters (records delimited by newline
fields
(txt_line char(512)))
location (‘alert_orcl.log’)
);

Note that we set the length of the line to 512. Just set it to the maximum length of the line. Oracle will read to the end of the record as delimited by the new line.

At this point, we can read the table as any oracle table.

Select * from alert_log;

Create table ora_alert_log as (select rownum as line_number, txt_line from alert_log);

If you run the create table as select, you now have a standard Oracle table containing lines of text. At this point, you can do all kinds of queries and data manipulations. You can search for shutdown and startup times, look for Oracle error messages, or anything else you may need. This external table feature is very handy, and there is a lot you can do with it.

Don Burleson recently sent out to one of his Oracle DBA mailing lists some information about rebuilding indices (or is it indexes?) to his Oracle tips mailing list. He also provided a link to more information on his web site: http://www.dba-oracle.com/t_index_rebuilding_issues.htm. His information is valuable, and I have a few more observations from my experience at my previous job.

Burleson notes that Oracle recommends rebuilding indexes when the b-level is more than four or the percent deleted is greater than 20.

At my previous employer, the hardware budget was always very tight. We couldn’t always lay out the tablespaces in the ideal manner to optimize performance, but I think we did a pretty good job within the constraints we had. We rarely hit a b-level higher than three, but we discovered that in Oracle Versions 8i, 9i, and 10g, the larger the table the more noticeable the performance degradation on index reads as the percentage of deletes goes up. That is, on larger tables the performance degradation was much quicker and more noticeable than on smaller tables. A large table would see noticeable performance degradation on as few as 5% delete, whereas on smaller tables it would need to hit 10-15% before the degradation became noticeable. These were not minor differences, measured in milliseconds. The performance changes were dramatic, highly noticeable to end users. For the purposes of this article, a smaller table is from one hundred thousand to one million rows, a medium size table could be up to five million rows, and a large table is more than five million rows.

Obviously this sort of statistic is highly dependent on hardware, but I suspect there are plenty of Oracle shops that have seen similar results.

I eventually wrote a simple query that generates a list of candidates for index rebuilds, and the commands necessary to rebuild the indexes once the tables reached a point where it was necessary. The query reads a table we built called TABLE_MODIFICATIONS that we loaded each night from DBA_TAB_MODIFICATIONS before we ran statistics. Monitoring must be turned on to use the DBA_TAB_MODIFICATIONS table.

select
‘exec analyzedb.reorg_a_table4(‘||””||rtrim(t.table_owner)||””||’,'||””||
rtrim(t.table_name)||””||’);’,
t.table_owner||’.'||t.table_name name,
a.num_rows,
sum(t.inserts) ins,
sum(t.updates) upd,
sum(t.deletes) del,
sum(t.updates)+sum(t.inserts)+sum(t.deletes) tot_chgs,
to_char((sum(t.deletes)/(decode(a.num_rows,0,1,a.num_rows)))*100.0,’999999.99′) per_del,
round(((sum(t.updates)+sum(t.inserts)+sum(t.deletes))/(decode(a.num_rows,0,1,a.num_rows)) *100.0),2) per_chg
from analyzedb.table_modifications t,
all_tables a
where t.timestamp >= to_date(‘&from_date’,'dd-mon-yyyy’) and
t.table_owner = a.owner and t.table_owner not in (‘SYS’,'SYSTEM’) and
t.table_name=a.table_name
having (sum(t.deletes)/(decode(a.num_rows,0,1,a.num_rows)))*100.0 >=5
group by t.table_owner, t.table_name, a.num_rows
order by num_rows desc, t.table_owner, t.table_name;

Of course, your mileage will vary. Reorg_a_table4 was the stored procedure that actually generated the DDL. It just reads the data dictionary to get the index name and tablespace name and generates the alter index rebuild command.

This was a site with a traditional one night per week operations window, and we usually didn’t have enough time to rebuild everything, so the list was sorted so that those tables most in need of work were listed first.