CHECKPOINT TUNING

This information is garnered DIRECTLY from Oracle Documentation. I did not write this and I am not claiming credit for it.

CHECKPOINT TUNING

What is a Checkpoint?

A Checkpoint is a database event which synchronizes the modified data blocks in memory with the datafiles on disk. A checkpoint has two purposes: (1) to establish data consistency, and (2) enable faster database recovery. The checkpoint must ensure that all the modified buffers in the cache are really written to the corresponding datafiles to avoid the loss of data which may occur with a crash (instance or disk failure).

Checkpoints and Performance

Checkpoints present a tuning dilemma for the Database Administrator.  Frequent
checkpoints will enable faster recovery, but can cause performance
degradation. How then should the DBA address this? Depending on the number of datafiles in a database, a checkpoint can be a
highly resource intensive operation, since all datafile headers are frozen
during the checkpoint.  There is a performance trade-off regarding frequency
of checkpoints.  More frequent checkpoints enable faster database recovery
after a crash.  This is why some customer sites which have a very low
tolerance for unscheduled system downtime will often choose this option.
However, the performance degradation of frequent checkpoints may not justify
this philosophy in many cases. Let's assume the database is up and running 95%
of the time, and unavailable 5% of the time from infrequent instance crashes
or hardware failures requiring database recovery.  For most customer sites, it
makes more sense to tune for the 95% case rather than the rare 5% downtime.
This bulletin assumes that performance is your number one priority and so
recommendations are made accordingly. Therefore, your goal is to minimize the frequency
of checkpoints through tuning.

Tuning checkpoints involves four key initialization parameters
           -  FAST_START_MTTR_TARGET
           -  LOG_CHECKPOINT_INTERVAL
           -  LOG_CHECKPOINT_TIMEOUT
           -  LOG_CHECKPOINTS_TO_ALERT

Note: Log file switches will always override checkpoints caused by following paarameters.

• FAST_START_MTTR_TARGET

Since Oracle 9i FAST_START_MTTR_TARGET parameter is the preferred method
of tuning incremental checkpoint target. FAST_START_MTTR_TARGET enables you
to specify the number of seconds the database takes to perform crash recovery
of a single instance. Based on internal statistics, incremental checkpoint
automatically adjusts the checkpoint target to meet the requirement of
FAST_START_MTTR_TARGET.
V$INSTANCE_RECOVERY.ESTIMATED_MTTR shows the current estimated mean time to
recover (MTTR) in seconds. This value is shown even if FAST_START_MTTR_TARGET
is not specified.
V$INSTANCE_RECOVERY.TARGET_MTTR shows the effective MTTR target in seconds
enforced by the system.
V$MTTR_TARGET_ADVICE shows the number of I/Os resulted by the current workload
under the current MTTR setting and the estimated number of I/Os that would be
resulted by the current workload under other MTTR settings. This view helps
the user to assess the trade-off between runtime performance and setting
FAST_START_MTTR_TARGET to achieve better recovery time.

• LOG_CHECKPOINT_INTERVAL

LOG_CHECKPOINT_INTERVAL parameter specifies the maximum number of redo blocks
the incremental checkpoint target should lag the current log tail.
If FAST_START_MTTR_TARGET is specified, LOG_CHECKPOINT_INTERVAL should not
be set or set to 0.
On most Unix systems the operating system block size is 512 bytes.
This means that setting LOG_CHECKPOINT_INTERVAL to a value of 10,000 would
mean the incremental checkpoint target should not lag the current log tail
by more than 5,120,000 (5M) bytes. .  If the size of your redo log is 20M, you are taking 4
checkpoints for each log. LOG_CHECKPOINT_INTERVAL influences when a checkpoint occurs, which means
careful attention should be given to the setting of this parameter, keeping it
updated as the size of the redo log files is changed.  The checkpoint
frequency is one of the factors which impacts the time required for the
database to recover from an unexpected failure.  Longer intervals between
checkpoints mean that if the system crashes, more time will be needed for the
database to recover.  Shorter checkpoint intervals mean that the database will
recover more quickly, at the expense of increased resource utilization during
the checkpoint operation.
This parameter also impacts the time required to complete a database recovery
operation during the roll forward phase of recovery.  The actual recovery time
is dependent upon this time, and other factors, such as the type of failure
(instance or system crash, media failure, etc.), and the number of archived
redo logs which need to be applied.

• LOG_CHECKPOINT_TIMEOUT

The LOG_CHECKPOINT_TIMEOUT parameter specifies the maximum number of seconds
the incremental checkpoint target should lag the current log tail.
In another word, it specifies how long a dirty buffer in buffer cache can
remain dirty.

Checkpoint frequency impacts the time required for the
database to recover from an unexpected failure.  Longer intervals between
checkpoints mean that more time will be required during database recovery. Oracle recommends using LOG_CHECKPOINT_INTERVAL to control the checkpoint
interval rather than LOG_CHECKPOINT_TIMEOUT, which will initiate a checkpoint
every "n" seconds, regardless of the transaction frequency.  This can cause
unnecessary checkpoints in cases where transaction volumes vary.  Unnecessary
checkpoints must be avoided whenever possible for optimal performance.
It is a misconception that setting LOG_CHECKPOINT_TIMEOUT to a given
value will initiate a log switch at that interval, enabling a recovery
window used for a stand-by database configuration.   Log switches cause a checkpoint,but a checkpoint does not cause a log switch.  The only way to cause a log switch is manually with
ALTER SYSTEM SWITCH LOGFILE or resizing the redo logs to cause
more frequent switches.  This is controlled by operating system
blocks, not a timed interval.
Sizing of the online redo logs is critical for performance and recovery.
See additional sections below on redo logs and checkpoints.

• LOG_CHECKPOINTS_TO_ALERT

LOG_CHECKPOINTS_TO_ALERT lets you log your checkpoints to the alert file.
Doing so is useful for determining whether checkpoints are occurring at
the desired frequency.
Prior to Oracle9i this parameter was STATIC.
Oracle generally advises this be set to TRUE as the overhead is
negligible but the information in the alert log may be useful.

See Note:76713.1 to have more detail on How those instance parameters can influence the checkpoint.

Redo logs and Checkpoint

A checkpoint occurs at every log switch.  If a previous checkpoint is already
in progress, the checkpoint forced by the log switch will override the current
checkpoint. 

This necessitates well-sized redo logs to avoid unnecessary checkpoints as a
result of frequent log switches. 

The lag between the incremental checkpoint target and the log tail is
also limited by 90% of the smallest online log file size. This makes sure
that in most cases log switch would not need to wait for checkpoint.
Because of this, log file sizes should be configured large enough.
A good rule of thumb is to switch logs at most every twenty minutes.
Having your log files too small can increase checkpoint activity and reduce performance.
Oracle recommends the user to set all online log files to be the same size,
and have at least two log groups per thread.   The alert log is a valuabletool for
monitoring the rate that log switches occur, and subsequently, checkpoints
occur.
 
The following is an example of quick log switches
from the alert log:
 
Fri May 16 17:15:43 1997
Thread 1 advanced to log sequence 1272
  Current log# 3 seq# 1272 mem# 0: /prod1/oradata/logs/redologs03.log
Thread 1 advanced to log sequence 1273
  Current log# 1 seq# 1273 mem# 0: /prod1/oradata/logs/redologs01.log
Fri May 16 17:17:25 1997
Thread 1 advanced to log sequence 1274
  Current log# 2 seq# 1274 mem# 0: /prod1/oradata/logs/redologs02.log
Thread 1 advanced to log sequence 1275
  Current log# 3 seq# 1275 mem# 0: /prod1/oradata/logs/redologs03.log
Fri May 16 17:20:51 1997
Thread 1 advanced to log sequence 1276
  Current log# 1 seq# 1276 mem# 0: /prod1/oradata/logs/redologs01.log
 
If redo logs switch every 3 minutes, you will see performance degradation.
This indicates the redo logs are not sized large enough to efficiently handle
the transaction load.
See Note:1038851.6 for further detail on how to estimate an adequate size of the redolog files. See Note:1035935.6 Example of How To Resize the Online Redo Logfiles

Understanding Checkpoint Error messages  (Cannot allocate new log and Checkpoint not complete

Sometimes, you can see in your alert.log file, the following corresponding
messages:   Thread 1 advanced to log sequence 248
    Current log# 2 seq# 248 mem# 0: /prod1/oradata/logs/redologs02.log
  Thread 1 cannot allocate new log, sequence 249
  Checkpoint not complete
 
This message indicates that Oracle wants to reuse a redo log file, but
the current checkpoint position is still in that log. In this case, Oracle must
wait until the checkpoint position passes that log. Because the
incremental checkpoint target never lags the current log tail by more than 90%
of the smallest log file size, this situation may be encountered if DBWR writes
too slowly, or if a log switch happens before the log is completely full,
or if log file sizes are too small.
When the database waits on checkpoints,redo generation is stopped until the
log switch is done.

Using Statspack to determine Checkpointing problems

Statspack snapshots can be taken every 15 minutes or so, these reports gather useful
information about number of checkpoints started and checkpoints completed and number
of database buffers written during checkpointing for that window of time . It also contains
statistics about redo activity. Gathering and comparing these snapshot reports gives you
a complete idea about checkpointing performance at different periods of time.

Another important thing to watch in statspack report is the following wait events,
they could be a good indication about problems with the redo log throughput and checkpointing:

log file switch (checkpoint incomplete)
log file switch (archiving needed)
log file switch/archive
log file switch (clearing log file)
log file switch completion
log switch/archive
log file sync

In the case when one or more of the above wait events is repeated frequently
with considerable values then you need to take an action like adding More
online redo log files or increasing their sizes and/or modifying checkpointing parameters.

Optimizer Index Cost

After a year of being the sole DBA here I learned something incredibly useful that I wish I had understood long ago. Almost all of my posts that involve me fighting performance issues are resolved by this one little parameter.

*.optimizer_index_cost_adj=50

Many years ago when I first started exploring parameters I was told not to mess with this one because it was already setup exactly as needed. Many of our parameters are like that. Set them once and deploy them like that 100x without any problems. This little bastard though...

What does this do? It's a parameter that controls the optimizer's preference of Index vs Full Table Scan. It has a range of 1-10000 and defaults to 100. Apparently setting it to 100 causes the optimizer to think that both explain plans would have equal fetch times... I'm not sure which engineer decided that should be the default but I am sure that he was hitting himself in the head with a hammer when he considered it a good idea.

We have 1-10 indexes for each data set that we import and nearly an infinite number of data sets. After setting this parameter to 50 my optimizer started using the indexes exclusively and now my longest running SQL isn't even considered problematic anymore. Yipee!

Apparently 1-10,000 is the 'cost' differential. So when the optimizer looks at explain plan of FTS vs Index they each have a cost represented as a numeric value 0-10000. This parameter is a baseline cost. The previous DBA here used to call this 'Oracle Dollars'. Actually he was Asian so it was more like 'Oracle Doroos'. In any case I finally understand what he meant by that as each SQL in the database that gets ran through optimizer has this 'cost' associated with it.

Playing with the optimizer is not something I'm very familiar with but that's my understanding as of right now :)

It's the Oracle Cost Based Optimizer that applies to any SQL running in the database. The hand tuning would be to rewrite the SQL to duplicate the most effective explain plan. I've never heard of disabling the CBO but I'm sure it can be done.

Database Upgrade

When major patch sets or database releases are put into production we completely ignore them. The only time we consider patches or new releases important is when there is a direct need for them. For instance I have a system with a corrupt tablespace. The fix for this specific issue was to upgrade the database to the next major revision. No big deal.

We have to perform silent installations where I work because we only have SSH access to the systems. Please note that we're running on Linux (always). So here are the steps involved with a silent installation and the cause and fix for the following errors:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option

Set up your environment ORACLE_SID, ORACLE_HOME, ORACLE_BASE, the usual shit. You know what, if you don't have this part figured out by now then don't bother with the upgrade. Next!

Next you unzip your patch file and jump into the Disk1/Install/response directory. You should see a single (or multiple) response files. Pick the one that sounds pertinent and edit it. This part should be pretty straight forward. Note that if you have an existing name for your Oracle Home that they must match. If you don't know what it is then it will tell you when it's ran (It errors out with: Oracle home name XXXX already exists at the Oracle Home you specified). Why they did not automate this I have no clue. Once you're satisfied with all of the entries move on.

Now we need to shut down EVERYTHING that's related to that home. dbconsole, CRS, ASM, Listener and RDBMS. Details.

$ORACLE_HOME/bin/emctl stop dbconsole


sqlplus "/ as sysdba"
shutdown immediate

ORACLE_SID=+ASM
sqlplus "/ as sysASM"
shutdown immediate


$ORACLE_HOME/bin/crsctl stop crs (as ROOT)


LSNRCTL stop

Now everything is shutdown and you have your patch unzipped and the response file ready. Change your directory to the runInstall script and kick it off silently:

./runInstaller -silent -responseFile /u01/Disk1/response/patchset.rsp

Once that completes you ARE NOT FINISHED. I made the mistake of thinking that I was. There are still 2 very important steps. If you attempt to start your database you get bootstrap and forced disconnection errors. The traces generated say that you have to open the database in UPGRADE mode. What they don't mention is that you skipped a step.

Start everything up (dbconsole, CRS, ASM, Listener and RDBMS) and then:

sqlplus "/ as sysdba"
startup UPGRADE
@?/rdbms/admin/catupgrd.sql
shutdown immediate
startup
@?/rdbms/admin/utlrp.sql

And voila! You are finished. Well done you crazy DBA you! I only say that because reading how to upgrade a database from me is terrifying. Go flog yourself.

Oracle Enterprise Manager

I deploy my databases without Enterprise Manager installed. To create a cookie cutter quick and easy tarball deployment with a configured EM/dbconsole would take me far longer than it's worth. So instead I've streamlined the steps involved with uninstalling and reinstalling EM/dbconsole. I have only tested the following steps on 10/11g.

Let's drop the preliminary repository drop first:

sqlplus "/ as sysdba"
exec DBMS_AQADM.DROP_QUEUE_TABLE(queue_table=>'SYSMAN.MGMT_NOTIFY_QTABLE',force=>TRUE);
alter user sys identified by oracle;
alter user system identified by oracle;
drop user sysman cascade;
drop role MGMT_USER;
drop user MGMT_VIEW cascade;
drop public synonym MGMT_TARGET_BLACKOUTS;
drop public synonym SETEMVIEWUSERCONTEXT;

Now we go about creating the password file:
sqlplus "/ as sysdba"
alter system set remote_login_passwordfile=exclusive scope=spfile;
!orapwd file=$ORACLE_HOME/dbs/orapw$ORACLE_SID password=oracle entries=5 

Okay, now to do some more dropping:
$ORACLE_HOME/bin/emca -deconfig dbcontrol db
$ORACLE_HOME/bin/emca -deconfig dbcontrol db -repos drop
rm -r $ORACLE_HOME/$HOSTNAME\_nams
rm -r $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_$HOSTNAME\_nams

And finally time to create:
$ORACLE_HOME/bin/emca -config dbcontrol db -repos create 

Now lets start it up!
emctl dbconsole start
check /etc/sendmail.cf for outgoing server.

Linux Kernel Parameters

Linux Kernel Parameters
=================

SHMMAX - kernel parameter controlling
maximum size of one shared memory segment
 

SHMMNI - kernel parameter controlling
maximum number of shared memory segments
in the system
 

SHMSEG - kernel parameter controlling
maximum number of shared memory segments
a process can attach
 

SEMMNS - kernel parameter controlling
maximum number of semphores in the system
 

SEMMNI - kernel parameter controlling
maximum number of semaphore sets.
Semphores in Unix are allocated in sets of
1 to SEMMSL.
 

SEMMSL - kernel parameter controlling
maximum number of semaphores in a semphore set.
 

SHMLBA - kernel parameter controlling
alignment of shared memory segments; all
segments must be attached at multiples of
this value. Typically, non-tunable.

Oracle RMAN Tuning

So I'm still in combat with the RMAN backups. I've increased performance with several methods mentioned in older posts. Now it's time to cover channels and maybe even a little bit about backup sets. First off Oracle defines an RMAN channel as a direct connection between the database and the device backup is being written to. Each channel represents a level of parallelism and can be seen as a unique process on the operating system.

The default number of channels is 1. This means there is a single connection between the database and the disks the backup is writing to. If you are on a single hard drive and you're writing to a tape then is probably okay. On large systems with 16 processors, 16gb of RAM and 32 physical disks we can obviously bump up the number of channels. Remember that this should only be done if you are NOT CPU bound by backups as spawning a server process that is doing intense work per channel can cause your system load to skyrocket if you aren't careful.

I am I/O bound and I'm sure that's the case for many of you out there reading this. It seems like I/O on very large systems is the main offender of most performance issues. There are clever ways around it but none of them are simple or even obvious. I will be doing a great deal of channel testing over the next couple of days.

For those of you truly interested there is a lot of math involved in getting everything correct. Here is a bit of documentation: RMAN Documentation

I set my channels to 10 by setting the RMAN parameter:

RMAN> CONFIGURE DEVICE TYPE DISK PARALLELISM 10 BACKUP TYPE TO BACKUPSET;
old RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters:
CONFIGURE DEVICE TYPE DISK PARALLELISM 10 BACKUP TYPE TO BACKUPSET;
new RMAN configuration parameters are successfully stored   

In my case I have 477 datafiles and a parallelism of 10. Rounded up that is 48 files per backup set. RMAN compares 48 to 64 and chooses 48 as the filesperset parameter. 477/48=10 (rounded up). So I'll have 10 backup sets each containing 48 files per set and hopefully producing a single piece. You can also force the basic and advanced compression algorithms by changing the above parameters so be careful. I'll post more as I learn it.

So now it looks a little something like this:

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name #### are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 4 DAYS;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/backup/rman/%F';
CONFIGURE DEVICE TYPE DISK PARALLELISM 10 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BZIP2'; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/111/dbs/snapcf_nams.f'; # default

NOTE: This caused my full level 0 backup to take 2-3 hours instead of 12.

Block Change Tracking (RMAN)

As we all know I combat RMAN regularly. It's a vicious cycle of never ending hate and confusing parameters. Well, I've discovered a way to speed up the backups. Specifically incremental RMAN backups.

First off you need to know how your IO is. As root use iostat to see your reads versus your writes and how high the numbers are for your backup partitions. One thing that you should notice is that your reads are drastically higher than your writes. This is due to Oracle scanning every block of the database in an attempt to detect changes and then write them to backups. You can completely eliminate this preliminary read by implementing a block change tracking file. It's very easy and quick.

Now we're at the point of altering the database which will force it to create a binary change file and update block changes to it:

SQL> alter database enable block change tracking using file '/u01/oradata/nams/change_tracking.f';
SQL> alter database enable block change tracking using file '/u01/oradata/nams/change_tracking.f' resuse;

Alternately if you have DB_CREATE_FILE_DEST parameter set for Oracle Managed Files then you just issue an enable command:

SQL> alter database enable block change tracking;

In this case Oracle will create the change file in whatever directory you specified Oracle Managed Files (OMF) to read/write to. If DB_CREATE_FILE_DEST parameter is not set you will receive an error saying that you must specify the file name.

Now that you have enabled block change tracking it's time to make sure that it is working as intended. Information about the block change tracking file is kept in a view called: v$block_change_tracking. This view always contains a single record. If the status is ENABLED then the FILENAME contains the name of the file being used for tracking and BYTES contains the size of the file (in bytes obviously).

SQL> desc v$block_change_tracking
 
 Name                  Type
 ------------------ ---------------------
 STATUS              VARCHAR2(10)
 FILENAME        VARCHAR2(513)
 BYTES                NUMBER

SQL> select status,filename from v$block_change_tracking;
 
STATUS           FILENAME
---------------- --------------------------------------------
ENABLED         /u01/oradata/nams/change_tracking.f    

The v$backup_datafile view contains a column named used_change_tracking. A value of yes in this column means that Oracle RMAN used block change tracking to speed up incremental backups. This will help identify how effective block change tracking was in improving backup duration and I/O.

Oracle Support

As anyone that has ever had the pleasure of dealing with Oracle Support can attest their services leave a lot to be desired. First off it's impossible to talk to someone right away. Service Orders must be opened online. Over that if you select the incorrect department or have a question that oversteps their knowledge by the merest fraction then a new SR is created in a different department and you have to start all over. Even high severity Service Requests will sometimes go over a week without a response. Well I'm fed up with it and I've discovered some tips and tricks that help navigate the maze of endless Habib Jar Hakashar al Hasheed's from the east.

• First off and one of the most important factors in quick problem resolution is to use OCM to collect your hardware/software information. This gathers all sorts of info about the environment, what is installed and how it is used. Tickets submitted with an OCM configuration are resolved much faster and are given priority handling.

• You also need to call when you aren't happy with response times. A lot of time (75% or more) you are assigned to an engineer that is on a different continent. That's all fine and dandy until you need more than one update every 24 hours. Especially when most updates are asking you to perform redundant tasks or to gather information. Just call and ask for an engineer in your time zone.

• If response time does not improve ask for manager oversight. This causes some lazy bastard to get a swift kick in the ass from his manager. It doesn't always work and if it doesn't then you escalate to Severity 1 if it isn't already. Manager oversight also means that the manager SHOULD oversee the progress on the ticket rather than just ask for an update from his engineer.

• If the SR your working with isn't exactly pertinent to the group handling it then you will get passed off. This happens A LOT if you have any questions at all or if you don't select department and descriptions perfectly. When this happens make sure to ask the engineer to perform a WARM HANDOFF of the case and to link the 2 SRs to each other. This will save the new engineer all of the trouble of asking you 100 questions that you have already answered to a different Habib.

Besides that everything else should be self explanatory. Sometimes getting angry helps while most of the time it's pointless. Remember, if you're truly pissed then call your sales manager and let them know what's going on. You can also keep escalating the SR until you reach an Oracle Master/Professional that can usually make a system dance within a very short window of time. I've only had to do this once and the support I received from the Oracle Master was nothing short of amazing. So don't get frustrated when it seems like progress has stalled, instead get clever and start manipulating their own system.

Moving and/or Renaming Datafiles

I have abandoned ASM for all systems with the exception of RAC environments. Because of this I occasionally have a system analyst name datafiles incorrectly when adding datafiles to a tablespace or even more urgent is when a operating system partition is nearly full.

You cannot simply move the datafiles at the OS level. Instead you have to move/rename them at both the OS and database levels. This is very simple if you're not using ASM. Just follow the 5 easy steps below:

1. The first thing you do is shut down the database:

sqlplus "/ as sysdba"
SQL> shutdown immediate


2. Then you move the files at the OS level:
# mv /u01/oradata/$SID/datafilename.dbf /u02/oradata/$SID/datafilename.dbf


3. Now we mount the database:
sqlplus "/ as sysdba"
startup mount


4. Now we update the data dictionary. Make sure that you use full paths:
alter database rename file '/u01/oradata/$SID/datafilename.dbf ' to '/u02/oradata/$SID/datafilename.dbf';

5. Finally we open the database:
sqlplus "/ as sysdba"
alter database open

That's all there is to it. ASM can do similar things but it's much less needed due to the flexibility and power of ASM. Too bad that comes at the cost of complexity. I prefer to have user managed storage for several reasons. The most important of which is quick and easy database deployment. Simply create a tar archive of your database and you can re-deploy it anywhere very easily.

Gaming Keyboard G510 Review

The Logitech Gaming Keyboard G510 is fun to play with. Like the Cooler Master's mouse that I reviewed earlier in the year it has a very sleek and 'cool' design that is eye catching. Standard features include:

• Dynamic color adjustment of key backlighting
• A slew of extra macro keys
• A volume wheel and controls
• Programmable (software included)
• LED Screen
• Windows key lock
• Wrist rest
• USB
• Green

It's important to keep in mind that to use this keyboard to the full extent you have to have a bit of knowledge with macros and modifier keys (alt+key, ctrl+key, ctrl+alt+key, etc). The software is confusing once you start to dig into it. The most popular features such as back lighting and key assignments are straight forward but anything beyond that requires you to spend some time with it. It also requires updates to the software after you download and install it. Shouldn't we just download the latest release in the first place? Logitech...

My favorite feature by far is the LED screen. It is very usable rather than a gimmick. There are custom modules for all of the popular video games and several downloadable modules for use with it. Let's not forget to mention you can write your own. After using the LCD screen on here I feel like I'm missing something if it's not there.

The macro recorder seemed a bit buggy and difficult to get just right. At one point I thought I had it doing exactly what I wanted but when I tried to execute the macro it just simply did not have the intended effect. So instead of turning all of my shiny new keys into complex macros that may or not function exactly how I wanted I just set them all up with their modifiers. The 18 keys on the side are each set with complex modifiers that allow me to bind them in-game. In World of Warcraft it worked great, in any FPS it was terrible. Most fast paced video games don't expect you to be using a lot of modifiers that may not be recognized/saveable.

My biggest complaint by far is the volume control wheel at the top left. The buttons and the wheel all function flawlessly. The actual problem comes into play when you notice that your volume keeps reseting to default. I play with my Windows volume maxed out (100/100) and then I adjust my actual speaker volume dial as needed. This ensures that I'm not missing sounds/noises/voiceovers in my videogames because of Windows. It is rather frustrating to notice halfway through a Call of Duty match that you haven't heard a call out in a while.

The windows key lock was completely worthless for me as I use the windows key a LOT (Try using Windows Key + TAB if you run Vista or 7). Both in and out of game I find myself using the shortcuts enough to completely ignore this feature.
One of the nice things was profile switching on the fly using the M1,M2 and M3 buttons above the macro keys. I would setup colors, macros and custom functions for each profile. M1=MMO, M2=FPS, M3=Standard. This was actually a very welcome feature as I use the keyboard differently for each game type and I assume others do as well.


Conclussion:
If you see this go on sale and you want something that looks really nice then by all means grab it. I would strongly recommend to purchase it for specific features rather than just because it's a gaming keyboard. If you don't need macro keys and you could care less about an LCD screen then go get a different keyboard for your "Ohhhhh"s and "Ahhhhh"s.

I'm considering nominating this for the turd in a box award.

EDIT: I would like to add that I returned this keyboard under warranty and Logitech's return procedures are like most, horrendous. I think it took something like 2-3 weeks to get the replacement. They also temporarily charge a credit card until you return the old one. If I remember correctly shipping and handling had to paid too.

INACTIVE Sessions

Two posts in one day, aren't you lucky! I've got something for you people hitting ORA-4030 errors to try. We all know 4030 is caused by 100 different things and it takes some time as well as probably trial and error to fully resolve them. It can be anything from SGA Sizing, Kernel Parameters, User Limits and today I introduce Inactive Sessions to the list of known causes.

Let's see how much memory INACTIVE sessions are taking:
SQL> select sum(PGA_ALLOC_MEM) from v$process p, v$session s where p.addr = s.paddr and s.status = 'INACTIVE';

SUM(PGA_ALLOC_MEM)
------------------
            598802 

These little bastards can really cause a headache after a while. Here's a little information about why it's important that I find and destroy these lazy turds. Each session in the database takes up memory. This is usually no big deal but if you aren't closing out sessions it will eventually become one. Most sessions will do a clean close and there's no issue. The problem for us comes into play when our end users run a report and either exit out of the application or it just simply never finishes. I've always been told that bouncing the report server will kill these spinning reports in the database: WRONG!

First off we have to find them by querying the v$session view:

SQL> set linesize 150

SQL> set pagesize 10000

SQL> select SID, SERIAL#, USERNAME, PROCESS, LOGON_TIME from v$session where status = 'INACTIVE' order by LOGON_TIME;

       SID    SERIAL# USERNAME          PROCESS      LOGON_TIM
---------- ---------- ------------------------------ ------------ ---------
       867      40145 MRNAMS                        18617        27-JUN-11
       617      36394 MRNAMS                        29566        25-JUL-11
       714      50687 RMAXWELL                   20605        25-JUL-11
       727      59474 MRNAMS                         5540         25-JUL-11
       922       4096  MRNAMS                         5542         25-JUL-11
      1046     14840 NOC                                   4116         25-JUL-11
       938      58646 MTOMA                          11678        25-JUL-11


MRNAMS is the name of our administrative user. Most of the MRNAMS sessions that are inactive are legitimate. Then we see our end user accounts (RMAXWELL, NOC, MTOMA). These accounts are running reports from our application. While none of these are having problems we will sometimes see one of these sessions stay inactive indefinitely.

We discovered this by correlating these sessions to the OS Process ID. A simple grep of the process shows us this:
ps -ef | grep 11678
oraoas   11678  2725  0 06:39 ?        00:00:00 f60webm webfile=5,33,PID2725

The process f60webm is our forms server. Either MTOMA is running a report or he is logged into the application and has been idle. To resolve this we simply kill the long running sessions manually and then enact a profile resource limit for inactive time on the owning profile. Let's just pretend that MTOMA is a long running session:

Syntax:
ALTER SYSTEM KILL SESSION 'sid,serial#';

ALTER SYSTEM KILL SESSION '938,58646';

Syntax:
PROFILE_NAME limit RESOURCE ####;

SYSTEM_ADMINISTRATOR limit IDLE_TIME 2160;

There ya go, now you have identified all of the inactive sessions, killed the pertinent ones and enabled a limit within the profile that will kill all FUTURE inactive sessions (that's why we had to manually kill the existing ones, right?) once they have been running for 36 hours.

Weblogic and Portal, Forms & Reports

As a Senior Oracle DBA you're pretty much tasked with anything that involves the word Oracle or that deals with the databases. As an Oracle house the company I work for has been using Oracle Developer's 6i (no, you didn't misread that) for the last 10 years (that either).

So now it's my job to upgrade 50+ systems from 6i to 11g. Suffice to say this is not 'just another upgrade' or patch. This is a genuine bitch to get working and I'm determined to make it work. The biggest problem I'm facing right now is that Oracle no longer uses Apache as their native http server. Instead they use Weblogic (which is also a prerequisite for the new forms and reports. Now on top of that I can only do silent installations with the universal installer using a response file because I do not have desktop access.

UPDATE: Wow, you cannot imagine how difficult this is. Weblogic is installing 32bit JDK/JRE no matter how I install it. I got with Oracle Support and tried installing it using a 64bit Java '.jar' file and it still installs the 32bit versions. Usually this wouldn't be a big deal but Forms and Reports forces you to have matching bit lengths between Web Logic, Forms&Reports, Java and the Operating System (Which means hardware as well to and extent.). Le sigh

Let the fun begin!

JDK1.6/OCM Installation Script

I completely forgot about this blog. Eat that non-existent readers! Anyways my Oracle knowledge has grown exponentially since my last update. I'm now much more knowledgeable about kernel parameters (fighting those pesky 4030 bugs) as well as Oracle's configuration manager (OCM). I actually wrote a script that installs JAVA and OCM on the systems. I then called it a patch and asked all of the analyst to apply it to the 50+ databases out there. I don't know about you but I thought it was a damn clever way to hide my laziness.


File 1 (This installs Java Development Kit 1.6.blah.blah then passes the rest of the work off to file2)


#!/bin/ksh


#
# Verify that this script is being executed as root.
#
if [ -z $LOGNAME ]
then
   echo "The environmental variable LOGNAME is undefined."
   exit 1
fi


if [ $LOGNAME != "root" ]
then
   echo "This install script must be executed as root!"
   exit 1
fi



#
# Create Java directory if it does not exist
#
if [ ! -d "/usr/java" ]; then
        mkdir /usr/java
fi



#
# Install Java BIN
#
cp jdk-6u21-linux-i586.bin /usr/java/jdk-6u21-linux-i586.bin
cd /usr/java
./jdk-6u21-linux-i586.bin
cd -


if [ -e /usr/bin/java ] ; then
        /bin/rm /usr/bin/java
fi


ln -s /usr/java/jdk1.6.0_21/bin/java /usr/bin



#
# Change ORACLE profile
#
sed -i 's/JAVA_HOME=\/usr\/local\/java/JAVA_HOME=\/usr\/java\/jdk1.6.0_21/g' /home/oracle/.bash_profile 2>/dev/null
sed -i 's/JAVA_HOME=\/usr\/java/JAVA_HOME=\/usr\/java\/jdk1.6.0_21/g' /home/oracle/.profile 2>/dev/null


#
# Copy and execute second installation file
#
cp *.zip /tmp
cp install2 /tmp
cp v4.0.66.sql /tmp
cp passchange.sql /tmp
chown oracle:dba /tmp/*.zip
chown oracle:dba /tmp/install2
su - oracle -c "/tmp/install2"


#
sqlplus standarduser/hispassword@nams-cs @v4.0.66.sql
#  Provide patch installation feedback and exit.
#
echo "Patch v4.0.66 has been installed."
echo "Please check all patch logs for errors."
exit 0                           



FILE 2 (This actually installs, collects and uploads OCM and it's reports)


#!/bin/sh
env | grep SHELL | grep ksh > /dev/null 2>&1
if [ "$?" -eq "0" ]; then
        . /home/oracle/.profile
else
        . /home/oracle/.bash_profile
fi


#
# Change SYS password to ORACLE
#
sqlplus "/ as sysdba" @/tmp/passchange.sql


#
# Unzip OCM installation files
#
cd $ORACLE_HOME
unzip /tmp/p5567658_1035_LINUX.zip

#
# Install OCM
#
export ORACLE_SID
export ORACLE_HOME
cd $ORACLE_HOME/ccr/bin
./setupCCR -s ######## myemailaddress@theplaceIwork.net US


#
# Collect and send OCM config
#
cd $ORACLE_HOME/ccr/admin/scripts
./installCCRSQL.sh collectconfig -s $ORACLE_SID -r SYS 

I hate Oracle

That is all.

I/O Balancing

Ok. So these motherfucking redo logs are seriously a pain in the ass. It turns out that I'm experiencing way more I/O than the typical system should. Even after resizing my redo logs to 1gb they're still giving us a ton of shit. The reason for this is because I was not load balancing my I/O. This was discovered using iostat:

[root@bigbend-nams] /root> iostat -d 2
Linux 2.6.9-42.ELsmp (bigbend-nams)     07/26/11

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               5.49        17.74        63.92  969322392 3491656284
sda1              0.00         0.00         0.00       6787      12806
sda2              0.83         1.48         6.40   80848396  349664082
sda3              2.82         1.81        21.78   98920459 1190003096
sda4              0.00         0.00         0.00          3          0
sda5              0.24         0.24         1.73   13353111   94377744
sda6              0.01         0.02         0.04    1261618    2047496
sda7              2.39         0.03        19.09    1631259 1042572456
sda8              0.12         0.06         0.93    3540603   50659912
sda9              2.51         0.30        20.02   16543683 1093492880
sda10             2.09         3.04        16.34  166199116  892333654
sda11            27.10        10.74        56.22  586920044 3071246558
sdb              28.14        27.15        28.12 1483348051 1536078134
sdb1             16.95        21.76        55.90 1188885604 3053367054
sdb2             28.19         5.36        50.84  292617560 2777443685

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               3.00         0.00        40.00          0         80
sda1              0.00         0.00         0.00          0          0
sda2              0.00         0.00         0.00          0          0
sda3              0.50         0.00         4.00          0          8
sda4              0.00         0.00         0.00          0          0
sda5              0.00         0.00         0.00          0          0
sda6              0.00         0.00         0.00          0          0
sda7              2.50         0.00        20.00          0         40
sda8              0.00         0.00         0.00          0          0
sda9              2.00         0.00        16.00          0         32
sda10             0.00         0.00         0.00          0          0
sda11             0.00         0.00         0.00          0          0
sdb               3.50         0.00        61.00          0        122
sdb1              6.50         0.00        52.00          0        104
sdb2              2.00         0.00         9.00          0         18  

We've rebuilt the logfile groups but we didn't put the members in different OS partitions. Because everything is being written to a single OS pertition/device there is a natural and somewhat obvious bottleneck. So what we have to do is rebuild the logfile groups AGAIN except this time we put each logfile member in a different operating system partition. It sounds complicated but it isn't. 

SQL> alter database add logfile group 1 ('/u02/oradata/nams/log1anams.rdo','/u03/oradata/nams/log1bnams.rdo') size 1g;

Not so bad, eh?

Resizing Online Redo Logs

So it turns out that my issues with the database performing terrible had to do with the redo log size being too small. The way I found out was the alert log was switching logs every 2 seconds or something like that. Also my archive logs were 74mb which is tiny.

So this is what I had to do:

First I had to see all of the logs and their respective sizes:

SQL> SELECT a.group#, a.member, b.bytes
     FROM v$logfile a, v$log b WHERE a.group# = b.group#;

Once I saw the results it was pretty obvious all of the logs were undersized. Now I need to create new logs of appropriate size and make them active. Then I need to drop and re-create the logfile group 1 with appropriate size (including removing the .rdo files at the OS level):

SQL> alter database add logfile group 4 ('/u02/oradata/nams/log4anams.rdo','/u02/oradata/nams/log4bnams.rdo') size 1g;

SQL> alter system switch logfile; (Until 4 was the active logfile group)

SQL> alter database drop logfile GROUP 1;

SQL> !rm /u02/oradata/nams/log1anams.rdo

SQL> !rm /u02/oradata/nams/log1bnams.rdo 

SQL> alter database add logfile group 1 ('/u02/oradata/nams/log1anams.rdo',
     '/u02/oradata/nams/log1bnams.rdo') size 1g;

At this time we should be active on logfile group 4 and logfile group 1 completely rebuilt. So now you can perform log switches until redo group 1 is active. Once you've done that you can drop and re-create any remaining groups very easily using the similar procedures above.

 EDIT: FURTHER I/O BALANCING IS NEEDED, PLEASE REFER TO THE FOLLOWING POST

Why The RMAN Backups Never Finished

It turns out the backups weren't finishing because the SGA was over sized. The asshat who built the system (are you starting to see a trend in my fires?) gave the databases on several systems about a GB more SGA than they should have. Once I got the sga_max and pga_aggregate_target to be under 2.7gb combined the backups finished without error. Just to be safe I also set user limits to unlimited and tweaked the kernel parameters.

The Kernel Parameters that we're most interested in involve shared memory between the operating system and the database.

SHMAX - Kernel parameter controlling the maximum size of one shared memory segment

SHMALL- Kernel parameter sets the total amount of shared memory pages that can be used system wide 

SHMMNI - Kernel parameter controlling maximum number of shared memory segments in the system

Kernel limitations:
# cat /etc/sysctl.conf | grep shm
kernel.shmall = 2097152
kernel.shmmax = 4294967295
kernel.shmmni = 4096   

User limits:
# cat /etc/security/limits.conf | grep oracle
oracle           hard    nofile          65536
oracle           soft    nofile          65536
oracle          hard    stack           65536
oracle          soft    stack           65536
oracle          soft    nproc           unlimited
oracle          hard    nproc           unlimited


Process Global Area:
SQL> show parameter pga

NAME                         TYPE               VALUE
----------------------- ---------------- --------------
pga_aggregate_target    unknown      1073741824

System Global Area:
SQL> show sga

Total System Global Area       1740926976 bytes
Fixed Size                                 1300436 bytes
Variable Size                            1140852780 bytes
Database Buffers                     587202560 bytes
Redo Buffers                           11571200 bytes 


How to change the Kernel parameters:
In RedHat all I did was vi /etc/sysctl.conf and change the 3 kernel parameters listed above. I actually left shmmni alone. I always have and always will. I did change shmmax to the exact limitation of the linux 32bit kernel (4gb). This will allow Oracle to take a maximum of 4gb of memory from the OS (note this is only on 32bit machines, 64bit machines have no limits). I set the shmall to a lower number. If the system needs more it will take as much as needs to up until it meets shmmax.


How to change the SGA parameters:

Then I had to tune the SGA in the database to fit within 2.7gb. I had to change SGA parameters in my spfile and bounce the database to achieve this. To see all of your SGA parameters issue a 'show parameter sga' statement. Note that not all memory parameters should be set unless you want to manually manage the memory allocation (and I know I sure as hell don't).


If you set SGA_TARGET and SGA_MAX_SIZE then the rest of the SGA parameters will be auto tuned by Oracle. So in this case I set them equal: sga_target=1740926976 and sga_max_size=1740926976. Do make sure that you issue a scope=both before bouncing the database (or at the very least scope=spfile). Just for examples sake:

alter system set sga_target=1.7g scope=both;

alter system set sga_max_size=1.7g scope=both;

shutdown immediate

startup
 

Being A DBA Sucks!

You might wonder why being a DBA sucks ass. Well, wonder no longer. Here is a little rant just to satisfy your curiousity:

Being a DBA sucks because when shit hits the fan the first place anyone points their snubby little fingers to is the database. Even after the database is cleared of wrongdoing it's still suspect. I wish people did that to their OS instead of my database. We might be running a much cleaner operation if they did.

Remember that when things go wrong and it IS your fault that there is nobody else to help shoulder that weight. You have to bite the bullet if you can't recover from a backups that were never taken because you didn't check their logs weekly. Not saying I've ever done that...

I think I've faced the worst of the worst in the DBA world and I'm not coming out unscathed. One of my larger databases just got completely fucked. It wasn't my fault. The person who built the damn thing built the ASM disk groups incorrectly.

Now I'm eating shit on a daily basis and getting my job threatened because some motherfucker didn't do his research before Cowboy coding a database that wouldn't be his responsibility in a year or two. In short: FUCK HIM.

/rant

RMAN Backup Fails to Complete

Ok, so the partition script from the last post could have been an absolute nightmare but luckily everything was heavily tested before I let it loose in production. Today I'm working on a different problem. This little bastard of a bug has been staring me in the face for over a week now.

What happens is my scheduled RMAN backup fails halfway through. It gives me a nondescript ORA-4030 bug. Because I've dealt with this particular error before I had an idea of where to start. The first thing I do is set my stack size to unlimited (ulimit -s unlimited) and then I restart the database and try again. Of course it doesn't work. After trying a few other minor things not even worth mentioning I open a ticket with Oracle support...

In their infinite wisdom they ask me to allocate less channels (WAAAAY slower backups), backup less datafiles (meaning another complete backup would never happen again) and set my ulimit for data to unlimited (already done...).

So Oracle basically just asked me to gimp my backup software, take incomplete backups and ignored the current configuration that I reported. I guess I have to go it alone. When I find a solution I'll post it.

UPDATE: SOLUTIONS POSTED
LINK TO SOLUTION  for ORA-4030
LINK TO SOLUTION for RMAN Channels
LINK TO SOLUTION for Speeding up Incremental (Block Change Tracking)

The New Away From Keyboard

Welcome to the new and improved Away From Keyboard. This blog's content is now officially changing focus to Oracle Database Administration.

Okay, so maybe new and improved is the incorrect terminology. I should say, as of this post. Details, details. I will have a much easier time with content updates and management if I write about what my life focus is. Let's be honest, marijuana and video games is pretty much juvenile at this point. I will leave the original content below because I can.

Today I'm writing about another script that I've written. It's a korn shell script but it works equally well in bash. This script will grab all partitions in the database that conform to a specified (via input) day and remove them. I just finished this today and even though it is extremely simple I cannot help but glow with pride.

#
#  Used for dropping specific (site, year and, day) partitions
#

. /home/stufflings/.profile

echo -e "\nPlease enter the site ID:"
read SITE

echo -e "\nPlease enter the year:"
read YEAR

echo -e "\nPlease enter the Julian Day:"
read DAY

PARTITION=Y"$YEAR"_D"$DAY"_S$SITE

/u01/app/oracle/product/instantclient_10_2/sqlplus -s stufflings/ncheese<<EOF
set HEAD off
set FEED off
set SQLP ""
set ECHO off

spool /home/nams/utils/dropit.sql

select 'alter table '||table_name||' drop partition '||partition_name||';' from dba_tab_partitions where partition_name = '$PARTITION';

spool off

/

EOF

exit;

     If this seems complex in any way then please let me explain:

1. I open with notes specifying what the script is for.
2. I then load the user profile 'stufflings'. It is the user that will be using the script.
3. After that I ask for input: Site, year and, day.
4. I then take the 3 separate inputs and put them together, creating the partition_name.
5. I sqlplus in silent mode with -s (very helpful for scripting)
6. I set headers, feedback, SQL prompt and, echo's off
7. I spool a new file (dropit.sql)
8. I then perform a special select statement that allows me to alter the output.
9. I end the spooling and exit

If I wanted to completely automate the process I would kick off the script after ending the spool. Because dropping partitions can be so dangerous I do not set it up this way. This can be easily modified for nearly any purpose as long as you are using partitions that are created using a similar format.

The Future Hangs Within a Balance

I am taking strong consideration of turning this into a non-hobby blog and building ShaggyDad.com into something completely different.

HAMP (Home Affordable Modification Program)

I purchased my home in 2008. I slapped a fat down payment on the loan and got a decent interest rate. My interest rate was locked at %6.125. I was told this is a good rate. Now 2 years down the road I hear about the H.A.M.P. (Home Affordable Modification Program).

After a lot of research it appears as if this is a great program but that my lender (Chase) doesn't play by the rules. There are class action lawsuits springing up everywhere and if you do a Google on the issue you see that 1,000s of Chase customers are extremely pissed.

Now here's the deal. If I can get approved for this then I'll be saving $300 a month because my interest will drop down to 2%. Motherfucking Chase seems to have a problem with approving people though. Also, if they do get approved it seems as if Chase is really shitting on them by reporting late or incomplete payments on their credit statements. I really could use that $300 a month so I'm still considering the application. I have digital access to most of the paperwork required and I don't see any reason not to apply for it other than the risk to my credit during the 'trial' phase.

The caveats of any investment have to be head spinning or nobody would agree to their ludicrous terms and conditions.

Nintendo 3DS

Nintendo stock was at a low a while back. I sent an e-mail out to several people saying they should buy it. Nintendo is always the first to come out with the creative use of technology. They were the first to really pull off using motion controllers. Before that we have had several motion controllers but none that were embraced by the mainstream the way the Wii was. They are now the first to integrate genuine 3D images without glasses.

When the Nintendo Wii was released it was difficult to find. The price was a very gentle $300 for a new console with games and a controller. The controller was motion controlled too, you can't beat that! Analysts said that the unit was sorely under priced and could have sold competitively with a much higher price tag. Now it seems as if they're attempting to better price their next release to market conditions.

I'm not sure why, but nothing about that looks fun.

It has more than investors excited... it has the whole gaming industry buzzing. The possibilities of this are astounding. Think about your XBOX displaying your Heads Up Display (HUD: ammo, health, mana, etc) as a 3D holographic display while you play on a absolutely clutter free 3D enabled LCD screen. Or solving a holographic rubic cube on your Playstation with a movement controller and 3D image interaction as in you actually touch the hologram to make it move. Nintendo just crossed a boundary that gamers and science fiction alike have been dreaming of for a long time.

It's one of the most exciting times I've ever seen in the gaming industry. We're right on the verge of this awesome trifecta of 3D (genuine 3D) high resolution graphics, motion controlled gameplay and years of experience in game production. Remember how during the renaissance there was also an explosion of art. Well be prepared because we're in the beginning of a new renaissance and our art isn't on a canvas anymore. It's now on digitally downloaded videogames and 3D theaters with high resolution screens that wrap around your vision. Exciting time indeed.

How to Avoid AFK in WoW

There are several ways to avoid going away from keyboard. The most popular is to get a macro that does a lot of movement, but macros are limited and only run for so long. One way is to use the built in windows ease of access controls. You can set ease of access up to remember keystrokes and then execute them. If done right you can have windows keep you from going AFK indefinitely. There are add-ons from curse that will help you with this as well (UPDATE: The add-ons are no longer available. )

There aren't a lot of reasons to avoid AFK unless you're worried about mana biscuits disappearing. This used to be helpful in battlegrounds to farm honor but now you get reported by other players for being AFK before the round ends which means you never receive honor. A lot of people use that search to arrive here, I figured giving them what they're looking for would be nice.

So here's the gist for Vista and above. Click start and type in 'Ease'. The first result should be Ease of Access. Click that. From there you're on your own (I know, I'm helpful.).



UPDATE: The built in macro abilities have been removed from Windows. If you want to create a Macro it is now bundled with Windows Software such as Visual Studio. The method I was familiar with requires Visual Studio. There are several free programs out there and several methods to program it yourself. If you use Windows7 Ultimate I believe you can download this free Microsoft auxiliary software: http://support.microsoft.com/kb/237179 . In any case, it's done with Macros regardless of how they are made.

Back from Vacation

It's good to be home. Vacations can seem like a lot of work just in and of themselves. Luckily I did everything right and had an awesome time. I climbed well above the timber line in a 2 door Cavalier. I must say that I'm impressed with how my little 4 banger climbed the mountain.

On the way up there we met up with a band in Chama Pass. While on the search for magic mushrooms our newfound guide invited us for a ride on the magic schoolbus  as their band sought out a new venue even higher in the mountains. I met the original family from Lonesome Dove and also snagged a glass for Kips Taco Shop winter closing in Creed (another historical landmark). Here's a picture of the view on the way to Creed:

I took this picture from a dangerous ledge. Tourist like stupidity: ACTIVATE!

After waking up very drunkenly I snagged some home made biscuits and gravy and then saw a dairy on the way to Del Norte. I stopped and bought several shares of a cow. Then I milked 'my' cow and took raw milk home. If you've never had raw milk I must say: YOU'RE MISSING OUT.

We also went fly fishing and hiking. All in all it was an excellent trip that started out the way it ended: just right. Of course no trip is complete without a little mishap. I dusted my brake pads while coming down Wolf Creek Pass and lost a protective undercarriage shield and wire runner after turning into an embankment. On the way back the entire vehicle shook to high heaven but we luckily hit a tire shop RIGHT before the blowout. Then as we leave Santa Rosa to head back to Texas my navigator (see: retarded hamster) manages to direct me down the wrong highway and we hit 'Amarillo by Morning'...

With all of that said and done it was still an excellent trip filled with good memories and even better buds. Cheers to a great vacation!