Jason L. Froebe: Ramblings of a Geek

Because Sybase has yet to address the GLIBC (LD_POINTER_GUARD) issue with any of their products on Linux, we are forced to find alternative ways of getting Sybase software to run on modern Linux distribution.  In this situation, we will be using the Ubuntu 8.04 Just Enough OS (Ubuntu JEOS) Linux distribution.

After installing Ubuntu 8.04 Server JEOS on the physical or virtual machine, download the Ubuntu 7.04 Server iso image.  Why Ubuntu 7.04?  Ubuntu 7.04 is the last Ubuntu release that has an old enough GLIBC that Sybase software will work with.

% sudo debootstrap feisty /home/sybase file:///media/cdrom0

• Change the home directory of the sybase user to be /dbms/sybase by editing the /etc/passwd file

% sudo cp /etc/resolv.conf /home/sybase/etc/resolv.conf

% sudo cp /etc/hosts /home/sybase/etc/hosts

% sudo cp /etc/passwd /home/sybase/etc/passwd

% sudo cp /etc/shadow /home/sybase/etc/shadow

% sudo `cat /etc/apt/sources.list | sed -e ’s/hardy/feisty/g’ > /home/sybase/etc/apt/sources.list`

% sudo mount –bind /dev /home/sybase/dev

% sudo mount –bind /proc /home/sybase/proc

% sudo mount -t devpts none /home/sybase/dev/pts

% sudo mount –bind /tmp /home/sybase/tmp

% sudo aptitude update

% sudo aptitude upgrade

% sudo apt-get install locales dialog wget debconf devscripts gnupg

% sudo aptitude install libstdc++5 libstdc++6 libaio

• If you’re installing v12.5x software, you will need to install an even older copy of libstdc++ (not needed for v15 or higher software):

% aptitude install libstdc++2.10-glibc2.2

• Install Sybase software into /dbms/sybase (as root in the chroot environment which you should still be in)

% sudo cp /dbms/sybase/SYBASE.sh /dbms/sybase/SYBASE.env

% sudo echo DSQUERY=MYASE >> /dbms/sybase/.SYBASE.env

% sudo echo sa_password > /dbms/sybase/.sapwd

% sudo chmod go-rwx /dbms/sybase/.sapwd

% sudo chmod u-wx /dbms/sybase/.sapwd

% sudo chown -R sybase /dbms/sybase

• Install the sybase_chroot script into /usr/local/bin and make it executable by root

• Install the sybase_ase rc script into /etc/init.d and make it executable by root

• Exit the chroot environment by typing ‘exit’

• create a symbolic link outside of the chroot environment to make /home/sybase/dbms appear as /dbms.  This will allow you to access the Sybase software, notably OpenClient, as a normal user outside of the chroot environment.

% sudo ln -s /home/sybase/dbms /dbms

You are now able to start Sybase ASE using sudo /etc/init.d/sybase_ase start and stop Sybase ASE using sudo /etc/init.d/sybase_ase stop.  Starting Sybase Replication Server, OpenServer, or similar Sybase software is simply a matter of copying the /etc/init.d/sybase_ase and tweaking the script copy.

If you feel this is way complicated and Sybase should just fix their software, let them know.  Please point them to this page and refer to CR455393.

UPDATE: The GLIBC issue is fixed in v15.0.2 esd 4.  It doesn’t seem to be working for everyone though.  More information is needed about those systems where esd 4 doesn’t resolve the matter.  No fix is available for 12.5x from Sybase.

UPDATE: Keep in mind, that the Sybase ASE 15.0.2 esd 4 patch is available only to Sybase customers with a current maintenance contract with Sybase.

Listen to this podcast

Many of you may not know that Sybase openly lists the release schedules for ASE 12.5 and 15.0. The release schedules list not only when the estimated release date of the patches, but the list of bug fixes that are expected to be in each patch.

Targeted ASE 12.5.x Release Schedule and CR list Information

Targeted ASE 15.0 Release Schedule and CR list Information

How cool is that? Sybase has been providing this information for a number of years as you can still see the release schedule for the old 12.0 releases!

A word of caution: The release schedule and the bugs listed to be fixed in the patches are tentative and are not ’set in stone’. Sometimes during testing issues are found with bug fixes (it might cause other issues) or some other thing might come up that might delay the bug fix or release of the patch.

Listen to this podcast

Starting in ASE 12.5.2, Sybase added the with compression = X syntax, prior releases required the compress::X:: syntax. What is the difference?

In releases prior to 12.5.4 and 15.0.2, the two were functionally the same. Since 12.5.4 and 15.0.2, they are different:

ASE tells the backup server to load the zlib library and dump the database model to the file /tmp/model.dmp through the compress filter. The entire data stream is compressed. If you look at the dump file in a hex editor, then you will see the entire file is compressed.

ASE tells the backup server to load the zlib library and dump database model to the file /tmp/model.dmp but only the data volume is actually compressed. If you look at the dump file in a hex editor, you will see the first two volumes are mostly empty data (binary 0’s) but the embedded third volume is compressed.

You need to understand that whenever you dump a database to a file using this syntax, there are three volumes that are dumped. Two are administrative and the third is the data of your database.

In ASE 12.5.4 and 15.0.2, the Archive Database functionality was added

ASE requires that the archive file, your database dump file, is dumped with the “with compression = X” format, if it is compressed, as it needs access to the administrative volumes.

So, how does this affect us as DBAs? If you dump a database with ‘with compression = X’ on 12.5.4 esd5, you will NOT be able to load it into 12.5.3 esd 5 without using the 12.5.4 backup server. This has bitten us in the past when we attempted to load the customers’ database dumps.

Don’t forget to look at Sybase’s manual from time to time ;-).

Listen to this podcast

Scenario: Disaster Recovery Test :

1. Unix group restores the filesystems and creates the necessary raw devices for Sybase ASE (12.5.4 esd 5 64bit on Solaris)
2. We manually build the master (dataserver ….) and change the sort order to no_case iso_1 so we can load the master backup
3. We manually update sysdatabases.status for load (32) and sysdatabases.status2 for don’t recover (16) for the databases we wish to recover as we can’t load the database since the databases are marked suspect. restart, etc. Note that this is the only way for us to load the other databases without recreating each database (which can take hours of build time that we don’t have)
4. Dataserver starts properly noting that the databases are marked for load and don’t recover. We are now able to load the database backups.
5. Load database results in a 903 on an index for a sysobjects before the rpc is sent to the backupserver to start the backup - which doesn’t exist because the raw devices haven’t been written to yet. This occurs only in the v12.5x versions and doesn’t appear to occur in the v15 series.

DR Test involves a ‘virgin’ machine and SAN. As this was during a disaster recovery test, no errorlogs are available (no outside access) and the physical machines have been recycled for other DR tests.

Workaround: set sysdatabases.status to 320 and drop the database using “dbcc dbrepair(<dbname>, dropdb)”… then recreating the databases manually. Ed Barlow’s sp__revdb is a life saver

I’ve opened a case with Sybase Technical Support but I’m not very hopeful they will be able to reproduce the issue.

UPDATE!  SOLUTION:  The status2 column wasn’t being updated in the script file.  Damn.  Looks like a typo in the slightly modified script.
As there is much confusion within TechSupport as to why this method is necessary let me explain:

The time required to perform a “dbcc dbrepair(, dropdb)” and a “create database …. for load” can take hours or even days depending on the size of the databases and the speed of the devices. This is in no way acceptable for a real disaster recovery environment.

One might argue, “Why not just use Repserver?” (or equivalent?) Well, that’s all fine and dandy but what happens if your DR site and the primary are gone (flood, fire, terrorism, etc)?  You have to restore at an unknown DR site (we have dozens of *potential* DR sites) that you normally wouldn’t have access to.

I realize that this scenario is as foreign from what the TSEs are trained for at Sybase TechSupport as you can get. However, the old ways of a single DR site are long gone. Good or bad, depends on your point of view.

Listen to this podcast

When you try to install a Sybase patch for Openclient, ASE, ASIQ or a myriad of other Sybase products, you may receive an error such as:

“ASE 12.5.4 is NOT installed”

This can appear if you delete the $SYBASE/installed and/or the $SYBASE/uninstall directories. (Why would you delete those directories? They are not needed for the operation of any Sybase product, therefore can be deleted for space reasons.)

So how to install the patch? Set INSTALL_ALL_PATCH to “1″ prior to running setup:

export INSTALL_ALL_PATCH=1

Listen to this podcast

The class registration and listing is available on Sybase’s TechWave website

Here is my current schedule:

My Agenda     Time Session Title Learning Track Status Tuesday, August 7, 2007 2:30 pm - 5:00 pm  EDU106-1 Performance and Tuning: ASE Query Optimization   Education Courses and iAnywhere TechSummits Registered Wednesday, August 8, 2007 8:00 am - 10:15 am  EDU106-2 Performance and Tuning: ASE Query Optimization   Education Courses and iAnywhere TechSummits Registered 1:00 pm - 2:00 pm  DAT508 Using Java and Web Services within Adaptive Server Enterprise (ASE)   Enterprise Database, Data Management and Business Continuity Registered 2:15 pm - 3:45 pm  DEV311 Designing High Performance Applications/Transaction Design (Prevention not Diagnosis & Medication)   Application Development Trends Registered 4:00 pm - 5:30 pm  DEV339 Case-Express: Using Sybase tools and bleeding edge technologies for a feature-rich web application   Application Development Trends Registered Thursday, August 9, 2007 8:00 am - 10:15 am  EDU114-3 Migrating from ASE to Sybase IQ: An ASE DBA’s perspective   Education Courses and iAnywhere TechSummits Registered 1:00 pm - 2:00 pm  IAW410 Setting up Replication Server and OpenSwitch for Continuous Availability and Disaster Recovery   Data Integration, Analysis, and Warehousing Registered 3:30 pm - 5:00 pm  DEV319 Connecting Adobe Flex and Flash to Sybase Adaptive Server Enterprise (ASE) using Web Services   Application Development Trends Registered Friday, August 10, 2007 8:00 am - 10:15 am  EDU106-4 Performance and Tuning: ASE Query Optimization   Education Courses and iAnywhere TechSummits Registered 1:00 pm - 2:30 pm  DAT502B Top Ten Adaptive Server Enterprise (ASE) MDA Tables   Enterprise Database, Data Management and Business Continuity Registered

Listen to this podcast

On Linux, have you ever tried to load a database from backups with dozens of stripes and it failed even though you increased the resources on the backup server? Well, I recently ran into it while trying to load a 1TB database from over 100 stripe files.

Backup Server: 1.72.1.17: Bitmap routine called by function AT_ISSTRIPELDED returned -1
Backup Server: 1.72.1.18: Bitmap routine called by function AT_ISSTRIPELDED returned -1

This turns out to be a new bug with the Sybase Backupserver (both 12.5x and 15.0x codelines): CR 466497. If you run into this bug, please open a case with Sybase TechSupport and have your ticket associated with this bug.
WORKAROUND: Unfortunately, you will need to backup your database(s) with 32 stripes or less.

Listen to this podcast

Mich Talebzadeh, on the Sybase Product Futures mailing list, wrote up a verbose introduction to his proposed dump database …. COMPRESSION = n ESTIMATE_ONLY feature.  Basically it would give an estimate of time that the backup will take.  In my opinion, it would be a useful feature but I can’t see how it would be implemented as the compressability of the data could radically differ.  The Z Library, like any other compression mechanism, has no concept of an estimated time for compressing the current data chunk let alone for the rest of the data stream.  If the backup server were to give an estimated time, it wouldn’t be even remotely accurate in most cases.  Kudos to Mich for proposing it though. 

As we now have the capability to use archive database utility in ASE
12.5.4, the ability to interrogate and identify performance issues by
loading the problem database to archive DB is well in hand.

The question that comes to mind is the usual frequently asked question
during the middle of the trading day.

The users have identified a problem with trade capture. We need to
backup
the database, load it into a DEV server and quickly identify and fix
the
problem.

Technically nothing special about it. However, from a business point of
view it is highly visible as the business is impacted and few mangers
are involved including the DBA.

The DBA can give a guesstimate of how long is going to take to back up
the database. However, this is usually based on the overnight backups,
when the server is quiet.

Much like ‘kill SPID with statusonly’ that allows us to give an
estimate of how long is going to take for a rollbacked transaction to
complete, we also need the ability to give a fairly reasonable estimate
of how long is going to take to backup a large database and the
projected dump size, taking into account the load on ASE during the
business hours. This in my opinion is a feature that is desirable to
have.

DUMP DATABASE to ‘/dumpdevice/database_name.dmp’ WITH
COMPRESSION = n ESTIMATE_ONLY

This utility can also be used to estimate the timings of backups and
the
dump area size required on occasions when the OS or ASE is upgraded and
databases need to be backed up.

Listen to this podcast

With this plugin the destination of your dump no longer has to be a named file. It can now also be a stdin of an arbitrary command or a port on any host on your network.

Similarly, an existing dump can be LOADed from stdout of a command or from a port on any host.

Why would you want that? A number of possibilities open up:

• offload the CPU-intensive compression by running the compression utility on another machine;
• archive dumps on another host without NFS issues
• integrate with an off-site backup provider
• encrypt dumps without unencrypted data ever touching temporary storage;
• transfer databases from one server to another /directly/
• use the included “sybdump” utility (open sourced) in your own scripts, processing dumps without waiting for them to finish…

Pipe from Virtual Estates, uses the undocumented Backupserver API for the sybmultbuf to load in their libpipe.so / libpipe.dll file.  The end of the pipe can be just about anything, meaning you could conceivably dump your database using their pipe plugin, compress the stream, encrypt it, transform the data stream to morse code, send it over HAM radio (as morse code) to your buddy on another continent to reverse the method, loading it into her database.

Listen to this podcast

ASE 12.5.4 setup (java version of Installshield) unsets LD_POINTER_GUARD, which will cause some linux distributions to fail to ’setup’ dataserver.

This appears to be a variation of the issue raised in the "Sybase’s Backup Server for Linux (64bit) is broken " post.  What is unclear is whether this variable is being undefined by the Sybase supplied Java, Installshield itself or whether Sybase engineering is explicitly undefining it in the Installshield script.  I’ve opened a case with Sybase to ensure that this gets resolved with the ASE 12.5.4 esd 6 ebf.

Read the rest of this entry »

Listen to this podcast

Compress level 0 specifies:

1. do not compress the data
2. store the data using zlib

Think of it as the same as:

$ gzip -0 dumpfile.dmp

produces:  dumpfile.dump.gz

The file is simply encapsulated within the zlib archive.  Note that the compression mechanism within the backup server (and gzip btw) is zlib.

When you specify ‘compress level 0′ when dumping a database, you still need specify ‘compress’ to load the database.  The backup server isn’t intelligent enough to determine if the data is encapsulated with zlib or not.

Listen to this podcast

Apparently v12.5.3 esd 7 64bit on Linux has a problem with the backup server. It throws a signal 11 and dies. Nice feature, huh?

The problem is that esd 7 introduced a 64bit backupserver binary and still uses the old compat-libcstd++-5 library. The 64bit version of the library is somewhat flakey. Using the 32bit binary from esd 6 is the workaround (make sure to include the esd7 version of sybmultbuf too).

UPDATE!

The workaround is to set LD_POINTER_GUARD to 0 in the RUN_<SERVER>_BS file just prior to the backupserver command line.  Now, Sybase fixed the ASE binary (dataserver) atleast for the 12.5.x codeline but missed the backupserver and sybmultbuf (64bit versions).

UPDATE #2!

According to Sybase Techsupport, the problems with GLIBC 2.4 will be fixed within ASE 12.5.4 ESD #6! 

Listen to this podcast