1. Home
  2. Tutorials
  3. Linux Optimization
Yolinux.com Tutorial

Linux System Enhancements, Optimization and Compiling the Kernel

This tutorial covers optimization techniques to make your system run faster or support your needs better. This may include tweaking configuration specifications, turning off un-needed processes, and compiling a new kernel tweaked for your system. Note that graphics device drivers are NOT compiled into the kernel although support for interfacing with their particular chipset can be compiled in. This is because X-Windows is an application running on Linux and not integral to the kernel.

Optimize hard drive access performance:

Hdparm:

The following optimizations techniques can be applied to (E)IDE drives to utilize 32 bit I/O and DMA more effectively.

  • Get info: hdparm /dev/hda
  • Get more info: hdparm -I /dev/hda
  • Test Speed: hdparm -tT /dev/hda
  • Increase Speed: hdparm -d1 -c3 -u1 /dev/hda
    • -d1: Turn on DMA (0: off)
    • -c3: IDE chipset support
      • 0: Disable
      • 1: Enable
      • 3: 32 bit transfer
    • -u1: Get/set interrupt umask. Driver ignores other interrupts during processing of disk interrupts. (0: off)
  • Test Speed Again: hdparm -tT /dev/hda
The Red Hat configuration uses /etc/rc.sysinit to call scripts /etc/sysconfig/harddisks. (Earlier systems used: /etc/sysconfig/harddiskhda, /etc/sysconfig/harddiskhdb, etc) to run hdparm upon boot.

File: /etc/sysconfig/harddisks:

    USE_DMA=1
EIDE_32BIT=1
Turn off hdparm DMA access: hdparm -d0 /dev/hda

Installation: RH/CentOS: yum install hdparm

Links:


Mount "noatime":

If the drive is mounted with the directive "noatime", then the access time for the file is not updated in the inode table when the file is read. This increases performance because it reduces time to seek and time of writing the access time to the drive. This is done most commonly for read only file systems such as on many news servers.
Example: mount -o noatime,ro,dev,nouser -t ext3 /dev/hdc /mnt/news1


Ramdisk:

Go even faster by using ramdisk - Memory mapped as a disk. See: How to use a Ramdisk for Linux
Also see: /usr/src/linux-2.4/Documentation/ramdisk.txt

Tools:

Kernel Tweaks:

Direct kernel to specify processor affinity for a process:

Set (or retrieve) the CPU affinity. The scheduler attempts to keep the process on the specified CPU. This is especially effective for threaded applications to improve cache coherency on SMP computers. It is also useful for software licensed to a CPU or for a single CPU system when it is in fact running on an SMP system. (i.e. Oracle)

Kernel enhancements to optimize SCSI for scanner access:

  • See man page for sane: man sane-scsi
  • Excerpt from man page:
           First,  make  sure  your  kernel  has SCSI generic support
           enabled.  In ``make xconfig'', this shows up under  ``SCSI
           support->SCSI generic support''.
    
           To keep scanning times to a minimum, it is strongly recom-
           mended to use a large buffer size  for  the  generic  SCSI
           driver.   By  default,  Linux  uses a buffer of size 32KB.
           This works, but for  many  cheaper  scanners  this  causes
           scanning  to be slower by about a factor of four than when
           using a size of 127KB.  Linux defines  the  size  of  this
           buffer    by    macro    SG_BIG_BUFF    in   header   file
           /usr/include/scsi/sg.h.   Unless  a  system  is  seriously
           short  on memory, it is recommended to increase this value
           to the maximum legal value of  128*1024-512=130560  bytes.
           After  changing  this  value, it is necessary to recompile
           both the kernel (or the SCSI generic module) and the  SCSI
           backends.
    
           AUTHOR
                  David Mosberger    13 May 1998
                          
    
              

Kernel enhancements to increase the per process limit of open files:

Change the definition of NR_OPEN in local files:

Compiling the Kernel:

The default Red Hat Kernel comes with as much stuff compiled into the Kernel as they could fit. This allows it to support a vast array of devices and system needs. If you must add support for something new or if you wish to streamline your kernel with only what is needed so that it uses less memory, operates faster, and is optimized for your processor, then one must recompile the kernel.

To find out about the latest kernel, issue the command: finger @finger.kernel.org

Kernel source path:

Typically the source code as installed by the package manager installation is located in:

  • Red Hat / Fedora:
    • /usr/src/kernels/2.6.9-55.0.12.EL-i586 - It is now typical that the kernel image comes with the kernel source (Red Hat Enterprise / CentOS 4) all in the "kernel" RPM.
    • /usr/src/redhat/SOURCES/ - patches and kernel will be placed here by kernel source RPM. (Fedora Core 3)
      (bzip2 -d linux-2.6.9.tar.bz2, tar xf linux-2.6.9.tar)
    • /usr/src/linux-2.4/ - (Soft linked to /usr/src/linux-2.4.xx) Kernel 2.4. (Red Hat 7.1)
    • /usr/src/linux/ - (Soft linked to /usr/src/linux-2.2.xx) Kernel 2.2 and older. (Red Hat 6.x, 7.0)
  • Ubuntu / Debian:
    • /usr/src/linux - (Soft linked to /usr/src/linux-source-2.6.15/) Kernel 2.6 (Ubuntu 6.11)
      (install package: apt-get install linux-source will install the package linux-source-2.6.15)
    • linux-headers-2.6.15-27/
      (install package: apt-get install linux-headers will install the package linux-headers-2.6.15-27)

If you get new kernel source download to a new directory, create a soft linkt to /usr/src/linux/. i.e. /usr/src/linux-2.X.XX then ln -s /usr/src/linux-2.X.XX /usr/src/linux

Typically the kernel RPM installation (kernel-source-XXX.i386.rpm) will install the source and generate the appropriate links.

Dependencies:

Pre-requisite packages for building the kernel from source:
(see: /usr/src/linux/Documentation/Changes for the software versions required and how to check them)

  • Linux kernel 2.6:
    • gcc, GNU make, binutils, util-linux, module-init-tools, e2fsprogs, jfsutils, reiserfsprogs, xfsprogs, pcmcia-cs, quota-tools, PPP, isdn4k-utils, nfs-utils, procps, oprofile
  • Linux kernel 2.4:
    • gcc, GNU make, binutils, util-linux, modutils, e2fsprogs, reiserfsprogs, pcmcia-cs, PPP, isdn4k-utils

Ubuntu kernel package build:

Ubuntu 6.11 / Kernel 2.6:

The source code for Ubuntu installations is obtained and built into a Debian package by:

  • Install Ubuntu prerequisites: apt-get install build-dep build-essential fakeroot kernel-package linux-kernel-headers linux-headers-`uname -r` module-init-tools linux-image linux-386 linux-kernel-devel module-init-tools iptables libdrm2 autofs ncurses-dev
  • apt-get install linux-source
    This will also download the specific kernel source. In this case linux-source-2.6.15
  • Source location: /usr/src/linux-source-2.6.15.tar.bz2
  • cd /usr/src/
  • tar xjvf /usr/src/linux-source-version-number-here.tar.bz2
  • cd linux-source-version-number-here
  • cp -vi /boot/config-`uname -r` .config
    (config file for existing configuration)
  • Customize kernel config: make menuconfig or xconfig
  • Create a Debian ".deb" kernel package: fakeroot make-kpkg --initrd --revision=custom.1.0 kernel_image
    Where "mykernel.1.0" is any label you want to apply.
    or make-kpkg --initrd --append-to-version=mykernel kernel_image kernel_headers
  • Install Debian package: dpkg -i kernel-image-XXXXX.deb
See: Ubuntu compile

Typical Linux build from source:

Perform the following steps: (as root)

  1. Setup:

    • The source may already be installed as part of the installation or from an RPM (See /usr/src/).
      OR
      You may also download the latest source from http://www.kernel.org/pub/linux/kernel/.
      Move the downloaded file linux-2.X.XX.tar.gz to /usr/src/.
      Uncompress file: tar -xzf linux-2.X.XX.tar.gz which will install everything in a subdirectory linux/. (To view contents use command tar -tzf linux-...tar.gz)
      If using the file with the bz2 extension: bunzip2 linux-2.X.XX.tar.bz2
    • cd /usr/src/linux/
    • less README (Read the documentation. It tells you everything. Use space bar to view next page)
      (Local file link to: /usr/src/linux/README)
    • Update /usr/include/
      NOTE: PERFORM THIS STEP FOR KERNEL 2.2.x AND OLDER. DO NOT DO THIS FOR KERNEL 2.4 or 2.6.
      The header files for Kernel 2.4/2.6 and the C library header files may no longer match and thus must remain separated.
      • cd /usr/include
      • rm -fr asm linux scsi
      • ln -s /usr/src/linux/include/asm asm
      • ln -s /usr/src/linux/include/linux linux
      • ln -s /usr/src/linux/include/scsi scsi
      The soft links (ln -s) to the include directories are important to software developers who need to compile with the new include files.

      Note: The sym link is optional and some don't do it due to potential differences in the header files used by glibc and the kernel.

    • cd /usr/src/linux (or /usr/src/linux-2.4)
    • make distclean (Cleans up junk from old compiles) (Warning: removes .config)
    • make mrproper (build the source tree)

    At this time the sources are correctly installed. This also resets the /usr/src/linux/.config file to the system default.

    SuSE setup:

    • For SuSE (9.3), there are two kernel-source RPMs. One RPM resides on the "Source" DVD-2 and the other resides on the "Installation" DVD-1. Use the kernel-source RPM found on the "Installation" DVD-1. This one will register in the RPM database and creates the directories expected
      Check with the command: rpm -q kernel-source
      Kernel source code location: /usr/src/linux
      Docs location: /usr/share/doc/packages/kernel-source
    • The SuSE kernel-source RPM will not produce a ".config" file for the kernel compilation.
      Produce the default ".config" file by: zcat /proc/config.gz > /usr/src/linux/.config

  2. Configuration:

    Use one of the following tools to create the .config file. This gives you the chance to choose what goes into the kernel. You can choose support for many of the latest capabilities, device drivers, and can tune the kernel for particular uses. i.e. you can also optimize the compilation for the Pentium. (Default for RH 5.2 was a 386 with no math co-processor!) The default config file used to compile the default distribution kernel can be found in the source directory. (i.e. Fedora Core 3: /usr/src/redhat/SOURCES/kernel-2.6.9-i686.config, /usr/src/redhat/SOURCES/kernel-2.6.9-x86_64-smp.config, ... and all the others for the various hardware platforms.)

    Your kernel must support your hardware. During the kernel configuration process, you will have to know what hardware support is needed in the kernel. To view the system hardware use the commands:

    • PCI bus attached hardware: /sbin/lspci
    • CPU information: cat /proc/cpuinfo

    Pick one of the following:

    • make config (Bash shell script)
    • make menuconfig (uses text window curses)
    • make xconfig (uses tck and wish) - recommended due to online help feature and intuitive interface. Save configuration to file: .config
    • make oldconfig Build a configuration file based on defaults found in current .config file.

    menuconfig Notes: "< > options can be compiled as a module.

    • <*> = Compile into kernel (y)
    • <M> = Compile as a module (m)

    Recommendation for Pentium owners: General Setup choose :

    • Kernel math emulations: N
    • Processor type: Pentium
    • Development, experimental or kernel debugging features: N
    • If using netfilter capabilities, do not enable the kernel fast-switching option. That code will permits fast switching at a lower level in the IP stack than netfilter and thus will bypass the netfilter capabilities.
    • Note: When requiring gobs of memory (> 1Gb), there is a kernel compile option which sets the limit on the amount of user memory one can use: "Processor type and features / Maximum Virtual Memory".

  3. Compile:

    Next: (from /usr/src/linux)
    • make dep (make dependencies. Older kernels only. Attempt and if not needed, make will return a message stating that this step is un-necessary. Not required for 2.6+ kernels.)
    • make clean (At this point you have all the source gathered)
    • Choose one of the following to create the new kernel and then move it to /boot/vmlinux:
      • make zImage (compressed image)
      • make bzImage (very compressed image good for recovery floppies - slower to boot because of time to uncompress)
      • make image
      • make bzdisk (Boot floppy with no LILO)

  4. Install:

    • mv /usr/src/linux/arch/i386/boot/zImage /boot/vmlinuz-release
      ln -s /boot/vmlinuz-release /boot/vmlinuz
      (If "make zImage" was chosen)
      or
      mv /usr/src/linux/arch/i386/boot/bzImage /boot/vmlinuz-release
      ln -s /boot/bzImage-release /boot/vmlinuz
      (If "make bzImage" was chosen)
      Also move/copy the file /usr/src/linux.../System.map to /boot/System.map-2.X.XX
      OR
      make install
      (Preferred. Potential Pitfall: On my SCSI system I had to execute this command after issuing the mkinitrd command below.)
    • Modules documentation: (not that helpful)
    • make modules
    • rm /lib/modules/Kernel_release (i.e. /lib/modules/2.2.16-3 )
    • ln -s /lib/modules/new_modules_directory /lib/modules/Kernel_release
    • make modules_install
      Generates the file: /lib/modules/2.X.XX/modules.dep

  5. Configure Boot Loader:

    (Grub or Lilo)
    • Configure grub:
      File: /etc/grub.conf
      default=1             - Kernel to boot (2nd in list)
      timeout=5
      splashimage=(hd1,0)/boot/grub/splash.xpm.gz   - Boot image location (hd1, 1st partition)
      hiddenmenu
      title Fedora Core (2.6.12-1.1372_FC3)
              root (hd1,0)                          - Root of GRUB reference. Same as /boot
              kernel /boot/vmlinuz-2.6.12-1.1372_FC3 ro root=LABEL=/ rhgb quiet
              initrd /boot/initrd-2.6.12-1.1372_FC3.img
      title Fedora Core (2.6.11-1.14_FC3)
              root (hd1,0)
              kernel /boot/vmlinuz-2.6.11-1.14_FC3 ro root=LABEL=/ rhgb quiet
              initrd /boot/initrd-2.6.11-1.14_FC3.img
              
      Description:
      • default: Default kernel to boot. All options displayed but one will boot by default.
      • timeout: Time for user to select kernel. After "timeout" seconds, default kernel chosen and boot begins.
      • splashimage: Background image shown by Grub and location (hard drive hd1 and "0" or first partition (/boot).
      • hiddenmenu: I recommend commenting this out while compiling and developing kernels. This option takes way the user's ability to choose which kernel to boot. You always want the ability to choose a known reliable kernel just in case the one you just built is a disaster.
      • root: Grub root directory. In this example, drive hd1 and "0" or first partition (/boot).
      • kernel: Kernel to boot, read only. Note that this is the same as the result of cat /proc/cmdline
      • initrd: Initial ramdisk for a Linux format boot image. Sets parameters in the Linux setup area in memory.
      See:

    • Configure lilo: Lilo must point to the new kernel. Edit /etc/lilo.conf. Add a new image statement to point to the new kernel. Keep the old as backup in case you need to boot it.

      SCSI systems must perform an additional step to load drivers upon system boot:

      • Execute the following command which creates the file used to let the system boot:
              mkinitrd  /boot/initrd-2.2.16-3.img  2.2.16-3
        The second argument is the name of the sub-directory of the modules under the directory /lib/modules/. (In this case /lib/modules/2.2.16-3/

        This step is required by systems using modular SCSI support. If the SCSI support is compiled into the kernel and is NOT a module, then this step may not be necessary. The file /boot/initrd-2.2.16-3.img is specified as the RAM disk image used before the file system is available. (initrd = initial ram disk)

      Run /sbin/lilo -v to configure the master boot record with data from lilo.conf.

      Sample lilo.conf:

      boot=/dev/sda
      map=/boot/map - Location of the kernel used for boot install=/boot/boot.b prompt timeout=50 linear - Specific to SCSI configurations default=linux image=/boot/vmlinuz-2.2.16-3 - New kernel label=linux - Name to be displayed by Lilo boot manager initrd=/boot/initrd-2.2.16-3.img - Specific to SCSI boot drives. read-only root=/dev/sdb6 image=/boot/vmlinuz-2.2.14-5.0 label=OldLinux initrd=/boot/initrd-2.2.14-5.0.img - SCSI specific. read-only root=/dev/sdb6

      When the system boots and gives you the "Lilo:" prompt, you can type "?" to see your boot choices as defined by the above labels in lilo.conf. Type the label name of the kernel or system you wish to boot. A carriage return will choose the default.

    For a full list of boot parameters see: bootparam man page

    Also see the Boot Prompt Howto.

Kernel version verification: uname -rv will print the compile date of the kernel currently running.


Modules:

Modules are used to reduce the amount of memory used to hold the kernel. There is a slight penalty for the time taken to load and unload the module.

If the code is required for general operation of the kernel or is needed often or required by the boot process, it is best to compile it into the kernel and it should NOT be compiled as a module.

Module commands:

The kernel will use the modprobe utility to determine if the module is compatible with the kernel.
The program used is specified by a proc file:

    cat /proc/sys/kernel/modprobe
/sbin/modprobe - Result of cat operation

Modules are loaded by init scripts which call insmod/rmmod to load/unload modules.

Command Description
lsmod List loaded modules

  [ /root]# lsmod
  Module                  Size  Used by
  3c59x                  19716   0  (autoclean) (unused)
  autofs                  9120   1  (autoclean)
  emu10k1                45688   1 
  soundcore               2596   4  [emu10k1]
  aic7xxx               137400   3 
                
insmod Inserts a module into the active kernel
rmmod Remove a loaded module. Just specify the module name. No ".o" or path necessary.
modprobe High level handling of loadable modules.
Loads module and dependencies.
depmod Creates dependencies file for a module (used by modprobe)
modinfo Display information about a kernel module

List of modules held in /etc/modules.conf (or for older systems: /etc/conf.modules)
Sample:

alias scsi_hostadapter aic7xxx
alias eth0 eepro100
alias eth1 eepro100
alias parport_lowlevel parport_pc
alias sound-slot-0 emu10k1
alias char-major-81 bttv

Other useful commands:

  • Display system status gathered from /proc: /usr/bin/procinfo -a
  • List Processor type: cat /proc/cpuinfo
  • List devices:
    • cat /proc/devices
    • /sbin/lsmod
  • List IO ports (device address used drivers): cat /proc/ioports
  • List DMA channels: cat /proc/dma
  • View interrupts used by the system:
    • cat /proc/interrupts
    • procinfo -i
    IRQ Description
    0 Timer channel 0
    1 Keyboard
    2 Cascade for controller 2 (which controls IRQ 8-15)
    3 Serial Port 2 (COM2)
    4 Serial Port 1 (COM1)
    5 Parallel Port 2 (LPT2)
    6 Floppy Diskette Controller
    7 Parallel Port 1 (Printer LPT1)
    8 Real-time Clock
    9 Redirected to IRQ2
    10 --
    11 --
    12 PS/2 Mouse
    13 Math Co-processor
    14 Hard Disk Controller (IDE)
    15 --

    Also see file:
    • /etc/isapnp.gone for a list of protected IRQ's which will not be allocated to ISA Plug and Play device.
    • Command: pnpdump
      This command scans ISA PnP cards and can be used to generate /etc/isapnp.conf
      pnpdump > /etc/isapnp.conf
    • Command: isapnp -C /etc/isapnp.conf
      Uses file /etc/isapnp.gone and /proc files to avoid IRQ's already allocated and to determine safe IRQ's.
    • Plug and Play ISA Specification
  • Create devices. Run script: /dev/MAKEDEV
  • Display kernel version in current use: /proc/version
  • Display boot messages: dmesg
  • Display sound driver status: cat /dev/sndstat


[Potential Pitfall]: On platforms without much disk space (such as embedded platforms), a kernel compile can potentially fill up your drive potentially terminating the compile prematurely. This is most often a result of the GNU C compiler cache hogging up as much as 0.5Gb to 1Gb of disk space in $HOME/.ccache/. The GNU C compiler generates and uses this cache to speed up the compile. This can be disabled by setting the following environment variable CCACHE_DISABLE by employing the command: export CCACHE_DISABLE=
The entire cache can be deleted and is no longer required once the environment variable has been set: rm -Rf $HOME/.ccache
In no way will the compiled results be any different but it will save allot of disk space.

Kernel Source and Patches:

Linux kernel naming convention: Version.MajorRelease.MinorRelease.PatchNumber
(In Linux kernel vernacular: KernelVersion.PatchLevel.Sublevel.Extraversion)

Patch File Description: The patch files are file "diffs" (differences from the stated source file) which can be applied to create a newer updated file.

Downloading kernel source and applying a patch:

  • Obtain kernel source: ftp://ftp.us.kernel.org/pub/linux/kernel/v2.X/linux-2.X.X.tar.gz
    Uncompress: tar xzf linux-2.X.X.tar.gz
    (Or: tar xjf linux-2.X.X.tar.bz2)
    This creates the local directory: linux/...
    OR
    use the source from the kernel-source RPM package released by your distribution. Using the source released by your distribution will include patches and source code needed to support all the features of your distribution. Of greatest importance is the default ".config" file which describes the features used by your distribution. i.e. Red Hat / Fedora Core use the ext3 file system, devfs and turn on many features by default.
  • Download patch:
    • Download patch: ftp://ftp.us.kernel.org/pub/linux/kernel/v2.X/patch-2.X.X
    (Place patches in directory (create it first): /usr/src/KernelPatches/)
  • Apply patch: Apply patches from default source directory /usr/src/linux (linked to actual source directory):
    • gzip -cd ../KernelPatches/patch-file.gz | patch -p0
    • bzip2 -dc ../KernelPatches/patch-file.bz2 | patch -p0
    • If patch file resides in directory /usr/src/linux:
      zcat patch-file.gz | patch -p0
    Use script patch-kernel if applying multiple patches at once: (i.e. patch-2.6.11.5 to patch-2.6.11.7)
    (Script located in directory linked to kernel source)
    • Place all patches in /usr/src/KernelPatches/ (.gz or .bz2)
    • Apply all the patches to kernel (i.e. 2.6.11.4)
      /usr/src/linux/scripts/patch-kernel linux KernelPatches
      (Execute from directory /usr/src)
    • Or apply all the patches to kernel to bring it up to specified level.
      /usr/src/linux/scripts/patch-kernel linux KernelPatches 2.6.11.7
      (patch-kernel kernel-source-directory-to-act-on highest-version to include)
  • Set-up directory links:
    • mv linux linux-2.X.XX (only required on some older RPMs)
    • ln -s linux-2.X.XX linux
    • /usr/include/asm should point to /usr/src/linux/include/asm-i386
    • /usr/include/linux should point to /usr/src/linux/include/linux
    • /usr/include/scsi should point to /usr/src/linux/include/scsi


Example: Applying the devfs patch to kernel 2.2.20

  • Download kernel: http://www.kernel.org/pub/linux/kernel/v2.2/linux-2.2.20.tar.gz
  • mv linux-2.2.20.tar.gz /usr/src
  • cd /usr/src/
  • Remove previous source: rm linux (May use rm -Rf linux/)
  • Unload to subdirectory linux/: tar xzf linux-2.2.20.tar.gz
  • Download patch: http://www.atnf.csiro.au/~rgooch/linux/kernel-patches.html
  • Download patch file devfs-patch-current to /usr/src/
  • cd /usr/src
  • Test: patch -p0 --dry-run < devfs-patch-current
  • Apply patch: patch -p0 < devfs-patch-current
  • Now go through regular kernel build as described above. (From the beginning) make distclean; make mrproper
  • The configuration has been altered by the patch thus make xconfig will have options not seen in the original 2.2.20 kernel.
    • "Code maturity level" option
    • Select "development and/or incomplete code/drivers"
      This will provide lots of options for kernel.
    • Go to Filesystems - check /dev file support "Experimental"
      This feature was introduced by the patch. If it is not selected the devfs feature introduced by the patch will not be included.
Kernel source must be in /usr/src/linux/ (From kernel.org) or /usr/src/linux-2.2.20/ for the patch command as given, to work. The patch is pre-configured for this.

Patch command options:

Option Description
-DMACRO-NAME Define pre-processor macro
-c Use diff file format.
-d directory Change to directory.
--dry-run Don't modify file but output results.
-s Silent mode.
-f Force change.
-n Using patch file which is a regular diff file.
-R Reverse or undo patch. (i.e. if patch rejected.)
-p0 Do not strip directory path prefix. Path intact. i.e. /usr/src/linux/...
-p1 Strip first "/" in directory path. i.e. usr/src/linux/...
-p2 Strip first directory in path. i.e. src/linux/...
-p# Directory path "/" prefixes to strip (# - 1).
Look at the patch file to see the directory path you will reference.
View patch file: zcat patch-file.gz | more

patch man page.

Tips:

More Information:

Book imageBooks:

"Understanding the Linux Kernel"
by Daniel Pierre Bovet, Marco Cesati
O'Reilly & Associates, ISBN# 0596005652
3 edition (November 1, 2005)

Amazon.com
"Linux Kernel Development"
by Robert Love
Sams, ISBN# 0672327201
2 edition (January 12, 2005)

Amazon.com
"Linux Core Kernel Commentary: Guide to Insider's Knowledge on the Core Kernel of the Linux Code"
by Scott Maxwell
Coriolis Group Books, ISBN# 1576104699
(October 20, 1999)

Amazon.com
"Linux Kernel Module Programming Guide"
by Ori Pomerantz
Iuniverse Inc, ISBN# 0595100422
(August 2000)

Amazon.com