Custom actions provide a generalized way of extending the Windows Installer engine.  Due to the fact it's a generalized extension mechanism, it's difficult to provide a universal cookbook for custom actions.  I suggest starting with the Windows Installer portion of the Platform SDK which contains details on the base constructs as well as a few custom action recipes.  MSDN also has a few examples of how other Microsoft technologies have used custom actions to configure their underlying technologies.

 

Windows Installer Start Page
About Windows Installer
Rollback Installation
Using Windows Installer
Changing the Target Location for a Directory
Hiding the Cancel Button During an Installation
Windows Installer Guide
User Interface
User Interface Reference
Controls
MaskedEdit Control
Control Events
DoAction ControlEvent
Standard Actions
Using Standard Actions
Standard Actions Reference
InstallValidate Action
Custom Actions
About Custom Actions
Custom Action Sources
Synchronous and Asynchronous Custom Actions
Rollback Custom Actions
Commit Custom Actions
Deferred Execution Custom Actions
Executable Files
Dynamic-Link Libraries
Scripts
Formatted Text Custom Actions
Error Message Custom Actions
Nested Installation Actions
Custom Action Security
Using Custom Actions
Invoking Custom Actions
Sequencing Custom Actions
Obtaining Context Information for Deferred Execution Custom Actions
Adding Custom Actions to the ProgressBar
Debugging Custom Actions
Determining UI Level from a Custom Action
Uninstalling Custom Actions
Returning Error Messages from Custom Actions
Setting a Restore Point from a Custom Action
Functions Not for Use in Custom Actions
Changing the System State Using a Custom Action
Accessing the Current Installer Session from Inside a Custom Action
Accessing a Database or Session from Inside a Custom Action
Using a Custom Action to Launch an Installed File at the End of the Installation
Authoring the Launch Custom Action
Adding Launch to the CustomAction and Binary Tables
Adding a Control Event at the End of the Installation to Run Launch
Using a Custom Action to Create User Accounts on a Local Computer
Authoring the Custom Actions
Adding a Custom CustomUserAccounts Table
Authoring the CustomAction Table
Authoring the ActionText and Error Tables
Authoring the InstallExecuteSequence Table
Authoring the User Interface for Password Input
Securing the installation
Using 64-bit Custom Actions
Custom Action Reference
Summary List of All Custom Action Types
Custom Action Type 1
Custom Action Type 2
Custom Action Type 5
Custom Action Type 6
Custom Action Type 7
Custom Action Type 17
Custom Action Type 18
Custom Action Type 19
Custom Action Type 21
Custom Action Type 22
Custom Action Type 23
Custom Action Type 34
Custom Action Type 35
Custom Action Type 37
Custom Action Type 38
Custom Action Type 39
Custom Action Type 50
Custom Action Type 51
Custom Action Type 53
Custom Action Type 54
Custom Action Execution Scheduling Options
Custom Action Hidden Target Option
Custom Action In-Script Execution Options
Custom Action Return Processing Options
Custom Action Return Values
Return Values of JScript and VBScript Custom Actions
Properties
Property Reference
RollbackDisabled property
ReplacedInUseFiles Property
Patching and Upgrades
Major Upgrades
Preventing an Old Package from Installing Over a Newer Version
Patching
Removing Patches
Uninstallable Patches
Uninstalling Patches
Patch Uninstall Custom Actions
Package Validation
Internal Consistency Evaluators - ICEs
ICE Reference
ICE12
Windows Installer on 64-bit Operating Systems
Using 64-Bit Windows Installer Packages
64-Bit Custom Actions
Windows Installer Examples
Windows Installer Scripting Examples
Windows Installer Reference
Installer Database
Installer Database Reference
Database Functions
MsiGetLastErrorRecord function
Database Tables
CustomAction Table
MsiPatchMetadata Table
LockPermissions Table
Errors Reference
Windows Installer Error Messages


here’s some other write-ups others have done around using custom actions to address their aspect of technology

Media Center Add-In:
Windows XP Media Center Edition SDK Creating a Windows Installer File for an Add-in
Smart Tags and VSI:
Deploying Smart Tag DLLs by Using the Visual Studio Installer (Smart Tags Technical Articles)
SQL and Script Custom Actions:
Scripting a Custom Database Installation (SQL Server Magazine 2001)
SharePoint WebParts:
Using Wppackager to Package and Deploy Web Parts for Microsoft SharePoint Products and Technologies
Business Portal:
Adding custom actions to the installer
MSDE 2000:
Embedding MSDE 2000 Setup into the Setup of Custom Applications (Microsoft SQL Server 2000 Technical Articles)
DIFx:
How DIFxApp Works
Installing Driver Packages
PocketPC:
Developing and Deploying Pocket PC Setup Applications (Pocket PC (General) Technical Articles)

[Author: Robert Flaming]
This posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.