SQLite

Team LiB
Next Section
Book Cover
• | Table of Contents
• | Index
SQLite
By Chris Newman
Publisher | : Sams Publishing
Pub Date | : November 09, 2004
ISBN | : 0-672-32685-X
Pages | : 336

SQLite is a small, fast, embeddable database. What makes it popular is the combination of the database engine and interface into a single library as well as the ability to store all the data in a single file. Its functionality lies between MySQL and PostgreSQL, however it is faster than both databases.

In SQLite, author Chris Newman provides a thorough, practical guide to using, administering and programming this up-and-coming database. If you want to learn about SQLite or about its use in conjunction with PHP this is the book for you.

Team LiB
Next Section


Table of Contents

Team LiB
Book Cover
• | Table of Contents
• | Index
SQLite
By Chris Newman
Publisher | : Sams Publishing
Pub Date | : November 09, 2004
ISBN | : 0-672-32685-X
Pages | : 336

Copyright
Foreword
About the Author
We Want to Hear from You!
Reader Services
Introduction Welcome to SQLite
 | Why Use SQLite?
 | Who This Book Is For
 | How the Book Is Organized
 | Versions of Software Covered
 | Additional Resources
Part I: General SQLite Use
 | Chapter 1. Getting Started
 |  | Introduction
 |  | Features and Limitations
 |  | When Not to Choose SQLite
 |  | Looking at SQLite Databases
 |  | Help and Support
 | Chapter 2. Working with Data
 |  | SQLite Basics
 |  | Querying and Updating the Database
 | Chapter 3. SQLite Syntax and Use
 |  | Naming Conventions
 |  | Creating and Dropping Tables
 |  | Anatomy of a SELECT Statement
 |  | Attaching to Another Database
 |  | Manipulating Data
 |  | Indexes
 |  | Views
 |  | Triggers
 |  | Working with Dates and Times
 |  | SQL92 Features Not Supported
 | Chapter 4. Query Optimization
 |  | Keys and Indexes
 |  | The EXPLAIN Statement
Part II: Using SQLite Programming Interfaces
 | Chapter 5. The PHP Interface
 |  | Configuring PHP for SQLite Support
 |  | Using the PHP SQLite Extension
 |  | Working with User-Defined Functions
 |  | Using the PEAR Database Class
 | Chapter 6. The C/C++ Interface
 |  | Preparing to Use the C/C++ Interface
 |  | Using the C Language Interface
 |  | Adding New SQL Functions
 | Chapter 7. The Perl Interface
 |  | Preparing to Use the SQLite Interface
 |  | About the Perl DBI
 |  | Using the SQLite DBD
 |  | Adding New SQL Functions
 | Chapter 8. The Tcl Interface
 |  | Preparing to Use the Tcl Interface
 |  | Using the Tcl Interface
 | Chapter 9. The Python Interface
 |  | Preparing to Use the Python Interface
 |  | Using the Python Interface
Part III: SQLite Administration
 | Chapter 10. General Database Administration
 |  | The PRAGMA Command
 |  | Backing Up and Restoring Data
 |  | Exploring the SQLite Virtual Database Engine
 |  | Access to the Database File
Part IV: Appendixes
 | Appendix A. Downloading and Installing SQLite
 |  | Obtaining SQLite
 | Appendix B. Command Reference for the sqlite Tool
 |  | Dot Commands
 | Appendix C. SQL Syntax Reference
 |  | Naming Conventions
 |  | SQL Command Syntax
 |  | ANSI SQL Commands and Features Not Supported
 | Appendix D. PHP Interface Reference
 |  | Predefined Constants
 |  | Runtime Configuration
 |  | Function Reference
 | Appendix E. C Interface Reference
 |  | The Core API
 |  | The Non-Callback API
 |  | The Extended API
 |  | Adding New SQL Functions
 | Appendix F. Perl Interface Reference
 |  | The Perl DBI
 | Appendix G. Tcl Interface Reference
 |  | The Tcl Library
 | Appendix H. Python Interface Reference
 |  | Opening and Closing a Database
 |  | Creating User-Defined Functions
 |  | Error Handling
 | Appendix I. The Future of SQLite
 |  | SQLite Version 3.0
Index

Team LiB


Copyright

Copyright © 2005 by Sams Publishing

All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein.

Library of Congress Catalog Card Number: 2004901350

Printed in the United States of America

First Printing: November 2004

07 06 05 04 4 3 2 1

Trademarks

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Sams Publishing cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Warning and Disclaimer

Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an "as is" basis. The author and the publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book.

Bulk Sales

Sams Publishing offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales. For more information, please contact

U.S. Corporate and Government Sales
1-800-382-3419
corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact

International Sales
international@pearsoned.com

Credits

Acquisitions Editor

Shelley Johnston

Development Editor

Damon Jordan

Managing Editor

Charlotte Clapp

Project Editor

Andy Beaster

Copy Editor

Margaret Berson

Indexer

Erika Millen

Proofreader

Elizabeth Scott

Technical Editors

Daniel Kennedy

Benjamin Carlyle

Publishing Coordinator

Vanessa Evans

Book Designer

Gary Adair


Foreword

When I first began composing SQLite in late May of 2000, I never imagined that a few years later it would be widely used in consumer electronics devices and in countless programs and that people I barely know would be writing books about it. Like most other open-source software, SQLite grew out of a personal need and was intended primarily for my own use. In January of 2000, I was working with a team from General Dynamics on a software project that was connected to an Informix database. When Informix works, it works well, but we were having problems getting it to start reliably when the host computer was rebooted. We substituted PostgreSQL for Informix on our development systems, but even that database was a hassle to administer. While I was struggling to deal with these issues, the idea arose to write a simple SQL database engine that was serverless, read ordinary disk files, and could be statically linked into the application. I took no action on the idea then. But five months later, I was without a contract for a few months and so I started writing SQLite with the thought that it would be handy the next time a similar problem appeared. The basic database engine was running within a day and was stable within a couple of weeks. Development was interrupted for a month when my wife, Ginger, and I traveled to central Africa to visit friends. After our return and exactly four years ago from the day that I write this, I posted SQLite version 1.0 on the web.

Version 1 of SQLite attracted a small number of users. But the library was of limited utility because it employed GDBM as a storage back end. GDBM has no transaction support and it is based on hashing, so indices could not be used to optimize queries that are constrained by inequalities. To address these limitations, I began working in my spare time on a replacement B-Tree back end in the spring of 2001. Several months passed. I remember that I was fixing some last-minute bugs in the new system one morning when Ginger called from work to tell me that an airplane had just crashed into the World Trade Center in New York. Version 2.0 was released a couple of weeks later, amid a zeitgeist of sadness.

The release of SQLite version 2.0 triggered a surge of interest. Within weeks, Christian Werner published the first SQLite add-on, an ODBC driver. Dozens of other wrappers would soon follow. Over the next 12 months, support was added for INTEGER PRIMARY KEY, for VIEWs, and LEFT OUTER JOINs. Dan Kennedy contributed code to implement trIGGERs. By the fall of 2002, SQLite was in essentially the same form as you find it today. All through that year and since, more and more people began using SQLite in their programs and products. In recent months, the SQLite website has received visitors from around three thousand distinct IP addresses per day. Source code downloads average more than 400 per day with almost twice that many binary downloads. Total website traffic is approaching one gigabyte per day. SQLite bindings now exist for over two dozen languages including Perl, Python, PHP, Tcl/Tk, Ruby, Lisp, and Java. SQLite has been incorporated into many popular open-source projects, such as Kexi, monotone, Mozilla, and Popfile to name a few. I have been told, in confidence, of many commercial software projects built around the library. SQLite has also been spotted in consumer electronics devices such the Philips HDD060 MP3 player and in the D-Link DSM-320 Wi-Fi Media Player. No doubt countless other uses of SQLite have escaped my notice.

From its inception, the primary design goal of SQLite has been simplicity. I have tried to keep SQLite simple to administer, simple to operate, simple to program, and simple to maintain and customize. Many people tell me that they like SQLite because it is small, fast, and reliable. Reliability is a consequence of simplicitywith less complication, there is less to go wrong. Small size and fast performance are just happy accidents. Simplicity is the ultimate goal. In this way, SQLite is different from enterprise-class database engines that give you most everything you could ever want, except for simplicity. SQLite may have fewer features, but it is much simpler to use and operate, which is more important than a rich feature set in many situations. This is not to say that SQLite will not add new features over time. SQLite will continue to advance and growwe are currently looking at adding support for ALTER TABLE and for foreign keys, for examplebut as long as I control its development, SQLite will continue to be as simple as I can make it.

Except for a few lines here and there, most of the code in SQLite was written by me and Dan Kennedy. But we have not been working in isolation. Suggestions and criticisms from the SQLite user community have been invaluable in helping to direct the library's progress. And though no outside code has been copied into SQLite, other software has been essential in the process of building and testing SQLite. Special thanks go to John Ousterhout and his Tcl scripting language. We would have never been able to get SQLite working had it not been for the Tcl language, which allowed us to write over 33,000 lines of regression test scripts. Other infrastructure software has also played a key role. There would be no Windows port of SQLite except for the MinGW build environment. SQLite would never even have been started but for the innumerable hordes of developers that have given us X11, GNU, and Linux. SQLite is not a work unto itself but rather a very small and insignificant part of a much larger software ecosystem.

I have formally dedicated my contributions to SQLite to the public domain, and I insist that other contributors do likewise. (I keep a file of signed copyright releases in the fire safe at my office.) This means that there are no legal restrictions on copying SQLite. My name appears nowhere in the code. You are free to do with SQLite whatever you want. I hope that you find SQLite to be helpful and that you will use it for good. You are welcome to use SQLite for commercial purposes and if you do, I wish you a good profit. Regardless of how you use it, I hope the fact that you have received SQLite freely will inspire you to give something back in returnnot to me personally but to the community or to the world as a whole. Finally, and most importantly, I encourage you to find forgiveness for your own soul and to forgive others in return. Be at peace.

D. Richard Hipp
Charlotte, NC
August 17, 2004


About the Author

Chris Newman is a consultant programmer specializing in the development of custom web-based database applications to a loyal international client base.

A graduate of Keele University, Chris lives in Stoke-on-Trent, England, where he runs Lightwood Consultancy Ltd, the company he founded in 1999 to further his interest in Internet technology. Lightwood operates web hosting services under the DataSnake brand and is proud to be one of the first hosting companies to offer and support SQLite as a standard feature on all accounts.

More information on Lightwood Consultancy Ltd can be found at http://www.lightwood.net, and Chris can be contacted at chris@lightwood.net.


We Want to Hear from You!

As the reader of this book, you are our most important critic and commentator. We value your opinion and want to know what we're doing right, what we could do better, what areas you'd like to see us publish in, and any other words of wisdom you're willing to pass our way.

You can email or write me directly to let me know what you did or didn't like about this bookas well as what we can do to make our books stronger.

Please note that I cannot help you with technical problems related to the topic of this book, and that due to the high volume of mail I receive, I might not be able to reply to every message.

When you write, please be sure to include this book's title and author as well as your name and phone or email address. I will carefully review your comments and share them with the author and editors who worked on the book.

Email: | opensource@samspublishing.com
Mail: | Mark Taber Associate Publisher Sams Publishing 800 East 96th Street Indianapolis, IN 46240 USA


Reader Services

For more information about this book or another Sams title, visit our Web site at www.samspublishing.com. Type the ISBN (excluding hyphens) or the title of a book in the Search field to find the page you're looking for.


Introduction Welcome to SQLite

Team LiB

Introduction
Welcome to SQLite

SQLite is one of the fastest-growing database engines around, but that's growth in terms of popularity, not anything to do with its size. In fact one of SQLite's greatest strengths is that it is extremely lightweight indeed yet still manages to retain a large number of features.


Why Use SQLite?

There are many reasons for choosing SQLite, including

  • Performance SQLite performs database operations efficiently and is faster than other free databases such as MySQL and PostgreSQL.

  • Size SQLite has a small memory footprint and only a single library is required to access databases, making it ideal for embedded database applications.

  • Portability SQLite runs on many platforms and its databases can be ported easily with no client/server setup or ongoing administration required.

  • Stability SQLite is ACID-compliant, meeting all four criteriaAtomicity, Consistency, Isolation, and Durability.

  • SQL support SQLite implements a large subset of the ANSI-92 SQL standard, including views, subqueries, and triggers.

  • Interfaces SQLite has language APIs for C/C++, PHP, Perl, Python, Tcl, and many more beyond those covered in this book.

  • Cost SQLite is in the public domain and therefore is free to use for any purpose without cost and can be freely redistributed.


Who This Book Is For

This book is aimed at intermediate- to advanced-level programmers looking to include database functionality within their applications. You should have at least a basic working knowledge of one of the languages covered by this bookC/C++, PHP, Perl, Python, or Tcl. The underlying library is written in C/C++; however, it is not necessary to understand that language in order to use the full capabilities of SQLite in your applications.

If you are a new programmer you should still be able to pick up SQLite fairly quickly, but this book does not cover programming basics in any of the languages for which there is a SQLite interface. The benefits of SQLite are only realized through using a programming API as it does not include the tools required to operate as a standalone database system.

It is not a prerequisite to have used a relational database in the past. If SQLite will be the first SQL-based database you have encountered, the book gives a thorough SQL tutorial covering the syntax as understood by SQLite. SQL veterans will still benefit from reviewing the sections that cover features included and omitted by SQLite.


How the Book Is Organized

This book is organized into three parts.

Part IGeneral SQLite Use

  • Chapter 1: Getting Started Gives some background on SQLite and discusses its strengths and weaknesses. Covers a few basic SQL commands to get things going and looks at the interactive interfaces to SQLite.

  • Chapter 2: Working with Data A concise SQLite tutorial introduces working with the SQLite back end. Discusses programming and database design issues related to SQLite.

  • Chapter 3: SQLite Syntax and Use Examines SQL syntax as supported by SQLite and suggests workarounds for the unsupported features.

  • Chapter 4: Query Optimization Discusses performance considerations related to SQL queries and gives some techniques that can be used to speed up your database application.

Part IIUsing SQLite Programming Interfaces

  • Chapter 5: The PHP Interface How to use the SQLite library within PHP scripts to create dynamic, database-driven web pages.

  • Chapter 6: The C/C++ Interface How to write C/C++ programs using the SQLite library.

  • Chapter 7: The Perl Interface How to write Perl scripts using the Database Interface module and SQLite Database Driver.

  • Chapter 8: The Tcl Interface How to write Tcl scripts using the supplied SQLite extension.

  • Chapter 9: The Python Interface How to write Python programs using the PySQLite extension.

Part IIISQLite Administration

  • Chapter 10: General Database Administration Discusses basic administration of SQLite and examines SQLite's internal architecture and the Virtual Database Engine (VDBE).


Versions of Software Covered

At the time of writing, the most recent stable version of SQLite is 2.8.15; however, SQLite 3 is already available as a beta and includes some changes and enhancements over the version 2 series. This book was written with the established, stable, and well-supported version 2 series in mind.

For the other software discussed in this book, the current versions are as follows:

  • PHP 5.0.1

  • Perl 5.8.5

  • Perl::DBI 1.4.3

  • DBD::SQLite2 0.32

  • Tcl 8.4.7

  • Python 2.3.4


Additional Resources

The following are the primary web pages for each of the packages used in this book:

  • SQLite http://www.sqlite.org/

  • PHP http://www.php.net/

  • Perl DBI http://dbi.perl.org/

  • Tcl http://www.tcl.tk/

  • Python http://www.python.org/

For support with any of these technologies the relevant mailing list is a good place to start. Instructions on joining them can be found in the following locations:

  • SQLite http://www.sqlite.org/support.html

  • PHP http://www.php.net/mailing-lists.php

  • Perl DBI http://dbi.perl.org/support/

  • Tcl http://wiki.tcl.tk/1301

  • Python http://www.python.org/community/lists.html


Part I: General SQLite Use

1 Getting Started

2 Working with Data

3 SQLite Syntax and Use

4 Query Optimization


Chapter 1. Getting Started

Welcome to SQLite. This chapter will give you an overview of what SQLite is and isn't, what it can and can't do, and how it compares to other databases that you might consider using or might already be familiar with.

We will look at a few basic SQL commands and see how to use the sqlite command-line tool and the SQLite Database Browser GUI to create, examine, and modify databases.


Introduction

SQLite is an embeddable SQL-driven database engine that implements both the database engine and its interface as a C/C++ library. Started in 2000 by D. Richard Hipp, it was written from the ground up and contains absolutely no legacy code, and the SQLite source code has been in the public domain since the first prerelease of version 2.0 in 2001.

The primary design goals when SQLite was conceived were that it should be

  • Simple to administer

  • Simple to operate

  • Simple to use in a program

  • Simple to maintain and customize

The fact that SQLite is small, fast, and reliablearguably its greatest strengthsis, according to Hipp, a happy coincidence. He concentrated on making SQLite simple, and reliability is a byproduct of having fewer things to go wrong. Having simpler code in the database engine makes it much easier to optimize.

Note

The acronym SQL is sometimes pronounced sequel, although in common usage it is most often said as three letters. SQLite, however, is pronounced sequel-lite by its creatorin the same way that Microsoft SQL Server is usually pronounced sequel-serverand therefore that is how we have assumed it is said in this book. As we will refer to a SQLite database and an SQL statement, it will help if you are used to hearing them this way as you read on.

SQLite is already widely used in many projects, and its popularity looks set to continue growing. At the time of writing, version 3 of SQLite is close to completion and a stable release is expected to be available by the end of 2004.

Note

Some of the projects that use SQLite are catalogued at http://www.sqlite.org/cvstrac/wiki?p=SqliteUsers, and you are encouraged to add details of your own projects here too.

Because SQLite 3 has a new API and a new database file format, this book deals primarily with version 2 because it is already available, stable, and well supported. The new features of SQLite 3 are covered in Appendix I, "The Future of SQLite."


Features and Limitations

In this section we will look at the key features of SQLite and some of its limitations. The nature of SQLite makes it an ideal choice for quite a number of tasks, but it's not suitable for everything.

It is important to decide whether SQLiteor any other database engine for that matteris the right choice for your application before committing to a particular technology.

Speed

SQLite is extremely efficient, benefiting from a highly optimized internal architecture and a small memory footprint. Because SQLite is not a client/server database, the overheads of running a database daemon and socket communication are eliminated.

The published speed comparison at http://www.sqlite.org/speed.html compares SQLite to both MySQL and PostgreSQL. It finds that SQLite can perform up to 20 times faster than PostgreSQL and more than twice as fast as MySQL for common operations.

These tests were performed with default installations of each database, and although it is possible to tune the MySQL and PostgreSQL servers for slightly better performance in a given environment, SQLite does not require any such optimization.

The tests found that SQLite is significantly slower than the other databases only on the operations to create an index and to drop a table. However, slowness in these areas will not affect performance on a production database.

Portability

Because SQLite databases are stored as single files on the filesystem, they are very portable indeed. A database can be copied from one file to another, even across different operating systems. This means that for a cross-platform distribution you just need to concentrate on making your code portableeven when a populated database is to be shipped with the application.

SQLite has no external dependencies. The SQLite library is self-contained, so the only system requirement to run an application with an embedded SQLite database is the SQLite library itself. Because SQLite can be freely distributed, you can always ensure that this is present.

Security

SQLite databases are stored to the filesystem and access control is performed by the underlying operating system based on that file's permission settings.

Though SQLite can be accessed by processes running as different users if the correct file permissions are set, the database engine does not detect which user is performing a particular operation.

The advantage of this is one of administrative simplicitythere is no need to set up a complex user grants scheme. Any user who has access to read the database file is able to access the database tables and records. Likewise, in a shared environment, users are able to create their own SQLite databases to their file space without any involvement from the system administrators.

The disadvantage comes when you want to control permissions at a more finely grained level. There is no GRANT operation that would allow access to particular tables to one set of users but not others. If users have read access, they are able to read the entire database, and if they have write access, you have to be sure of their competence and trustworthiness with the data!

SQL Implementation

SQLite supports a large subset of the ANSI SQL-92 standard. Some features have a limited implementation and a few features are not supported at all.

For example, atomic transactions are available but cannot be nested; simple subqueries are possible but correlated subqueries are not; triggers can fire for each row but not for each statement; views are available but are read-only.

The list of current limitations is maintained at http://www.sqlite.org/omitted.html with the items at the top of the list indicating which items are most likely to be added to SQLite first.

In the vast majority of cases, none of the limitations of SQLite will cause problems when developing your application. For those that you need to work around, the benefits of using a fast, portable embedded database will almost certainly outweigh the cost of the workaround.

Customization

The SQLite library includes a very powerful mechanism for adding user-defined functions to the SQL command set. Custom functions can even be written in many of the supported language APIs, not just C/C++.

Both simple functions and aggregating functions can be added this way and, as the hard work is done in the application code, there is no need to rebuild the SQLite library to add new functionality.

Additionally, as the SQLite source code is public domain, you are free to examine and modify it as you see fit. If SQLite is missing a feature that you absolutely must have, why not add it yourself and give something back to the community?

Supported APIs

SQLite now has extensive support for other programming languages through APIs that use the underlying C/C++ interface to communicate with SQLite database files.

C/C++

The core interface is implemented as a single library called libsqlite.so on Linux/Unix systems and sqlite.dll on Windows.

Only the SQLite library is required to allow all users to create their own databases, so very little administration is required to add SQL database capability to a shared system.

PHP

Support for SQLite in PHP has been available for a while, but since the release of PHP 5, it has been shipped with the standard distribution.

Traditionally, PHP and MySQL have gone hand in hand as the interface and back end for dynamic web sites, but it is expected that many more web hosting providers will offer SQLite as PHP 5 gains popularity. From the host's point of view, it is much simpler to administer than a client/server database as there are no complex permissions to manage and database files will be already counted in the disk quota.

Note

SQLite has proved itself to work very well with low- to medium-traffic web sites. As a very rough guideline, if you are expecting over 100,000 hits per dayand in practice, only a small number of web sites will have this level of trafficyou should consider how much database work is done by the web scripts and think about doing some kind of stress testing before committing to SQLite as your back end.

Perl

Perl allows access to SQLite through the Database Interface (DBI) module. This makes it very easy for existing Perl developers to use SQLite within their scripts. The DBI provides an abstraction layer to the Database Driver (DBD), so the same command set is used to access many different types of underlying databases.

The DBD::SQLite driver further allows access to the capabilities of SQLite that the Perl DBI does not allow. Although this does not create a Perl application that can be easily ported to another database back end, it does allow access to the powerful user-defined functions feature.

Tcl

The Tcl interface for SQLite is shipped as part of the SQLite distribution as a library that is imported into Tcl to activate the extensions.

Using Tcl and Tk together with SQLite, you have a platform that is ideal for rapid development of portable graphical user applications.

Other Programming Languages

SQLite has APIs for many other programming languages, including Java, .NET, Smalltalk, and Ruby. As more languages become supported, they are added to the list at http://www.sqlite.org/cvstrac/wiki?p=SqliteWrappers.

Scalability

The downside of using a single file to store databases is that SQLite is not as scalable as many client/server database systems.

The SQLite engine can address database files of up to 2TB in size; however, the restriction on the size of a database is more likely to be enforced by your operating system. In many cases, the size limit on a single file is 2GB.

File locking in SQLite is very coarse-grained. When a write operation takes place, the entire file is locked so that no other process can open it for reading or writing. Larger RDBMSs implement locking at the table or row level so that other processes are able to carry on working unless they are trying to access a specific piece of locked data.

Therefore, if you have a database that is likely to involve massive database files or a high frequency of slow write operations, SQLite may not be suitable and you should consider an RDBMS that is designed and tuned for multiple-user access.


When Not to Choose SQLite

There is no hard-and-fast rule that determines whether SQLite is the right choice for your application. It is a robust, fast database engine that implements as much SQL as you are likely to need, but in some situations there are probably better choices.

In this section we will look at some circumstances that require features that SQLite cannot provide, followed by some examples of situations where SQLite is a very good choice indeed.

When SQLite Is Probably Wrong

First the bad news: The following situations may be better handled by another database system.

Network or Client/Server Applications

SQLite is not well suited for multiuser access over a network. SQLite can read from and write to its databases on a network share, but performance will take a hit because of the high latency levels that are usually found on a network filesystem.

File locking can often be patchy across a network, which could lead to two SQLite processes writing to the database at the same time, inevitably causing corruption.

A client/server RDBMS avoids these issues because all filesystem access is performed by the server after its requests have been received using a network protocol.

High-Volume Websites

For the vast majority of websites SQLite is a good choice. However, some sites receive so much traffic that their database components would be better suited to a client/server RDBMS.

How to quantify high traffic volume is the million-dollar question. The traffic level itself is not as much of an issue as the number of database reads and, more importantly, writes that are performed when a typical visitor does some surfing around.

Because web servers can send multiple requests simultaneously, if the database is frequently locked these processes will often be hanging around waiting for each other to finish writing before they can do their own thing.

The last thing your busy website needs is a database bottleneck, so you should consider a client/server RDBMS with more finely grained database locking.

High Concurrency

Similar to the reasons that a high-volume website may not be suitable for SQLite, a multiprocess or multithreaded application that performs a large number of database accesses may run into file-locking issues that could be avoided in an RDBMS that implements locking on smaller subsets of the database.

Again, quantifying a high level of concurrency is difficult, but this is something you should consider if your application will attempt to perform many concurrent database operations.

Each lock on the database file may only last for a few milliseconds, but there are still times where this will be too long for the application to work efficiently.

When SQLite Is Probably Right

Now for the good news: There are many applications for which SQLite is a great choice.

Websites

We have already mentioned that SQLite is usually a great choice for database-driven websites. With the inclusion of SQLite support in PHP 5, it is likely to challenge MySQL as the de facto database back end for PHP scripts.

Embedded Devices

Being small and requiring virtually no administration makes SQLite ideal for hardware applications that require database storage without the need for human intervention. In addition to working on Embedded Linux and Windows CE, SQLite has been also ported to Palm OS.

Ad-hoc File Storage

If a program writes data to a file on disk, it can write it to a SQLite database just as easily. If you need to write application data to disk, why not use an SQL database rather than devising a custom file format? SQLite has a further advantage over, for instance, XML data files in that the whole file does not have to be read into memory before specific elements can be readSQLite provides fast random access to the data.

Internal Data Manipulation

When an application has to perform operations on data held in internal data structures, it can be easier, and often quicker, to load that data into an in-memory SQLite database. Then it can be manipulated using SQL commands and returned to the program.


Looking at SQLite Databases

As we have seen, SQLite provides APIs to allow a database to be embedded into a wide variety of programming languages. However, there are times when you want to take a look inside your database without writing a program to do so. This section introduces two tools that can be used to query and amend SQLite database files.

The sqlite Tool

SQLite comes bundled with the sqlite tool, which provides a command-line interface to work with database files. It is invoked from the shell taking simply the name of a database as its argument.

$ sqlite dbfile
SQLite version 2.8.15
Enter ".help" for instructions
sqlite>

The sqlite> prompt indicates that sqlite is waiting for a new command to be entered. There are two types of commandseither an SQL statement terminated with a semicolon or a command relating to the sqlite program itself, beginning with a dot.

The .help command lists all the dot commands available along with a brief description, as shown in the following:

sqlite> .help
.databases             List names and files of attached databases
.dump ?TABLE? ...      Dump the database in a text format
.echo ON|OFF           Turn command echo on or off
.exit                  Exit this program
.explain ON|OFF        Turn output mode suitable for EXPLAIN on or off.
.header(s) ON|OFF      Turn display of headers on or off
.help                  Show this message
.indices TABLE         Show names of all indices on TABLE
.mode MODE             Set mode to one of "line(s)", "column(s)",
                       "insert", "list", or "html"
.mode insert TABLE     Generate SQL insert statements for TABLE
.nullvalue STRING      Print STRING instead of nothing for NULL data
.output FILENAME       Send output to FILENAME
.output stdout         Send output to the screen
.prompt MAIN CONTINUE  Replace the standard prompts
.quit                  Exit this program
.read FILENAME         Execute SQL in FILENAME
.schema ?TABLE?        Show the CREATE statements
.separator STRING      Change separator string for "list" mode
.show                  Show the current values for various settings
.tables ?PATTERN?      List names of tables matching a pattern
.timeout MS            Try opening locked tables for MS milliseconds
.width NUM NUM ...     Set column widths for "column" mode

Let's issue a few simple SQL commands to create a new table and add some rows of data. Don't worry too much about how these commands work for now; we just need to get something into the database to show how sqlite works.

First of all, the following statement creates a new table called mytable that has two columns, col1 and col2.

sqlite> CREATE TABLE mytable (
   ...>   col1 NUMERIC,
   ...>   col2 TEXT
   ...> );
sqlite>

Notice how the prompt changes from sqlite> to three dots, indicating that the entered statement is not yet complete. The semicolon character signifies the end of an SQL statement, and after the semicolon is entered, the prompt changes back.

Note

If you get into a mess entering a statement and want to abort without having SQLite process a nonsense command that could potentially do harm, press Ctrl+C to send an interrupt signal to sqlite. You will be returned to the shell prompt and can start again.

Although no output is displayed to screen, SQLite has received and processed the CREATE TABLE command. The response from database operations is always silent unless there has been an error.

If you type the same command with a deliberate mistake, the error will be detected when the semicolon terminator is enteredbut not beforeas shown in the following example:

sqlite> CREATTE TABLE mytable (
   ...>   col1 NUMERIC,
   ...>   col2 TEXT
   ...> );
SQL error: near "CREATTE": syntax error

The .schema command can be used to see the schema of a table. What is displayed to screen is the CREATE TABLE command that was used to create the table. Therefore if you formatted or capitalized the statement differently than that shown, the following output will be slightly different.

sqlite> .schema mytable
CREATE TABLE mytable (
  col1 NUMERIC,
  col2 TEXT
);

If .schema is used without a table argument, the schema for all tables in the database is displayed.

The following commands insert a few simple records into the database. Again, the only response after entering each one is a new sqlite> prompt, unless an error occurs.

sqlite> INSERT INTO mytable VALUES (1, 'One');
sqlite> INSERT INTO mytable VALUES (2, 'Two');
sqlite> INSERT INTO mytable VALUES (3, 'Three');
sqlite> INSERT INTO mytable VALUES (99, 'Ninety Nine');

The SELECT command in SQL is used to retrieve rows from the database. Let's bring back the newly inserted rows to see how this is displayed in sqlite.

sqlite> SELECT col1, col2 FROM mytable;
1|One
2|Two
3|Three
99|Ninety Nine

The sqlite tool has various different output formats for the selected data. The default is list mode which, as shown, outputs one row for each database record with the columns separated by a | character.

To change the separator character, use the .separator command. The following example changes the separator to a comma. You could use this output format to create a comma-separated values file to load into another application, for instance.

sqlite> .separator ,
sqlite> SELECT col1, col2 FROM mytable;
1,One
2,Two
3,Three
99,Ninety Nine

Output modes are changed using the .mode command followed by the name of the desired mode. The column mode also outputs one row for each database record, but uses a formatted column output.

sqlite> .mode column
sqlite> SELECT col1, col2 FROM mytable;
1           One
2           Two
3           Three
99          Ninety Nin

By default each column width is 10 characters and, as shown, any values longer than the column width are truncated to fit. The .width command can be used to adjust this. It takes a series of number arguments with the first number representing the width of the first column and so on.

Column headings in this mode can be turned on or off using the .header command followed by on or off. The following example turns headings on and sets the columns widths just wide enough to display all the data and column headings.

sqlite> .mode column
sqlite> .header on
sqlite> .width 4 11
sqlite> SELECT col1, col2 FROM mytable;
col1  col2
----  ------------
1     One
2     Two
3     Three
99    Ninety Nine

On systems with the readline library installed, you can re-execute a command that has previously been entered using the up and down arrow keys to scroll through the command history a line at a time. In the preceding example it was possible to resubmit the SELECT statement after entering the two formatting commands by pressing the Up arrow key three times followed by pressing Enter.

Rather than one line per record, the line output mode causes one row to be output for each selected column with the column name prefixed.

sqlite> .mode line
sqlite> SELECT col1, col2 FROM mytable;
 col1 = 1
 col2 = One

 col1 = 2
 col2 = Two

 col1 = 3
 col2 = Three

 col1 = 99
 col2 = Ninety Nine

Setting output mode to html causes a SELECT statement to generate HTML code for a table that contains the fetched data. The .header setting determines whether column headings are included in <TH> tags.

sqlite> .mode html
sqlite> .header on
sqlite> SELECT col1, col2 FROM mytable;
<TR><TH>col1</TH><TH>col2</TH></TR>
<TR><TD>1</TD>
<TD>One</TD>
</TR>
<TR><TD>2</TD>
<TD>Two</TD>
</TR>
<TR><TD>3</TD>
<TD>Three</TD>
</TR>
<TR><TD>99</TD>
<TD>Ninety Nine</TD>
</TR>

The final output mode is insert, which causes a series of SQL INSERT statements to be generated. .mode insert optionally takes a database table argument, which will generate INSERT statements for the given table name rather than the one from which it was selected.

sqlite> .mode insert newtable
sqlite> SELECT col1, col2 FROM mytable;
INSERT INTO newtable VALUES(1,'One');
INSERT INTO newtable VALUES(2,'Two');
INSERT INTO newtable VALUES(3,'Three');
INSERT INTO newtable VALUES(99,'Ninety Nine');

If you want to check what the current display settings are, use the .show command. The following output shows the current settings if you have followed all the examples in this chapter so far.

sqlite> .show
     echo: off
  explain: off
  headers: on
     mode: insert
nullvalue:
   output: stdout
separator: ,
    width: 4 12

The sqlite tool can process both SQL statements and dot commands from standard input. Therefore it is possible to pipe or redirect a series of commands to the program rather than key them in.

If you have a script containing a series of SQL commands, for example to create and populate a database, the following command can be used:

$ sqlite dbfile < sqlfile

Alternatively, sqlite uses the .read command to interactively read and process a file containing SQL commands.

sqlite> .read sqlfile

With those basics of sqlite under your belt, you should be all set to start working with SQLite. We will use sqlite throughout this book and some of the more advanced dot commands will be introduced later on.

SQLite Database Browser

If you prefer to manage your databases through a graphical interface, take a look at SQLite Database Browser. This handy open-source application can be built on any platform supported by Trolltech's QT library, and precompiled versions are available for Linux, Windows, and OSX. You can download SQLite Database Browser from http://sourceforge.net/projects/sqlitebrowser/.

When SQLite Database Browser starts up, you will be presented with an empty-looking application window. Let's load in the sample database that we'll use in Chapter 2, "Working with Data." Download demodb.sql from the Sams Publishing website at http://www.samspublishing.com/. Enter this book's ISBN (without the hyphens) in the Search box and click Search. When the book's title is displayed, click the title to go to a page where you can download the code and save it locally. Then select File, Import, Database from SQL File, as shown in Figure 1.1.

Figure 1.1. Using SQLite Database Browser to import an SQL file.

[View full size image]

A file selector will appear. Select demodb.sql from the location you saved it to. SQLite Database Browser will ask if you want to create a new database to store the imported data. Answer Yes and another file selector will appear where you can choose the location of your database file. No file extension is necessary, so simply call the file demodb.

The program will then go ahead and load data from the SQL file to create a new database. Five table names will be shown under the Database Structure tab, and by clicking the + symbols to the left of each one you can expand the view to show the columns in each table.

Note

SQLite Database Browser can also load data from a CSV file into a SQLite database. Select File, Import, Table from CSV File.

Figure 1.2 shows the contents of demodb fully expanded. Notice that the Object column indicates the type of each item, and the schema is shown alongside each table object. Unfortunately where the CREATE TABLE statement was formatted across multiple lines, the schema cannot fit all the information into a single row of the display and sometimes this looks messy. Expanding the table to display its columns gives as much information as you would need anyway.

Figure 1.2. Viewing table columns through SQLite Database Browser.

[View full size image]

The icon bar at the top of the window allows you to create and drop both tables and indexeshover over each icon in turn and the bubble help will tell you what each one does. These options are also available from the Edit menu. SQLite Database Browser also provides a Modify Table option, which can be very useful as SQLite does not implement the ALTER TABLE database command.

Note

In the current version of SQLite Database Browser, there is an intermittent problem with the import process. Sometimes after you load a database from an SQL or CSV file, the icon bar and menu options to manipulate tables are disabled. If this is the case, you need to close the program and reopen your database filethere is no need to reimport it.

To add a new table, click the appropriate icon or select Edit, Create Table. Enter a table name and use the Add button to add columns to it. The dialog box will be similar to Figure 1.3.

Figure 1.3. Adding a new table using SQLite Database Browser.

The field type selection has only three valuestext, numeric, and blob. As we'll see in Chapter 2, SQLite is typeless, so there is not the range of data types you may be used to having in other database systems.

The Browse Data tab enables you to look at the actual records held in the database. Use the Table drop-down list to select a table and its records will be shown in a grid format below.

Figure 1.4 shows the records in the employees table viewed through SQLite Database Browser.

Figure 1.4. Viewing database records using SQLite Database Browser.

[View full size image]

If there are a large number of rows in the table, the first thousand will be displayed with a scrollbar on the right of the window. The buttons in the bottom left corner of the window allow you to flip forward and backward a thousand at a time.

SQLite Database Browser also allows you to narrow the data set displayed using a search option. Click the magnifying glass icon to bring up the Find screen. From here you can select a column to search on, pick an operator, and enter the search value.

For example, to find all male employees, select the employees table under the Browse Data tab and click the magnifying glass. In the Find window, select the sex column and the equals (=) operator and enter M as the value. Click the Search button and two rows will be displayed.

Only the record number and the selected column are displayed in this view, and as you searched on sex=M, showing that the value of the sex column is M for both of the records isn't much use. However, you can still use the Browse Data view with the Find window active, and clicking an entry in the Find window will automatically highlight the corresponding row in the main application window, as shown in Figure 1.5.

Figure 1.5. Performing a search in SQLite Database browser.

[View full size image]

Through the Execute SQL tab you can enter any SQL command to have it processed by SQLite. The application displays any error messages, allowing you to modify the statement until it is error-free. Any rows returned by a SELECT statement are displayed in the bottom half of the screen.

Finally, to make sure any changes you have made through SQLite Database Browser are saved to the database file, select File, Save Database. Changes are not saved automatically, although if you exit the program with unsaved changes you will be asked if you want to save them.


Help and Support

Before we get into using SQLite in the following chapters, let's take a look at the support available to you if you run into problems.

There is some very good online documentation at http://www.sqlite.org/docs.html, which is updated regularly when new features are added to SQLite.

If you need something more interactive, the SQLite mailing list is a good place to ask questions. Send a blank email to sqlite-users-subscribe@sqlite.org to join the list, or if you'd rather receive a daily digest of messages, send it to sqlite-users-digest-subscribe@sqlite.org. The mailing list is frequented by the author and many regular users of SQLite, so you should find someone who is able to give good advice.

If professional support for your database engine is a requirement, this is available from Hipp, Wyrick & Company, Inc. (known as Hwaci for short) for a fee. Support is provided by the author and maintainer of SQLite, D. Richard Hipp, so you know you will be talking to an authority when your moment of need arises. More information can be found at http://www.hwaci.com/sw/sqlite/prosupport.html.


Chapter 2. Working with Data

We begin this chapter by getting our hands dirty right away with a basic tutorial that will give you an overview of how to work with data in SQLite.


SQLite Basics

If you have used SQL with other database systems, the language elements will be familiar to you, but it is still worth following the tutorial to see some of the differences between SQLite's implementation of SQL and the version you are used to.

If you are new to SQL, you will begin to pick up the basics of the language by following the examples, and we will go into more depth on each topic in the following chapters.

Prerequisites

To follow the examples in this tutorial you will need access to a workstation or space on a server system on which you can create a SQLite database.

You will also need the sqlite command-line tool to be installed and in your path. Full installation instructions can be found in Appendix A, "Downloading and Installing SQLite," but for now the quickest way to get started is to use one of the precompiled sqlite binaries from http://www.hwaci.com/sw/sqlite/download.html.

Remember, SQLite writes its databases to the filesystem and does not require a database server to be running. The single executable sqlite (or sqlite.exe on Windows systems) is all that you need.

Obtaining the Sample Database

All of the code in this book is available for download on the Sams Publishing website at http://www.samspublishing.com/. Enter this book's ISBN (without the hyphens) in the Search box and click Search. When the book's title is displayed, click the title to go to a page where you can download the code to save you from retyping all the CREATE TABLE commands that follow. However, you don't have to download the sample database as we have included the full set of commands required for the tutorial in this book.

Creating and Connecting to a Database

SQLite stores its databases in files, and you should specify the filename or path to the file when the sqlite utility is invoked. If the filename given does not exist, a new one is created; otherwise, you will connect to the database that you specified as the filename.

$ sqlite demodb
SQLite Version 2.8.12
Enter ".help" for instructions
sqlite>

Even without executing any SQLite commands, without any tables being created, connecting to a database as indicated in the preceding example will create an empty database file with the name specified on the command line.

$ ls l
-rw-r--r--    1 chris    chris           0 Apr  1 12:00 demodb

No file extension is required or assigned by sqlite; the database file is created with the exact name specified. So how do we know that this is a SQLite database? You could opt for a consistent file extension, for example using somename.db (as long as .db is a unique extension for your system) or use a well-organized directory structure to separate your data from your program files.

When there is something inside your database, you can identify the file as a SQLite database by taking a look at the first few bytes using a binary-safe paging program such as less. The first line you will see will look like this:

** This file contains an SQLite 2.1 database **

We'll examine the format of the database file in more detail later in this chapter, but for now there's nothing more you need to know about creating and connecting to databases in SQLite. The CREATE DATABASE statement that you may be used to is not used.

Executing SQL from a File

The sample database is defined by a series of CREATE TABLE statements in a file called demodb.sql and there are some sample data records in sampledata.sql.

To execute the SQL in these files, use the .read command in sqlite.

$ sqlite demodb
SQLite version 2.8.12
Enter ".help" for instructions
sqlite> .read demodb.sql

Nothing will be displayed on screen unless there is an error in the SQL filesilence means that the commands were executed successfully.

It is also possible to execute an SQL file by redirecting it into sqlite from the command line, for instance:

$ sqlite demodb < sampledata.sql

As before, there will only be any output to screen if there are errors in the SQL file.

The Sample Database

The sample database used in this tutorial is a very simple time-tracking system that might be used by a company to log employees' hours against client projects. There are five tables in the system, with the schema illustrated in Figure 2.1.

Figure 2.1. Schema of the sample database.

The tables holding employee and client details are very simple for this tutorial. In a real system the number of fields in these tables would be much higher, but for simplicity we've limited the employees and clients tables to just the basics.

CREATE TABLE employees (
  id INTEGER PRIMARY KEY,
  first_name CHAR NOT NULL,
  last_name CHAR NOT NULL,
  sex CHAR NOT NULL,
  email CHAR NOT NULL
);
CREATE TABLE clients (
  id INTEGER PRIMARY KEY,
  company_name CHAR,
  contact_name CHAR,
  telephone CHAR
);

Each table has an id field specified as INTEGER PRIMARY KEY, which we'll look at in more detail shortly. In a nutshell, this field is used to autogenerate a unique id number for each record added so that we don't have to.

Suppose that the company is working on, or has undertaken in the past, more than one job for each client and wants to keep the time spent on each project separate. To enable them to associate the hours on a timesheet with a particular project, we will create a projects table that looks like this:

CREATE TABLE projects (
  code CHAR PRIMARY KEY,
  client_id INTEGER NOT NULL,
  title CHAR NOT NULL,
  start_date INTEGER,
  due_date INTEGER
);

This time, rather than having SQLite assign our primary key, we've specified it as CHAR so that the company would assign its own project codes and employees would know the textual codes of the projects they are working on.

There is no direct relationship between employees and projects or clientswe have assumed that it is possible that any member of the workforce could be working on any of the current projects. Our timesheets table creates a link between projects and employees when they have done some work and records the date and how long they spent on that project.

CREATE TABLE timesheets (
  id INTEGER PRIMARY KEY,
  employee_id INTEGER NOT NULL,
  project_code CHAR NOT NULL,
  date_worked INTEGER NOT NULL,
  hours DECIMAL(5,2) NOT NULL
);

The final table, employee_rates, stores data on how much employees get paid. Rather than a single attribute in the employees table, this system allows us to remember the previous rates that an employee was paid before any raises were given. If we wanted to recalculate the cost of a project from some time ago, we would want to use the rates of pay in effect at the time and not the present ones.

CREATE TABLE employee_rates (
  employee_id INTEGER NOT NULL,
  rate DECIMAL(5,2),
  start_date INTEGER NOT NULL,
  end_date INTEGER
);

Typeless Data Types

The way in which SQLite stores data is built upon the realization that strong typing of data, as found in virtually every RDBMS on the market, is actually not a useful thing. A database is designed to store and retrieve data, plain and simple, and as such the developer should not have to declare a column as numeric, textual, or binary, or tie it to any other specific data type. Needing to do so is a legacy weakness of the underlying system that had to be reflected in SQL, not a feature added to the language.

SQLite is "typeless." It makes no difference whether you put a text string into an integer column or try to shove binary data into a text type. In fact you canand shouldspecify the column types when each table is created. SQLite's SQL implementation allows you to do this, but it's actually ignored by the engine.

That said, it's still a good idea to include the column data types in your CREATE TABLE statements to help you think through the database design thoroughly and as a reminder to yourself and a hint to other programmers as to what your intended use of each column was. Suppose, also, that in the future you want to migrate or mirror your data onto an RDBMS that does require column typingyou'll be glad your table definitions are well documented.

You can always retrieve the CREATE TABLE statement used to create a table using the .schema command from within sqlite:

sqlite> .schema employees
CREATE TABLE employees (
  id INTEGER PRIMARY KEY,
  first_name CHAR NOT NULL,
  last_name CHAR NOT NULL,
  sex CHAR NOT NULL,
  email CHAR NOT NULL
);

In the demo database, all the data types were declared using familiar data type labels, but you may have noticed that we did not specify any lengths for the CHAR types, which would usually be required.

In fact the syntax of SQLite's CREATE TABLE statement specifies the column type attribute as optional. A valid column type is defined as any sequence of zero or more character strings (other than reserved words) optionally followed by one or two signed integers in parentheses.

Listing 2.1 shows a CREATE TABLE statement with some examples of ANSI SQL data types that are also valid in SQLite.

Listing 2.1. Creating a Table with ANSI SQL Data Types
CREATE TABLE dummy (
  mychar CHAR(6),
  myvarchar VARCHAR(10)
  mydecimal DECIMAL(7,2),
  myinteger INTEGER,
  mytinyint TINYINT,
  myfloat FLOAT,
  mydate DATE
);

Listing 2.2 creates a table identical to the one in Listing 2.1, with imaginary column types to show that SQLite pays no attention to the actual types given.

Listing 2.2. Creating a Table with Made-Up Data Types
CREATE TABLE dummy (
  mychar CHARACTER(6),
  myvarchar VARIABLE LENGTH CHARACTER
  mydecimal DECIMAL NUMBER(7,2),
  myinteger INT,
  mytinyint SMALL NUMBER,
  myfloat FLOATING POINT NUMBER,
  mydate CALENDAR DATE
);

Finally, Listing 2.3 shows that the same table can actually be created, if you are in a real hurry, without specifying any column types whatsoever.

Listing 2.3. Creating a Table Without Specifying Data Types
CREATE TABLE dummy (
  mychar,
  myvarchar,
  mydecimal,
  myinteger,
  mytinyint,
  myfloat,
  mydate
);

The INTEGER PRIMARY KEY

You saw in the preceding section that the words that make up the name of a data type in the CREATE TABLE statement have no bearing on the data that can be stored in the table. There is actually one exceptionif the column is defined exactly as INTEGER PRIMARY KEY, the value in the column must be a 32-bit signed integer; otherwise, an insert instruction will fail.

Note

INTEGER PRIMARY KEY must be used exactly as written here. If it's abbreviated, for instance INT PRIMARY KEY, the column will be created typeless.

So why, with the decision that data typing was a bad idea, has SQLite implemented one column type that's different? The reason for an INTEGER PRIMARY KEY is to allow for the equivalent of an AUTOINCREMENT or IDENTITY column.

You can insert any 32-bit signed integer value into an INTEGER PRIMARY KEY column, or insert a NULL to tell SQLite to assign the value itself. Inserting a NULL will cause a value of one greater than the largest value already stored in that column to be used.

If the largest key value is already the maximum value that the column type allows2^311, or 2147483647then it's not possible to use a greater value for a new key. In this situation, SQLite picks a new key randomly.

We have used an INTEGER PRIMARY KEY called id on the employees, clients, and timesheets tables so that as new records are added, we do not need to assign this value ourselves.

Working with Dates

In our sample tables, we have declared the date fields as INTEGER. Though SQLite has included some date and time functions since version 2.8.7, the typeless nature of SQLite does not provide a native storage format for dates. In our demo database tables, using an integer representation of the date serves our purposes well, for instance timesheets.date_worked is declared as INTEGER.

The SQLite date functions are based on a string date format, and we will examine them in detail in the next chapter.

Of course, SQLite will not enforce that the value stored in our (typeless) date columns must be an integer, but the use of INTEGER in the CREATE TABLE statement shows our intention for this field.

Using an integer date allows us to use one of two common formats. The UNIX timestamp format is the number of seconds since midnight on January 1, 1970 (known as the epoch), and a signed 32-bit integer is capable of storing dates up to 2037.

Some date calculations are made very easy using the timestamp format; for instance, to add one day to a value you just need to add 86400the number of seconds in a dayto it. Most programming languages that SQLite can be embedded in work with this date format, and the SQLite date functions are also able to convert to and from a timestamp.

A more readable integer format is to present a date as YYYYMMDD (or YYYYMMDDHHIISS if the time is also required). It's much easier to see that 20040622 is June 22, 2004, than its equivalent timestamp value of 1322222400.

Although the YYYYMMDD format does not make date arithmetic possible without splitting up the individual components or converting to some other format, it can still be used for comparisons because an earlier date will always be represented as a lower integer value than a later date.

For our sample system, we are storing the dates that an employee worked and a due date for each project, so we might want to find out the hours worked in a particular month or how many projects are overdue, but we don't need to do any calculations and the time element is not required. Therefore, YYYYMMDD is a suitable date format for this system.

Inserting the Sample Data

First of all, we'll add some employees and clients to the system.

INSERT INTO employees (id, first_name, last_name, sex, email)
VALUES (101, 'Alex', 'Gladstone', 'M', 'alex@mycompany.com');

INSERT INTO employees (id, first_name, last_name, sex, email)
VALUES (102, 'Brenda', 'Dudson', 'F', 'brenda@mycompany.com');

INSERT INTO employees (id, first_name, last_name, sex, email)
VALUES (103, 'Colin', 'Aynsley', 'M', 'colin@mycompany.com');

INSERT INTO employees (id, first_name, last_name, sex, email)
VALUES (104, 'Debbie', 'Churchill', 'F', 'peter@mycompany.com');

INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (501, 'Acme Products', 'Mr R. Runner', '555-6800');

INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (502, 'ABC Enterprises', 'Mr T. Boss', '555-2999');

INSERT INTO clients (id, company_name, contact_name, telephone)
VALUES (503, 'Premier Things Ltd', 'Mr U. First', '555-4001');

Note

The values in the primary key columns arbitrarily begin at 101 for employees and 501 for clients. Using different sequences will make it easier to show whether an id number refers to an employee or a client when we are selecting data in this tutorial. It doesn't actually create a problem if the primary key values of different tables overlap.

Figures 2.2 and 2.3 show the contents of the employees and clients tables after this data has been loaded.

Figure 2.2. Contents of the employees table.

Figure 2.3. Contents of the clients table.

The sample data we have loaded specifies each id value explicitly so that, for instance, we can be sure that projects belong to the right client. However, if we add a new client to the system using the following statement, it will be given a new id of the next highest available number:

sqlite> INSERT INTO clients (company_name, contact_name, telephone)
   ...> VALUES ('Integer Primary Key Corp', 'Mr A. Increment', '555-1234');

We don't have to go trawling through the data to find out what value SQLite assigned for the new primary key. The function last_insert_rowid() will return the integer value that was generated.

sqlite> select last_insert_rowid();
504

Next we'll insert details of the clients' projects. The due_date field is optional, and in the case of the second item in the following code being an ongoing project, we give it a NULL value.

INSERT INTO projects (code, client_id, title, start_date, due_date)
VALUES ('ACME01', 501, 'Ordering system', 20030401, 20031231);

INSERT INTO projects (code, client_id, title, start_date, due_date)
VALUES ('ABCCONS', 502, 'Ongoing consultancy', 20030601, NULL);

INSERT INTO projects (code, client_id, title, start_date, due_date)
VALUES ('PREM3K', 503, 'Thing 3000', 20031201, 20040430);

INSERT INTO projects (code, client_id, title, start_date, due_date)
VALUES ('PREM4K', 503, 'Thing 4000', 20031201, 20040731);

INSERT INTO projects (code, client_id, title, start_date, due_date)
VALUES ('PREM5K', 503, 'Thing 5000', 20031201, 20041031);

Figure 2.4 shows the contents of the projects table after these INSERT statements have been executed. Notice that the values in the client_id column correspond to records in the clients table, shown in Figure 2.3.

Figure 2.4. Contents of the projects table.

The following statements insert a rate of pay for each employee. The start_date is the date they joined the company and in most cases that rate is still in effect. We indicate that a rate is current by the end_date field being NULL.

Debbie Churchill, employee id 104, was given a raise, so there are two entries. Her hours worked before January 1, 2004, were paid at $20; from this date onwards she will be paid $25.

INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (101, 30.00, 20030401, NULL);

INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (102, 15.00, 20020601, NULL);

INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (103, 25.00, 20011001, NULL);

INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (104, 20.00, 20010401, 20031231);

INSERT INTO employee_rates (employee_id, rate, start_date, end_date)
VALUES (104, 25.00, 20040101, NULL);

Figure 2.5 shows the records inserted into the employee_rates table by the preceding statements. The employee_id column references records from the employees tablethe values in this column correspond to the id column in Figure 2.2.

Figure 2.5. Contents of the employee_rates table.

Finally we have some timesheet information. This is only a small sample of information that might be in such a system where new data is added every day, but should be sufficient for demonstration purposes.

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'ACME01', 20031229, 4);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'ABCCONS', 20031229, 2);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (101, 'PREM3K', 20040102, 6);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (102, 'ACME01', 20031229, 3);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (102, 'PREM4K', 20040102, 5);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'ABCCONS', 20031229, 2);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'PREM4K', 20040102, 3);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (103, 'ACME01', 20031229, 8);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (104, 'ACME01', 20040102, 8);

INSERT INTO timesheets (employee_id, project_code, date_worked, hours)
VALUES (104, 'ABCCONS', 20040102, 4);

Figure 2.6 shows the contents of the timesheets table after we have inserted this sample data. Notice that this table references two others, employees and clientsreferenced by employee_id and project_code respectively.

Figure 2.6. The timesheets table after it has had the sample data inserted.

BLOBs and Binary Data

When you define a column as BLOB the column is still typeless, though as you saw earlier this does tell SQLite to use a text-based comparison when sorting on the field. SQLite is fully capable of storing binary data in any column, provided that there are no NUL (0x00, ASCII character zero) characters in the data.

As SQLite will store strings in its columns much more often than binary data, a NUL is the terminator for a variable-length character string and must be encoded somehow if it appears inside a piece of binary data.

Encoding can be done at the application level, for instance by encoding to base-64 or with URL-style encoding where NUL becomes its hex value represented as %00, and the percent character itself is encoded to %25.

SQLite also includes a pair of functions to encode and decode binary data, sqlite_encode_binary() and sqlite_decode_binary(). How binary data is encoded for any given column is an implementation choice left to the developer, so you can use supplied functions, adapt them from the sources in src/encode.c, or create your own.


Querying and Updating the Database

So now that we have some records in our database, let's look at ways to fetch, modify, and delete records using SQLite's implementation of SQL.

The SELECT Statement

Use SELECT to fetch rows from a table. Either use a comma-separated list of column names or use an asterisk to fetch all columns in the table.

sqlite> SELECT first_name, email FROM employees;
Alex|alex@mycompany.com
Brenda|brenda@mycompany.com
Colin|colin@mycompany.com
Debbie|peter@mycompany.com

sqlite> SELECT * FROM clients;
501|Acme Products|Mr R. Runner|555-6800
502|ABC Enterprises|Mr T. Boss|555-2999
503|Premier Things Ltd|Mr U. First|555-4001

You'll notice that the format of the output isn't as readable as it could be. The sqlite program has a number of formatting options of which this mode, a pipe-separated list, is the default. The various output modes were discussed in Chapter 1, "Getting Started."

Although character-separated output can be a useful format for parsing by a program, a tabulated output is clearer in print. Therefore, for the rest of this tutorial we will use column mode with headings turned on.

sqlite> .mode column
sqlite> .header on
sqlite> select * from clients;
id          company_name   contact_name  telephone
----------  -------------  ------------  ----------
501         Acme Products  Mr R. Runner  555-6800
502         ABC Enterpris  Mr T. Boss    555-2999
503         Premier Thing  Mr U. First   555-4001
504         Integer Prima  Mr A. Increm  555-1234

The values of company_name and contact_name in the preceding code are truncated. Their full values are stored in the database, but the column display format causes the output to be arbitrarily limited. You can specify the width settings in sqlite with the .width command:

sqlite> .mode column
sqlite> .width 4 24 15 10
sqlite> select * from clients;
id    company_name              contact_name     telephone
----  -----------------------   ---------------  ----------
501   Acme Products             Mr R. Runner     555-6800
502   ABC Enterprises           Mr T. Boss       555-2999
503   Premier Thing             Mr U. First      555-4001
504   Integer Primary Key Ltd   Mr A. Increment  555-1234

For the rest of this tutorial, display widths have been adjusted to suit the output and the actual .width commands issued are not shown in the code.

The WHERE Clause

The WHERE statement specifies a conditional clause that can be used to restrict the dataset that is returned. To find all projects for Premier Things, we restrict the dataset on client_id.

sqlite> SELECT code, title
   ...> FROM projects
   ...> WHERE client_id = 503;
code        title
----------  ----------
PREM3K      Thing 3000
PREM4K      Thing 4000
PREM5K      Thing 5000

The AND and OR operators extend a WHERE condition logically. For instance, to find all projects for this client that were due on or before August 1, 2003, we could do the following:

sqlite> SELECT code, title, due_date
   ...> FROM projects
   ...> WHERE client_id = 503
   ...> AND due_date <= 20040801;
code        title       due_date
----------  ----------  ----------
PREM3K      Thing 3000  20040430
PREM4K      Thing 4000  20040731

You can use the BETWEEN operator to test if a value is in a given range. The following example finds all projects for Premier Things that were due during the month of July 2003.

sqlite> SELECT code, title, due_date
   ...> FROM projects
   ...> WHERE client_id = 503
   ...> AND due_date BETWEEN 20040701 AND 20040731;
code        title       due_date
----------  ----------  ----------
PREM4K      Thing 4000  20040731

You can use the IN operator to test if one value is in another given set of values. To select rows only where the value is not in the set, use NOT IN. The following example finds all projects for two specified clients:

sqlite> SELECT code, title
   ...> FROM projects
   ...> WHERE client_id in (501, 502);
code        title
----------  -------------------
ACME01      Ordering system
ABCCONS     Ongoing consultancy

We will look at the other relational operators that can be used in SQLite in Chapter 3, "SQLite Syntax and Use." A full list can be found in Table 3.1.

String Comparisons

The examples you have seen so far in this chapter all use integer values in the WHERE clause, and although SQLite is typeless, string values must still be enclosed in single quotes. The following are examples of some string comparisons:

sqlite> SELECT contact_name FROM clients
   ...> WHERE company_name = 'Acme Products';
contact_name
-------------
Mr R. Runner

Note that the equals operator for string comparisons is case sensitive. To find a case-insensitive match, use the LIKE operator. To test for inequality in strings, use != or NOT LIKE respectively.

sqlite> SELECT company_name FROM clients
   ...> WHERE company_name LIKE 'acME proDUCTs';
company_name
--------------
Acme Products

The LIKE operator can also be used to specify a substring search. The underscore character is used to represent any character, and a percent sign is a variable-length wildcard.

sqlite> SELECT company_name FROM clients
   ...> WHERE company_name LIKE 'A%';
company_name
---------------
Acme Products
ABC Enterprises

Table 2.1 shows some examples of wildcard matching using LIKE.

Table 2.1. Examples of Wildcard Matching with LIKE

Expression

Will Match

LIKE 'a%'

apple

An apple a day

LIKE 'b_'

be

BC

LIKE 'a%a'

agenda

ABBA

LIKE 'b_b'

Bob

B2B

NULL Values

It is important to remember that NULL is a special value, actually meaning that an item has no value. It is not a zero value or an empty string, and comparing either of these values to NULL will always return false. Because NULL has no value, the equals operator cannot be used and you must test for NULL values using IS NULL.

To check whether a column value is NULL, use the IS NULL operator. The opposite, IS NOT NULL, will match any value including zero and an empty string. The following query finds all projects that are ongoing, that is with no due date:

sqlite> SELECT code, title FROM projects
   ...> WHERE due_date IS NULL;
code        title
----------  -------------------
ABCCONS     Ongoing consultancy

SQLite includes the function ifnull(), which can be used to specify an alternate to be used when a column value is NULL. For instance, we can visually indicate the ongoing projects in the database using this query:

sqlite> .mode column
sqlite> .header on
sqlite> SELECT code, ifnull(due_date, 'Ongoing')
   ...> FROM projects;
code        ifnull(due_date, 'Ongoing')
----------  ---------------------------
ACME01      20031231
ABCCONS     Ongoing
PREM3K      20040430
PREM4K      20040731
PREM5K      20041031

Likewise in a WHERE clause, we can select all the rates of pay that were current as of December 1, 2003, as in the following example:

sqlite> SELECT employee_id, rate
   ...> FROM employee_rates
   ...> WHERE start_date <= 20031201
   ...> AND ifnull(end_date, 20031201) >= 20031201;
employee_id  rate
-----------  -----
101          30.00
102          15.00
103          25.00
104          20.00

The WHERE clause compares each start_date to see if the rate was valid on or before December 1, and checks that end_date falls on or after this date. For a row where end_date is NULL, the comparison is between 20031201 and 20031201, which will always be true.

Note

SQLite has tried to implement the way it handles NULL in a standards-compliant way; however, in some circumstances the standards do not clearly dictate how a NULL should he handled. For details of how SQLite handles NULL compared to other SQL engines, see http://www.hwaci.com/sw/sqlite/nulls.html.

Arithmetic Functions

Simple arithmetic functions in SQLite can be used in the column list of a SELECT statement or in the WHERE clause. In fact, to demonstrate the basic functions you don't even need to select from a table.

sqlite> SELECT 2 + 2;
2 + 2
----------
4

sqlite> SELECT 10 - 7;
10 - 7
----------
3

sqlite> SELECT 6 * 3;
6 * 3
----------
18

sqlite> SELECT 100 / 30;
100 / 30
----------
3

Note in the division example that a rounded integer is returned. To specify that you want a floating-point result, one of the arguments must be a float itself.

sqlite> SELECT 100.0 / 30;
100.0 / 30
----------------
3.33333333333333

The modulo operator is also supported, using the percent symbol, as in many other languages:

sqlite> SELECT 100 % 30;
100 % 30
----------
10

You can apply such operators to a selected column in the SELECT statement. For instance, to find all the current rates of pay plus 15%, we could do this:

sqlite> SELECT rate * 1.15 FROM employee_rates
   ...> WHERE end_date IS NULL;
rate * 1.15
-----------
34.5
17.25
28.75
28.75

Table 2.2 lists the arithmetic operators.

Table 2.2. Arithmetic Operators in Order of Precedence

Operator

Meaning

/

Division

*

Multiplication

%

Modulo

+

Addition

-

Subtraction

String Operators and Functions

SQLite's concatenation operator is ||, which can be used to join two or more strings together.

sqlite> SELECT last_name || ', ' || first_name
   ...> FROM employees;
last_name || ', ' || first_name
-------------------------------
Gladstone, Alex
Dudson, Brenda
Aynsley, Colin
Churchill, Debbie

A number of built-in functions are also available for string manipulation. length() takes a single string parameter and returns its length in characters:

sqlite> SELECT company_name, length(company_name)
   ...> FROM clients;
company_name              length(company_name)
------------------------  --------------------
Acme Products             13
ABC Enterprises           15
Premier Things Ltd        18
Integer Primary Key Corp  24

To extract a section of a string, use substr(). Three parameters are requiredthe string itself, the start position (using 1 for the leftmost character), and a length:

sqlite> SELECT last_name, substr(last_name, 2, 4)
   ...> FROM employees;
last_name   substr(last_name, 2, 4)
----------  -----------------------
Gladstone   lads
Dudson      udso
Aynsley     ynsl
Churchill   hurc

To capitalize a string or convert it to lowercase, use upper() and lower() respectively:

sqlite> SELECT company_name, upper(company_name)
   ...> FROM clients;
company_name              upper(company_name)
------------------------  -------------------
Acme Products             ACME PRODUCTS
ABC Enterprises           ABC ENTERPRISES
Premier Things Ltd        PREMIER THINGS LTD
Integer Primary Key Corp  INTEGER PRIMARY KEY

One of SQLite's powerful features is the ability to add user-defined functions to the SQL command set. We will see how this is done using the various language APIs in Part II, "Using SQLite Programming Interfaces," of this book.

Joins

The key concept of a relational database structure is that of separating your data into tables that model, generally speaking, separate physical or logical entitiesthe process of normalizationand rejoining them in a proper manner when executing a query.

In our sample database, we have tables containing a number of relationships. For instance, each employee has worked a number of hours, usually over the course of a number of days. The number of hours worked on each day is stored as a separate entry in the timesheets table.

The employee_id field in timesheets (often written as timesheets.employee_id) relates to the id field in employees (employees.id). To correlate each row of timesheet data with its respective employees record, we need to perform a join on these fields. In SQLite, this is how this looks:

sqlite> SELECT first_name, last_name, date_worked, hours
   ...> FROM employees, timesheets
   ...> WHERE employees.id = timesheets.employee_id;
first_name  last_name   date_worked  hours
----------  ----------  -----------  -----
Alex        Gladstone   20031229     4
Alex        Gladstone   20031229     2
Alex        Gladstone   20040102     6
Brenda      Dudson      20031229     3
Brenda      Dudson      20040102     5
Colin       Aynsley     20031229     2
Colin       Aynsley     20040102     3
Colin       Aynsley     20031229     8
Debbie      Churchill   20040102     8
Debbie      Churchill   20040102     4

The repeated information is the reason we split these tables up in the first place. Rather than store each name and any other employee details in the timesheets table, if we split off the employees data we only need to keep one copy.

In reality many more hours will be logged than in our example. So, supposing Brenda Dudson married and changed her name, we would have to update the last_name field for every row in timesheets. With the normalized relationship between the tables, only one data item needs to be changed.

In the preceding example we had to use employees.id to qualify that the join was to the id field in the employees tabletimesheets also has a field named id. Using tablename.fieldname is the way to specify exactly which field you are talking about. If this were left out, SQLite would throw an error rather than attempt to make the decision for you:

sqlite> SELECT first_name, last_name, date_worked, hours
   ...> FROM employees, timesheets
   ...> WHERE id = employee_id;
SQL error: ambiguous column name: id

Notice that our selected columns were not qualified in this way. When the column name is unique across all the tables in a query, this is fine. If we wanted to fetch both id fields, though, they would need qualifying the same way.

To save typing the table name in full repeatedly whenever a column name has to be qualified, you can specify a table alias immediately after each table name in the FROM section. The following example aliases employees to e and timesheets to t, and the single-letter aliases are then used to specify which id fields are referenced in the selected columns list and WHERE clause.

sqlite> SELECT e.id, first_name, last_name, t.id, date_worked, hours
   ...> FROM employees e, timesheets t
   ...> WHERE e.id = t.employee_id;
e.id  first_name  last_name   t.id  date_worked  hours
----  ----------  ----------  ----  -----------  -----
101   Alex        Gladstone   1     20031229     4
101   Alex        Gladstone   2     20031229     2
101   Alex        Gladstone   3     20040102     6
102   Brenda      Dudson      4     20031229     3
102   Brenda      Dudson      5     20040102     5
103   Colin       Aynsley     6     20031229     2
103   Colin       Aynsley     7     20040102     3
103   Colin       Aynsley     8     20031229     8
104   Debbie      Churchill   9     20040102     8
104   Debbie      Churchill   10    20040102     4

Note

We did not specify any id values when the timesheet data was inserted; the INTEGER PRIMARY KEY attribute has assigned sequential values starting from 1.

A query can join many tables together, not just two. In the following example, we join employees, timesheets, projects, and clients to produce a report showing the title of all the projects and the names of all the employees that have worked on each one.

sqlite> SELECT c.company_name, p.title, e.first_name, e.last_name
   ...> FROM clients c, projects p, timesheets t, employees e
   ...> WHERE c.id = p.client_id
   ...> AND t.project_code = p.code
   ...> AND e.id = t.employee_id;
c.company_name      p.title               e.first_name  e.last_name
------------------  --------------------  ------------  ------------
Acme Products       Ordering system       Alex          Gladstone
Acme Products       Ordering system       Brenda        Dudson
Acme Products       Ordering system       Colin         Aynsley
Acme Products       Ordering system       Debbie        Churchill
ABC Enterprises     Ongoing consultancy   Alex          Gladstone
ABC Enterprises     Ongoing consultancy   Colin         Aynsley
ABC Enterprises     Ongoing consultancy   Debbie        Churchill
Premier Things Ltd  Thing 3000            Alex          Gladstone
Premier Things Ltd  Thing 4000            Brenda        Dudson
Premier Things Ltd  Thing 4000            Colin         Aynsley

Aggregate Functions

Aggregate functions provide a quick way to find summary information about any of your data. The very simplest operation is a count, which is executed as follows:

sqlite> SELECT count(*) FROM employees;
count(*)
----------
4

The number of rows in the dataset is displayed. Of course this can be used in conjunction with a WHERE clause; for instance, we can see how many male employees there are.

sqlite> SELECT count(*) FROM employees
   ...> WHERE sex = 'M';
count(*)
----------
2

However, rather than executing one query for each possible value of sex, we can use a GROUP BY clause and an aggregate function to show us the number of employees of each sex.

sqlite> SELECT sex, count(*)
   ...> FROM employees
   ...> GROUP BY sex;
sex         count(*)
----------  ----------
F           2
M           2

Other aggregate functions available include sum(), avg(), min(), and max(), which find, respectively, the sum and average of a set of integers and the largest and smallest values from the set.

To find the average current hourly rate paid, we could use avg() as follows:

sqlite> SELECT avg(rate) FROM employee_rates
   ...> WHERE end_date IS NULL;
avg(rate)
----------
23.75

To find the averages for men and women, we would include sex in the column list, and then use the GROUP BY clause on this column.

sqlite> SELECT sex, avg(rate)
   ...> FROM employees e, employee_rates r
   ...> WHERE e.id = r.employee_id
   ...> GROUP BY sex;
sex         avg(rate)
----------  ----------
F           20
M           27.5

We'll see in Part II of this book how you can create your own aggregating functions using the language APIs that SQLite provides.

Ordering Data

Unless it is specified, the order in which selected rows are returned is undefinedthough they are usually brought back in the order in which they were inserted. SQLite uses the ORDER BY clause to specify the ordering of a dataset.

We actually inserted our employees in order of first name, but if we want the result of our SELECT statement to sort on last_name, we can add this in an ORDER BY clause:

sqlite> SELECT last_name, first_name
   ...> FROM employees
   ...> ORDER BY last_name;
last_name   first_name
----------  ----------
Aynsley     Colin
Churchill   Debbie
Dudson      Brenda
Gladstone   Alex

The ORDER BY can specify more than one column for sorting. Note that the columns listed need not actually appear in the list of columns selected. In the following example we order by sex first, then by last_name. The two women appear at the top of the list in alphabetical order, followed by the men.

sqlite> SELECT last_name, first_name
   ...> FROM employees
   ...> ORDER BY sex, last_name;
last_name   first_name
----------  ----------
Churchill   Debbie
Dudson      Brenda
Aynsley     Colin
Gladstone   Alex

The ORDER BY clause is one situation where the data type of the columns referenced actually does matter, because SQLite needs to make a decision on how to evaluate a comparison between two values.

For instance if a column does not have a data type, what determines whether a value of 100 is higher or lower than a value of 50? As two integers, clearly 100 is higher that 50, but a string comparison would consider 50 "higher" because its first character appears later in the ASCII character set.

In earlier versions of SQLite (version 2.6.3 and below), all comparisons for sorting were treated as numeric, with strings being compared alphabetically only if they could not be evaluated as numbers. Since version 2.7.0, a column will take one of two types, either numeric or text, and the CREATE TABLE statement actually determines which type is used.

Don't be confuseda numeric column type is still typeless inasmuch as it can store any kind of data, so it is not restricted to integer, as an INTEGER PRIMARY KEY is, or indeed to any other numeric type. The fact that a column is considered numeric only comes into play when a value has to be compared.

You saw before that any number of names can make up a data type name in the CREATE TABLE statement. If the name contains one or more of the following strings, the column will be declared text. For all other data type names, or if the data type is omitted, it will be numeric.

  • BLOB

  • CHAR

  • CLOB

  • TEXT

To confirm the data type of a column you can use the typeof() function. Because the function would apply to the entire dataset, limit the records returned using LIMIT.

sqlite> SELECT typeof(code), typeof(client_id)
   ...> FROM projects
   ...> LIMIT 1;
typeof(code)  typeof(client_id)
------------  -----------------
text          numeric

Expressions also have an implicit data type for comparisons, usually determined by the outermost operator regardless of the actual arguments. An arithmetic operator returns a numeric result and a string operator returns a text result, regardless of the implied types of the arguments themselves.

As you might expect, you can also check the data type of an expression using typeof():

sqlite> select 123 + 456, typeof(123 + 456);
123 + 456   typeof(123 + 456)
----------  ------------------------
579         numeric

sqlite> select 123 || 456, typeof (123 || 456);
123 || 456  typeof (123 || 456)
----------  ------------------------
123456      text

sqlite> select 'abc' + 456, typeof('abc' + 456);
'abc' + 1   typeof('abc' + 456)
----------  ------------------------
456         numeric

Limiting Data

Although this problem does not occur with our demo database, sometimes a query will return more rows than you want to process. You can use the LIMIT clause to limit the number of rows returned by specifying a number:

sqlite> SELECT * FROM employees LIMIT 2;
id    first_name  last_name   sex  email
----  ----------  ----------  ---  --------------------
101   Alex        Gladstone   M    alex@mycompany.com
102   Brenda      Dudson      F    brenda@mycompany.com

LIMIT can be used in conjunction with a WHERE clause to further restrict the dataset after any conditions have been evaluated. Any ORDER BY clause specified is also taken into account first before limiting takes place. The following example fetches only the first record alphabetically from all the male employees.

sqlite> SELECT * FROM employees
   ...> WHERE sex = 'M'
   ...> ORDER by last_name, first_name
   ...> LIMIT 1;
id    first_name  last_name   sex  email
----  ----------  ----------  ---  --------------------
103   Colin       Aynsley     M    colin@mycompany.com

LIMIT optionally takes an offset parameter, so that rather than returning the first N rows, it will skip a number of rows from the top of the dataset and then return the next N rows. In the following example, rows 3 and 4 are returned from the query, using an offset of 2 and a limit of 2.

sqlite> SELECT * FROM projects
   ...> ORDER BY due_date DESC
   ...> LIMIT 2, 2;
code      client_id   title            start_date  due_date
--------  ----------  ---------------  ----------  ----------
PREM3K    503         Thing 3000       20031201    20040430
ACME01    501         Ordering system  20030401    20031231

Updating and Deleting Records

To modify records in your database or remove them, use the SQL commands UPDATE and DELETE. Both use a WHERE clause to specify which rows are affected, so the syntax will be fairly familiar to you by now.

Suppose we want to split some of the hours worked on the Ongoing Consultancy project for ABC Enterprises off into another project. The UPDATE statement would look something like this:

UPDATE timesheets
SET project_code = new_project_code
WHERE condition

Let's follow this example through and move all the hours spent in 2003 into a new project code ABC2003. First we insert a new project record.

sqlite> INSERT INTO projects (code, client_id, title, start_date, due_date)
   ...> VALUES ('ABC2003', 502, 'Work in 2003', 20030101, 20031231);

Then we can run an UPDATE command to move all the 2003 hours to this new project.

sqlite> UPDATE timesheets
   ...> SET project_code = 'ABC2003'
   ...> WHERE project_code = 'ABCCONS'
   ...> AND date_worked BETWEEN 20030101 and 20031231;

Note

In this example we have assigned our own primary key value as project_code. If the insert had been to a table using an INTEGER PRIMARY KEY, the primary key value could have been found using last_insert_rowid().

The UPDATE is silent unless there is a problem, so there is no confirmation on screen. The following query to find the total number of hours worked on each project and the extent of the work dates can be used to show that the update has indeed taken place.

sqlite> SELECT project_code, count(*), min(date_worked), max(date_worked)
   ...> FROM timesheets
   ...> GROUP BY project_code;
project_code  count(*)  min(date_worked)  max(date_worked)
------------  --------  ----------------  ----------------
PREM4K        2         20040102          20040102
PREM3K        1         20040102          20040102
ABCCONS       1         20040102          20040102
ABC2003       2         20031229          20031229
ACME01        4         20031229          20040102

You can update multiple columns in one UPDATE statement by supplying a comma-separated list of assignments. For instance, to update the contact name and telephone number for a client in one statement, you could do the following:

sqlite> UPDATE clients
   ...> SET contact_name = 'Mr A. Newman',
   ...>     telephone = '555-8888'
   ...> WHERE id = 501;

Let's say you change your mind about the previous reallocation of timesheets and want to remove project code ABC2003. The DELETE syntax for this reads as you might expect:

sqlite> DELETE FROM projects
   ...> WHERE code = 'ABC2003';

A list of columns is not required for a DELETE because action is taken on a record-by-record basis. Asking SQLite to delete only a particular column makes no sense. To blank the values of specific columns while keeping the record in the database, use an UPDATE and set the necessary fields to NULL.

Note

The WHERE clause in an UPDATE or DELETE statement is very important. If no WHERE clause is specified, action will be taken against every row in a tablewhich is usually not the desired result. Performing DELETE FROM tablename; will empty that table completely!

Left Joins

The DELETE we did in the preceding section has actually left our data in quite a mess. The relationship between timesheets and projects is broken because we now have timesheet data with a nonexistent project_code.

Some database systems have a cascading delete facility, which means that when a FOREIGN KEY is defined on a table, if that key value is deleted from the master table, the corresponding child records are also deleted. SQLite does not support this functionality in the same way, although we will look at a way it can be implemented with a trigger in the next chapter.

However, now that we are in this situation, we can use it to demonstrate the LEFT JOIN operator in SQLite. The joins you saw previously used a WHERE clause to specify the relationship between the tables, for instance timesheets.project_code = projects.code.

A LEFT JOIN also uses a condition to combine the data from both the tables, but where a row from the first table does not have a corresponding row in the second, a record is still returnedcontaining NULL values for each of the fields in the second table.

Let's see this in action. The following query specifies a LEFT JOIN between timesheets and projects so that our orphaned timesheet records still appear, but with blank project information.

sqlite> SELECT p.title, p.code, sum(t.hours)
   ...> FROM timesheets t
   ...> LEFT JOIN projects p
   ...> ON p.code = t.project_code
   ...> GROUP BY p.title, p.code
   ...> ORDER BY p.title;
p.title                   p.code    sum(t.hours)
------------------------  --------  ------------
                                    4
Ongoing consultancy       ABCCONS   4
Ordering system           ACME01    23
Thing 3000                PREM3K    6
Thing 4000                PREM4K    8

So to find details of the timesheet entries that do not have a matching project code, we can modify the preceding query and use an IS NULL condition to take advantage of this property of a LEFT JOIN.

sqlite> SELECT t.*
   ...> FROM timesheets t
   ...> LEFT JOIN projects p
   ...> ON p.code = t.project_code
   ...> WHERE p.code IS NULL;
t.id  t.employee_id  t.project_code  t.date_worked   t.hours
----  -------------  --------------  --------------  --------
2     101            ABC2003         20031229        2
6     103            ABC2003         20031229        2

Nested Subqueries

There is another, often more readable, way to perform the same timesheet query using a nested subquery. SQLite allows you to use the dataset produced by any query as the argument to an IN or NOT IN expression.

The following query finds all the timesheet records where the project code is not one of those in the projects table.

sqlite> SELECT *
   ...> FROM timesheets
   ...> WHERE project_code NOT IN (
   ...>   SELECT code
   ...>   FROM projects
   ...> );
id    employee_id   project_code    date_worked     hours
----  ------------  --------------  --------------  --------
2     101           ABC2003         20031229        2
6     103           ABC2003         20031229        2

Though more readable, the NOT IN query in this example is usually less efficient than the LEFT JOIN method. We will deal with query optimization in more detail in Chapter 4, "Query Optimization."

Cartesian Joins

Another type of join possible in SQLite is the Cartesian join. This is something that usually only happens by mistake, so we mention it here only so that you are able to spot such problems.

If two (or more) tables are selected without a proper join in the WHERE clause, every row in table 1 is paired with each row in table 2 to produce a Cartesian productthe number of rows returned is the product of the number of rows in each table.

The following query shows a Cartesian join on the clients and projects tables, and you should be able to see that with tables containing significant amounts of data, a Cartesian join is generally not a desirable result.

sqlite> SELECT c.company_name, p.title
   ...> FROM clients c, projects p;
c.company_name            p.title
------------------------  ------------------------
Acme Products             Ordering system
Acme Products             Ongoing consultancy
Acme Products             Thing 3000
Acme Products             Thing 4000
Acme Products             Thing 5000
ABC Enterprises           Ordering system
ABC Enterprises           Ongoing consultancy
ABC Enterprises           Thing 3000
ABC Enterprises           Thing 4000
ABC Enterprises           Thing 5000
Premier Things Ltd        Ordering system
Premier Things Ltd        Ongoing consultancy
Premier Things Ltd        Thing 3000
Premier Things Ltd        Thing 4000
Premier Things Ltd        Thing 5000
Integer Primary Key Corp  Ordering system
Integer Primary Key Corp  Ongoing consultancy
Integer Primary Key Corp  Thing 3000
Integer Primary Key Corp  Thing 4000
Integer Primary Key Corp  Thing 5000

Transactions

Transactions are the way that databases implement atomicitythe assurance that the entirety of a request to alter the database is acted upon and not just part of it. Atomicity is one of the key requirements of a mission-critical database system.

Every time the database is changedin other words for any INSERT, UPDATE, or DELETE to take placethe change must take place inside a transaction. SQLite will start an implicit transaction whenever one of these commands is issued if there is not already a transaction in progress.

You would want to start a transaction explicitly if you wanted to make a series of changes that must all take place at the same time. Simply issuing the commands one after each other does not absolutely guarantee that if the earlier commands are executed the later commands will also have been processed.

Let's say in our demo database that we wanted to change one of the project codes, and because the timesheets table is joined to projects on project_code, we need to ensure that the same change is made to both tables at the same time. In SQLite we do this as follows:

sqlite> BEGIN TRANSACTION;
sqlite> UPDATE projects
   ...> SET code = 'ACMENEW'
   ...> WHERE code = 'ACME01';
sqlite> UPDATE timesheets
   ...> SET project_code = 'ACMENEW'
   ...> WHERE project_code = 'ACME01';
sqlite> COMMIT TRANSACTION;

Until the COMMIT TRANSACTION command is issued, neither of the UPDATE statements have caused a change to be made to the database. A query executed within the transaction will consider these changes to have been made; however, the ROLLBACK TRANSACTION command can be used to cancel the entire transaction. The following example shows a mistake made when making an update that was rolled back:

sqlite> BEGIN TRANSACTION;
sqlite> UPDATE clients
   ...> SET contact_name = 'Mr S. Error';
sqlite> SELECT * from clients;
id    company_name              contact_name    telephone
----  ------------------------  --------------  ----------
501   Acme Products             Mr S. Error     555-8888
502   ABC Enterprises           Mr S. Error     555-2999
503   Premier Things Ltd        Mr S. Error     555-4001
504   Integer Primary Key Corp  Mr S. Error     555-1234
sqlite> ROLLBACK TRANSACTION;
sqlite> SELECT * from clients;
id    company_name              contact_name     telephone
----  ------------------------  ---------------  ----------
501   Acme Products             Mr A. Newman     555-8888
502   ABC Enterprises           Mr T. Boss       555-2999
503   Premier Things Ltd        Mr U. First      555-4001
504   Integer Primary Key Corp  Mr A. Increment  555-1234

The sqlite_master Table

You've already seen that from the sqlite tool you can view the list of tables in a database with the .tables command and view the schema of any table using .schema.

However, SQLite is designed as an embedded database, and ultimately you will be writing applications in your language of choice and calling SQLite API functions where the commands beginning with a period that are available in sqlite cannot be invoked directly.

We finish this chapter by looking at the internal tables that can be used to query a database and its table schemas.

The master table holding the key information about your database tables is called sqlite_master, and using sqlite you can see its schema as you might expect:

sqlite> .schema sqlite_master
CREATE TABLE sqlite_master (
  type text,
  name text,
  tbl_name text,
  rootpage integer,
  sql text
);

The type field tells us what kind of object we are dealing with. So far we've only looked at tables, but indexes and triggers are also stored in sqlite_master, and we'll look at these in more detail in Chapter 3.

For a table, name and tbl_name are the same. An index or trigger will have its own identifier, but tbl_name will contain the name of the table to which it applies. The sql field contains the CREATE TABLE (or CREATE INDEX or CREATE TRIGGER) statement issued when each database object was created.

Note

The sqlite_master table is read-only. You cannot modify it directly using UPDATE, DELETE, or INSERT statements.

So, to find the names of all tables in a database, do the following:

sqlite> SELECT tbl_name
   ...> FROM sqlite_master
   ...> WHERE type = 'table';
tbl_name
--------------------
employees
employee_rates
clients
projects
timesheets

Or to find the SQL command used to create a table, the following query will workshown here in line output mode so that you can see the whole text.

sqlite> SELECT sql
   ...> FROM sqlite_master
   ...> WHERE type = 'table'
   ...> AND tbl_name = 'projects';
  sql = CREATE TABLE projects (
  code CHAR PRIMARY KEY,
  client_id INTEGER NOT NULL,
  title CHAR NOT NULL,
  start_date INTEGER,
  due_date INTEGER
)


Chapter 3. SQLite Syntax and Use

In this chapter we look in detail at the SQL syntax understood by SQLite. We will discuss the full capabilities of the language and you will learn to write effective, accurate SQL.

You have already come across most of the supported SQL commands in Chapter 2, "Working with Data," in the context of the demo database. This chapter builds on that knowledge by exploring the syntax and usage of each command in more detail to give a very broad overview of what you can do using SQLite.


Naming Conventions

Each database, table, column, index, trigger, or view has a name by which it is identified and almost always the name is supplied by the developer. The rules governing how a valid identifier is formed in SQLite are set out in the next few sections.

Valid Characters

An identifier name must begin with a letter or the underscore character, which may be followed by a number of alphanumeric characters or underscores. No other characters may be present. These identifier names are valid:

  • mytable

  • my_field

  • xyz123

  • a

However, the following are not valid identifiers:

  • my table

  • my-field

  • 123xyz

You can use other characters in identifiers if they are enclosed in double quotes (or square brackets), for example:

sqlite> CREATE TABLE "123 456"("hello-world", " ");

Name Length

SQLite does not have a fixed upper limit on the length of an identifier name, so any name that you find manageable to work with is suitable.

Reserved Keywords

Care must be taken when using SQLite keywords as identifier names. As a general rule of thumb you should try to avoid using any keywords from the SQL language as identifiers, although if you really want to do so, they can be used providing they are enclosed in square brackets.

For instance the following statement will work just fine, but this should not be mimicked on a real database for the sake of your own sanity.

sqlite> CREATE TABLE [TABLE] (
   ...>   [SELECT],
   ...>   [INTEGER] INTEGER,
   ...>   [FROM],
   ...>   [TABLE]
   ...> );

Case Sensitivity

For the most part, case sensitivity in SQLite is off. Table names and column names can be typed in uppercase, lowercase, or mixed case, and different capitalizations of the same database object name can be used interchangeably.

SQL commands are always shown in this book with the keywords in uppercase for clarity; however, this is not a requirement.

Note

The CREATE TABLE, CREATE VIEW, CREATE INDEX, and CREATE TRIGGER statements all store the exact way in which they were typed to the database so that the command used to create a database object can be retrieved by querying the sqlite_master table. Therefore it is always a good idea to format your CREATE statements clearly, so they can be referred to easily in the future.


Creating and Dropping Tables

Creating and dropping database tables in SQLite is performed with the CREATE TABLE and DROP TABLE commands respectively. The basic syntax for CREATE TABLE is as follows:

CREATE [TEMP | TEMPORARY] TABLE table-name (
  column-def[, column-def]*
  [,constraint]*
);

Simply put, a table may be declared as temporary, if desired, and the structure of each table has to have one or more column definitions followed by zero or more constraints.

Table Column Definitions

A column definition is defined as follows:

name [type] [[CONSTRAINT name] column-constraint]*

As you saw in Chapter 2, SQLite is typeless and therefore the type attribute is actually optional. Except for an INTEGER PRIMARY KEY column, the data type is only used to determine whether values stored in that column are to be treated as strings or numbers when compared to other values.

You can use the optional CONSTRAINT clause to specify one or more of the following column constraints that should be enforced when data is inserted:

  • NOT NULL

  • DEFAULT

  • PRIMARY KEY

  • UNIQUE

A column declared as NOT NULL must contain a value; otherwise, an INSERT attempt will fail, as demonstrated in the following example:

sqlite> CREATE TABLE vegetables (
   ...>   name CHAR NOT NULL,
   ...>   color CHAR NOT NULL
   ...> );

sqlite> INSERT INTO vegetables (name) VALUES ('potato');
SQL error: vegetables.color may not be NULL

Often, a column declared NOT NULL is also given a DEFAULT value, which will be used automatically if that column is not specified in an INSERT. The following example shows this in action.

sqlite> CREATE TABLE vegetables (
   ...>   name CHAR NOT NULL,
   ...>   color CHAR NOT NULL DEFAULT 'green'
   ...> );

sqlite> INSERT INTO vegetables (name, color) VALUES ('carrot', 'orange');
sqlite> INSERT INTO vegetables (name) VALUES ('bean');
sqlite> SELECT * FROM vegetables;
name        color
----------  ----------
carrot      orange
bean        green

However, if you attempt to insert NULL explicitly into a NOT NULL column, SQLite will still give an error:

sqlite> INSERT INTO vegetables (name, color) VALUES ('cabbage', NULL);
SQL error: vegetables.color may not be NULL

Functionally, a PRIMARY KEY column behaves just the same as one with a UNIQUE constraint. Both types of constraint enforce that the same value may only be stored in that column once, but other than the special case of an INTEGER PRIMARY KEY, the only point to note is that a table can have only one PRIMARY KEY column.

SQLite will raise an error whenever an attempt is made to insert a duplicate value into a UNIQUE or PRIMARY KEY column, as shown in the following example. This example also shows that a column can be declared as both NOT NULL and UNIQUE.

sqlite> CREATE TABLE vegetables (
   ...>   name CHAR NOT NULL UNIQUE,
   ...>   color CHAR NOT NULL
   ...> );
sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'red');
sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'green');
SQL error: column name is not unique

Resolving Conflicts

NOT NULL, PRIMARY KEY, and UNIQUE constraints may all be used in conjunction with an ON CONFLICT clause to specify the way a conflict should be resolved if an attempt to insert or modify data violates a column constraint.

The conflict resolution algorithms supported are

  • ROLLBACK

  • ABORT

  • FAIL

  • IGNORE

  • REPLACE

You could apply a constraint to the vegetables table from the preceding example as follows:

sqlite> CREATE TABLE vegetables (
   ...>   name CHAR NOT NULL UNIQUE ON CONFLICT REPLACE,
   ...>   color CHAR NOT NULL
   ...> );

This time, because REPLACE was specified as the conflict resolution algorithm, inserting the same vegetable name twice does not cause an error. Instead the new record replaces the conflicting record.

sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'red');
sqlite> INSERT INTO vegetables (name, color) VALUES ('pepper', 'green');
sqlite> SELECT * FROM vegetables;
name        color
----------  ----------
pepper      green

The REPLACE algorithm ensures that an SQL statement is always executed, even if a UNIQUE constraint would otherwise be violated. Before the UPDATE or INSERT takes place, any pre-existing rows that would cause the violation are removed. If a NOT NULL constraint is violated and there is no DEFAULT value, the ABORT algorithm is used instead.

The ROLLBACK algorithm causes an immediate ROLLBACK TRANSACTION to be issued as soon as the conflict occurs and the command will exit with an error.

When you use the ABORT algorithm, no ROLLBACK TRANSACTION is issued, so if the violation occurs within a transaction consisting of more than one INSERT or UPDATE, the database changes from the previous statements will remain. Any changes attempted by the statement causing the violation, however, will not take place. For a single command using only an implicit transaction, the behavior is identical to ROLLBACK.

The FAIL algorithm causes SQLite to stop with an error when a constraint is violated; however, any changes made as part of that command up to the point of failure will be preserved. For instance, when an UPDATE statement performs a change sequentially on many rows of the database, any rows affected before the constraint was violated will remain updated.

SQLite will never stop with an error when the IGNORE algorithm is specified and the constraint violation is simply passed by. In the case of an UPDATE affecting multiple rows, the modification will take place for every row other than the one that causes the conflict, both before and after.

The ON CONFLICT clause in a CREATE TABLE statement has the lowest precedence of all the places in which it can be specified. An overriding conflict resolution algorithm can be specified in the ON CONFLICT clause of a BEGIN TRANSACTION command, which can in turn be overridden by the OR clause of a COPY, INSERT, or UPDATE statement. We will see the respective syntaxes for these clauses later in this chapter.

The CHECK Clause

The CREATE TABLE syntax also allows for a CHECK clause to be defined, with an expression in parentheses. This is a feature included for SQL compatibility and is reserved for future use, but at the time of this writing is not implemented.

Using Temporary Tables

Using CREATE TEMPORARY TABLE creates a table object in SQLite that can be queried and manipulated exactly the same as a nontemporary table. However, the table will only be visible to the process in which it was created and will be destroyed as soon as the database is closed.

$ sqlite tempdb
SQLite version 2.8.12
Enter ".help" for instructions
sqlite> CREATE TEMPORARY TABLE temptable (
   ...>   myfield char
   ...> );
sqlite> INSERT INTO temptable (myfield) VALUES ('abc');
sqlite> .quit

$ sqlite tempdb
SQLite version 2.8.12
Enter ".help" for instructions
sqlite> INSERT INTO temptable (myfield) VALUES ('xyz');
SQL error: no such table: temptable

The data inserted into a temporary table and its schema are not written to the connected database file, nor is there a record created in sqlite_master. Instead a separate table, sqlite_temp_master, is used to reference temporary tables.

sqlite> CREATE TEMPORARY TABLE temptable (
   ...>   myfield char
   ...> );
sqlite> SELECT * FROM sqlite_temp_master;
    type = table
    name = temptable
tbl_name = temptable
rootpage = 3
     sql = CREATE TEMPORARY TABLE temptable (
  myfield char
)

Temporary tables are specific to the sqlite handle, not the process. Surprisingly, people often become confused about this, particularly in Windows, where a common design pattern is to open a separate sqlite handle to the same database from each thread.


Anatomy of a SELECT Statement

The syntax definition for an SQL statement is

SELECT [ALL | DISTINCT] result [FROM table-list]
[WHERE expr]
[GROUP BY expr-list]
[HAVING expr]
[compound-op select]*
[ORDER BY sort-expr-list]
[LIMIT integer [(OFFSET|,) integer]]

The only required item in a SELECT statement is the result, which can be one of the following:

  • The * character

  • A comma-separated list of one or more column names

  • An expression

The latter two bullet points should be combined into: "A comma-separated list of one or more expressions." The original two points make it seem as if the following would be an error:

SELECT a+1, b+1 FROM ab;

but this would be okay:

SELECT a, b FROM ab;

In fact, both are valid.

Using the * character or a list of columns makes no sense without a FROM clause, but in fact an expression whose arguments are constants rather than database items can be used alone in a SELECT statement, as in the following examples.

sqlite> SELECT (60 * 60 * 24);
86400

sqlite> SELECT max(5, 20, -4, 8.7);
20

sqlite> SELECT random();
220860261

If the FROM list is omitted, SQLite effectively evaluates the expression against a table that always contains a single row.

The FROM list includes one or more table names in a comma-separated list, each with an optional alias name that can be used to qualify individual column names in the result. Where aliases are not used, the table name in full can be used to qualify columns.

For instance, the two following SELECT statements are identicalthe latter uses a table alias m for mytable.

SELECT mytable.myfield
FROM myfield;

SELECT m.myfield
FROM mytable m;

The WHERE Clause

The WHERE clause specifies one or more conditions used to impose restrictions on the dataset returned by a SELECT. It is used both to limit the number of rows returned and to indicate a relationship used to join two tables together.

The general usage to impose a condition on the rows in a table is as follows:

SELECT result
FROM table-list
WHERE expr;

Expression expr generally involves a comparison of some kind on a table column, as shown in the following example:

SELECT *
FROM mytable
WHERE myfield = 'somevalue';

Table 3.1 shows the relational operators that can be used in a WHERE clause.

Table 3.1. Relational Operators

Operator

Meaning

a = b

a is equal to b

a != b

a is not equal to b

a < b

a is less than b

a > b

a is greater than b

a <= b

a is less than or equal to b

a >= b

a is greater than or equal to b

a IN (b, c)

a is equal to either b or c

a NOT IN (b, c)

a is equal to neither b nor c

When you perform a comparison using a relational operatorand particularly the greater-than and less-than operators, < and >the data type of the column comes into play.

The following example shows how comparisons between the numbers 8, 11, and 101 differ greatly when performed as string operations. As integers, the order is as you would expect, however as strings, '101' is less than '11', which is in turn less than '8'. The individual character values in the string are compared from left to right in turn to determine which is the greatest value.

sqlite> CREATE TABLE compare (string TEXT, number INTEGER);
sqlite> INSERT INTO compare (string, number) values ('101', 101);
sqlite> SELECT number FROM compare WHERE number > 11;
101
sqlite> SELECT string FROM compare WHERE string > '11';
8

However, note that if you use a relational operator with a number argument that is not contained in quotes, an integer comparison is performed regardless of the column data type.

sqlite> SELECT string FROM compare WHERE string > 11;
101

Selecting from multiple tables without a WHERE clause produces a Cartesian product of the datasets and is usually not a desirable result. With two tables, each record in table1 is paired with each record in table2. The total number of rows returned is the product of the number of rows in each table in the FROM list.

The following example shows the result of a Cartesian product of three tables, each containing just two rows. In total, eight rows are returned (2x2x2).

sqlite> SELECT table1.myfield, table2.myfield, table3.myfield
   ...> FROM table1, table2, table3;
table1.myfield  table2.myfield  table3.myfield
--------------  --------------  --------------
Table 1 row 1   Table 2 row 1   Table 3 row 1
Table 1 row 1   Table 2 row 1   Table 3 row 2
Table 1 row 1   Table 2 row 2   Table 3 row 1
Table 1 row 1   Table 2 row 2   Table 3 row 2
Table 1 row 2   Table 2 row 1   Table 3 row 1
Table 1 row 2   Table 2 row 1   Table 3 row 2
Table 1 row 2   Table 2 row 2   Table 3 row 1
Table 1 row 2   Table 2 row 2   Table 3 row 2

In this example each table has a field called myfield, so each column in the result has to be qualified with the appropriate table name. This is not necessary where a column name is unique across all tables in the FROM list; however, it is good practice to always qualify column names to avoid ambiguity. If a column name could refer to more than one table, SQLite will not make the decision. Instead an error is raised as shown in the following example:

sqlite> SELECT myfield
   ...> FROM table1, table2, table3;
SQL error: ambiguous column name: myfield

To join two tables on a common fieldknown as an equi-join because the relationship is an equalitythe general syntax is

SELECT result
FROM table1, table2
WHERE table1.keyfield1 = table2.keyfield2

SQLite supports outer joins via the LEFT JOIN keyword, whereby each row in the left tablethe one specified first in the SELECT statementis combined with a row from the right table. Where the join condition does not produce a match between the two tables, rows from the left table are still returned but with NULL values for each column that should be in the right table.

The general syntax for a LEFT JOIN is as follows:

SELECT result
FROM table1
LEFT [OUTER] JOIN table2
ON table1.keyfield1 = table2.keyfield2
[WHERE expr]

The LEFT JOIN operator can be written as LEFT OUTER JOIN as a matter of preference; the OUTER keyword is optional.

GROUP BY and Aggregate Functions

The GROUP BY clause is used to aggregate data into a single row where the value of one or more specified columns is repeated. This feature can be used to reduce the number of records to only find unique values of a column, but is particularly useful when used in conjunction with the SQLite's aggregate functions.

The GROUP BY clause takes a list of expressionsusually column names from the resultand aggregates data for each expression. In the vegetables table we created previously we had more than one green vegetable, so grouping on the color column will return each value only once.

sqlite> SELECT color
   ...> FROM vegetables
   ...> GROUP BY color;
color
----------
green
orange

More interesting is to use the aggregate function count() to show how many records there are for each value of color:

sqlite> SELECT color, count(color)
   ...> FROM vegetables
   ...> GROUP BY color;
color       count(color)
----------  ------------
green       2
orange      1

Using count(fieldname) will return the number of rows containing a non-NULL value in that field. If you want to return a count of the total number of rows, regardless of any NULL values, count(*) will do this, as the following example shows:

sqlite> CREATE TABLE mytable (
   ...>   field1 CHAR,
   ...>   field2 INTEGER
   ...> );

sqlite> INSERT INTO mytable VALUES ('foo', 5);
sqlite> INSERT INTO mytable VALUES ('foo', 14);
sqlite> INSERT INTO mytable VALUES ('bar', 25);
sqlite> INSERT INTO mytable VALUES ('bar', 8);
sqlite> INSERT INTO mytable VALUES ('bar', NULL);

sqlite> SELECT field1, count(field2), count(*)
   ...> FROM mytable
   ...> GROUP BY field1;
field1      count(field2)  count(*)
----------  -------------  ----------
bar         2              3
foo         2              2

There are also a number of aggregate functions for performing summary calculations on grouped data, as shown in the following example:

sqlite> SELECT field1, sum(field2), min(field2), max(field2), avg(field2)
   ...> FROM mytable
   ...> GROUP BY field1;
field1      sum(field2)  min(field2)  max(field2)  avg(field2)
----------  -----------  -----------  -----------  -----------
bar         33           8            25           16.5
foo         19           5            14           9.5

Table 3.2 lists all the aggregate functions available in SQLite.

Table 3.2. Aggregate Functions

Function

Meaning

avg(column)

Returns the mean average of all values in column

count(column)

Returns the number of times that a non-NULL value appears in column

count(*)

Returns the total number of rows in a query, regardless of NULL values

max(column)

Returns the highest of all values in column, using the usual sort order

min(column)

Returns the lowest of all values in column, using the usual sort order

sum(column)

Returns the numeric sum of all values in column

HAVING Clause

The HAVING clause is a further condition applied after aggregation takes place. In contrast to a WHERE clause, which applies a condition to individual elements in a table, HAVING is used to restrict records based on the summary value of a grouping.

To return only rows from the vegetables table where there is more than one of the same color, we can do this:

sqlite> SELECT color, count(*)
   ...> FROM vegetables GROUP BY color
   ...> HAVING count(*) > 1;
color       count(*)
----------  ----------
green       2

It is actually not necessary for count(*) to appear in the result, as shown in the following example:

sqlite> SELECT color
   ...> FROM vegetables GROUP BY color
   ...> HAVING count(*) > 1;
color
----------
green

Column Aliases

The column headings displayed in the output of a SELECT statement are usually the same as the items specified in the result section. For a straight column, the name of the column is displayed. For an expression, however, the expression text is used.

Although the column headings are only displayed in sqlite when .headers is set to on, it is important to know what each column's name is so that all the columns can be referenced correctly from within a programming API. A column alias is specified with the AS keyword to explicitly give a new name to a selected column.

In the following example, we give a column alias to the result of the count(*) aggregate function. In addition to renaming the column header, we can use the alias name in the HAVING clause, which can sometimes aid readability of code.

sqlite> SELECT color, count(*) AS num_colors
   ...> FROM vegetables GROUP BY color
   ...> HAVING num_colors > 1;
color       num_colors
----------  ----------
green       2


Attaching to Another Database

Using sqlite, the .databases command lists all the databases that are open for the current session. There will always be two databases open after you invoke sqlite main, the database specified on the command line, and temp, the database used for temporary tables.

sqlite> .databases
0     main        /home/chris/sqlite/demodb
1     temp        /var/tmp/sqlite VGazbfyWvuUr29P

It is possible to attach more databases to your current session with the ATTACH DATABASE statement. This adds a connection to another database without replacing your currently selected database.

The syntax is

ATTACH [DATABASE] database-filename AS database-name

The keyword DATABASE is optional and is used only for readability, but you must provide a unique database-name parameter that will be used to qualify table references, essential in case more than one database could have the same table name.

Suppose you are working on a new database called newdb and want to access some of the databases from our demo database from Chapter 2. The following example shows demodb being attached to the current sqlite session:

$ sqlite newdb
SQLite version 2.8.12
Enter ".help" for instructions
sqlite> ATTACH DATABASE demodb AS demodb;
sqlite> .databases
0     main        /home/chris/sqlite/newdb
1     temp        /var/tmp/sqlite_VGazbfyWvuUr29P
2     demodb      /home/chris/sqlite/demodb

Accessing tables from an attached database is straightforwardjust prefix any table name with the database name (the name given after the keyword AS, not the filename, if they are different) and a period.

We can perform a query on the clients table from demodb as follows:

sqlite> SELECT company_name FROM demodb.clients;
company_name
--------------------
Acme Products
ABC Enterprises
Premier Things Ltd

Tables in the main database can be accessed using their table name alone, or qualified as main.tablename. If a table name is unique across all databases attached in a particular session, it does not need to be prefixed with its database name even if it is not in the main database. However, it is still good practice to qualify all tables when you are working with multiple databases to avoid confusion.

Note

The SQL commands INSERT, UPDATE, SELECT, and DELETE can all be performed on an attached database by using the database name prefix. However, CREATE TABLE and DROP TABLE can only take place on the main databaseyou must exit sqlite and begin a new session if you want to manipulate tables from a different database.

Note the situation with multi-database transactions here. If a machine or software failure occurs, a transaction is only atomic within one database. If more than one database were written to within a single transaction, one database might be committed and the other rolled back in the event of a failure.

There is a compile-time limit of 10 attached database files by default. This can be increased to up to 255 concurrent databases by modifying the following line in src/sqliteInt.h:

#define MAX_ATTACHED 10

To detach an attached database, the syntax is simply

DETACH [DATABASE] database-name


Manipulating Data

Next we'll look at how records can be added to a database and demonstrate different ways of using the INSERT command, and examine the syntax of the SQL UPDATE and DELETE commands.

Transactions

Any change to a SQLite database must take place within a transactiona block of one or more statements that alter the database in some way. Transactions are the way in which a robust database system ensures that either all or none of the requests to alter the database is carried out; it can never be just partially completed. This property of a database is called atomicity.

Whenever an INSERT, UPDATE, or DELETE command is issued, SQLite will begin a new transaction unless one has already been started. An implicit transaction lasts only for the duration of the one statement but ensures that, for instance, an UPDATE affecting many rows of a large table will always carry out the action on every row orin the unlikely event of a system failure while processing this commandnone of them. The database will not reflect a change to any row until every row has been updated and the transaction closed.

A transaction can be started from SQL if you want to make a series of changes to the database as one atomic unit. This is the syntax of the BEGIN TRANSACTION statement:

BEGIN [TRANSACTION [name]] [ON CONFLICT conflict-algorithm]

The transaction name is optional and, currently, is ignored by SQLite. The facility to provide a transaction name is included for future use if the ability to nest transactions is added. Currently only one transaction can be open at a time. In fact the keyword trANSACTION is also optional, but is included for readability.

An ON CONFLICT clause can be specified to override the default conflict resolution algorithm specified at the table level, but can be superseded itself by the OR clause of an INSERT, UPDATE, or DELETE statement.

To end a transaction and save changes to the database, use COMMIT TRANSACTION. The optional transaction name may be specified. To abort a transaction without any of the changes being stored, use ROLLBACK TRANSACTION.

Inserting Data

There are two versions of the syntax for the INSERT statement, depending on where the data to be inserted is coming from.

The first syntax is the one we have already used in Chapter 2, to insert a single row from values provided in the statement itself. The second version is used to insert a dataset returned as the result of a SELECT statement.

INSERT Using VALUES

The syntax for a single-row insert using the VALUES keyword and a list of values provided as part of the statement is as follows:

INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
VALUES (value-list)

Although all our examples so far have included a column-list, it is actually optional. Where no column-list is provided, the value-list is assumed to contain one value for each column in the table, in the order they appear in the schema.

This can be a useful shortcut when you are adding data; for instance because we know the column-list is a name and then a color, a record can be inserted into the vegetables table simply using this format:

sqlite> INSERT INTO vegetables VALUES ('mushroom', 'white');

However if the schema of the table is not what you are expecting, the INSERT will fail with an error. SQLite would not make any assumption as to which columns you are referring to.

Because SQLite does not have an ALTER TABLE command, it is much harder to change a schema after a table has been created than it is with other database engines that include this command. We'll see a workaround for ALTER TABLE in the following example, and if for any reason the vegetables table had been expanded to include three columns, the same INSERT statement would produce this error:

sqlite> INSERT INTO vegetables VALUES ('mushroom', 'white');
SQL error: table vegetables has 3 columns but 2 values were supplied

Therefore it is good practice to always include the column-list in an INSERT statementalso known as performing a full insert.

The OR keyword is used to specify a conflict resolution algorithm in the same way we saw for the CREATE TABLE statement. The list of algorithms and their behavior is identical, but the keyword OR is used instead of ON CONFLICT to give a more natural-sounding syntax.

The conflict algorithm in the OR clause of an INSERT statement has the highest precedence possible, and will override any other setting present at the table or transaction level.

INSERT Using SELECT

The syntax to insert the result of a SELECT query into another table is as follows:

INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)] select-statement

The select-statement should return a dataset with the same number and order as the columns specified in the column-list (or every column in the destination table if no column-list is supplied). The full syntax of the SELECT statement is available, and any number of rows can be returned.

As with INSERT ... VALUES, the column-list is optional but including it is highly advisable.

Updating Data

The syntax of the UPDATE statement in SQLite is as follows:

UPDATE [OR conflict-algorithms] [database-name .] table-name
SET assignment [, assignment]*
[WHERE expr]

One or more assignments can be performed within the same statement upon the same subset of data, defined by the optional WHERE clause. An assignment is defined as

column-name = expr

Although the WHERE clause is not required, it is usually desirable. The following example would assign the value of color to green for every row in the table, when in fact we probably only meant to update one or a few records.

sqlite> UPDATE vegetables
   ...> SET color = 'green';

The WHERE clause can be as simple or complex as necessary, and all the conditional elements that can be used in the WHERE clause of a SELECT statement can be used here.

It is not logical to join two or more tables when performing an UPDATE; however, subselects can be used in the WHERE clause, as in the following example:

sqlite> UPDATE mytable
   ...> SET myfield = 'somevalue'
   ...> WHERE mykey IN (
   ...>   SELECT keyfield
   ...>   FROM anothertable
   ...> );

The OR keyword is used in an UPDATE statement to specify a conflict resolution algorithm with the highest precedence possible, in the same way as with an INSERT.

Deleting Data

The DELETE statement is used to remove rows from a database. Its syntax is

DELETE FROM [database-name .] table-name [WHERE expr]

As with the UPDATE statement, the WHERE clause is optional but is usually desiredperforming a DELETE on a table with no WHERE clause will empty the table. No column-list is required for a DELETE because the operation affects the entire row.

The WHERE clause can use the AND and OR operators to combine conditions and can use subselects to perform a DELETE operation conditional on the results of another query.

The following example modifies a query from Chapter 2 to remove records from the timesheets table where the project_code field does not correspond to a key in the projects table.

sqlite> DELETE FROM timesheets
   ...> WHERE project_code NOT IN (
   ...>   SELECT code
   ...>   FROM projects
   ...> );

Altering a Table Schema

There is no ALTER TABLE statement in SQLite; instead a table must be dropped and re-created with a new field added, with any data that you want to preserve extracted before the table is dropped and reloaded into the new structure.

A temporary table is the ideal place to hold such data, and the CREATE TABLE ... AS syntax gives us a very easy way to create a copy of an existing table. The syntax is simply

CREATE [TEMP | TEMPORARY TABLE] table-name AS select-statement

Let's suppose we want to add a new descriptive column to our vegetables table but without losing the data we have already created. The first step is to take a copy of the existing vegetables table to a new temporary table.

sqlite> CREATE TEMPORARY TABLE veg_temp
   ...> AS SELECT * FROM vegetables;

However, only the field specification has been copied when a table is created this way. The schema of the new table does not include any data type names or column constraints. It is not possible to give a set of column definitions when using CREATE TABLE ... AS in SQLite.

sqlite> .schema veg_temp
CREATE TEMP TABLE veg_temp(name,color);

Compare this to the schema of the original vegetables table, which we'll need for re-creating the table with the new field:

sqlite> .schema vegetables
CREATE TABLE vegetables (
name CHAR NOT NULL,
color CHAR NOT NULL DEFAULT 'green'
);

So now we can safely drop the old vegetables table and re-create it with our new field:

sqlite> DROP TABLE vegetables;
sqlite> CREATE TABLE vegetables (
   ...> name CHAR NOT NULL,
   ...> color CHAR NOT NULL DEFAULT 'green',
   ...> description CHAR
   ...> );

Finally, reinstate the copied data from the temporary table using the INSERT ... SELECT syntax:

sqlite> INSERT INTO vegetables (name, color)
   ...> SELECT name, color FROM veg_temp;

Loading Data from a File

The COPY command in SQLite was based on a similar command found in PostgreSQL and as a result is designed to read the output of the pg_dump command to facilitate data transfer between the two systems.

However, COPY can also be used to load data from most delimited text file formats into SQLite. It has the following syntax:

COPY [OR conflict-algorithm] [database-name .] table-name FROM filename
[USING DELIMITERS delim]

The destination table table-name must existthough it need not be emptybefore the COPY operation is attempted, and either filename must be in the current directory or a full path given.

Each line in the input file will become a record in the table, with each column separated by a tab character, unless a different delimiter character is specified in the USING DELIMITERS clause.

If a tabor the specified delimiterappears within a data column, it must be escaped with a backslash character. The backslash itself can appear in the data if it is escaped itself; in other words it will appear as two consecutive backslash characters.

The special character sequence \N in the data file can be used to represent a NULL value.

Tab is used to separate columns in the output of pg_dump, so it is the default delimiter for the COPY command. Another popular format is comma-separated values (CSV). Listing 3.1 shows a comma-separated data file that can be loaded into the three-column vegetables table.

Listing 3.1. vegetables.csv
cucumber,green,Long green salad vegetable
pumpkin,orange,Great for Halloween
avocado,green,Can't make guacamole without it

The COPY command to load this data file into SQLite is

sqlite> COPY vegetables FROM 'vegetables.csv'
   ...> USING DELIMITERS ',';

You can instruct COPY to read data from the standard input stream instead of a file by using the keyword STDIN instead of a filename. A blank line, or a backslash followed by a period, is used to indicate the end of the input.

COPY permits an overriding conflict resolution algorithm to be specified after the OR keyword, as with INSERT and UPDATE.


Indexes

The subject of keys and indexes and how they can affect the performance of your database will be addressed in Chapter 4, "Query Optimization," but first we will examine the syntax for creating and finding information on table indexes.

Creating and Dropping Indexes

The CREATE INDEX command is used to add a new index to a database table, using this syntax:

CREATE [UNIQUE] INDEX index-name
ON [database-name .] table-name (column-name [, column-name]*)
[ON CONFLICT conflict-algorithm]

The index-name is a user-provided identifier for the new index and must be unique across all database objects. It cannot take the same name as a table, view, or trigger. A popular naming convention is to use the table name and the column name(s) used for the index key separated by an underscore character.

To add an index to the color column of the vegetables table, we would use the following command.

sqlite> CREATE INDEX vegetables_color
   ...> ON vegetables(color);

The syntax of column-name allows for a sort order to be given after each column name, either ASC or DESC; however, currently in SQLite this is ignored. At the present time, all indexes are created in ascending order.

Removing an index is done with reference to the identifier given when it was created, which you can always find by querying the sqlite_master table if you cannot remember it.

sqlite> SELECT * FROM sqlite_master
   ...> WHERE type = 'index';
    type = index
    name = vegetables_color
tbl_name = vegetables
rootpage = 10
     sql = CREATE INDEX vegetables_color
ON vegetables(color)

The DROP INDEX command works as you might expect:

sqlite> DROP INDEX vegetables_color;

Don't worry if you misread the sqlite_master output and use the table name instead of the index name. SQLite only allows you to drop indexes with the DROP INDEX command and tables with the DROP TABLE command.

sqlite> DROP INDEX vegetables;
SQL error: no such index: vegetables

UNIQUE Indexes

The UNIQUE keyword is used to specify that every value in an indexed column is unique. Where an index is created on more than one column, every permutation of the column values has to be unique, even though the same value may appear more than once in its own column.

Since we have already inserted several vegetables of the same color into the table, SQLite will give an error if we attempt to create a unique index on the color field.

sqlite> CREATE UNIQUE INDEX vegetables_color
   ...> ON vegetables(color);
SQL error: indexed columns are not unique

The ON CONFLICT clause at the index level is only relevant for a UNIQUE index; otherwise, there will never be a conflict on the data it applies to. The conflict resolution algorithm is used when an INSERT, UPDATE, or COPY statement would cause the unique constraint of the index to be violated. It cannot be used in the preceding CREATE UNIQUE INDEX statement to force a unique index onto a column containing multiple values.

The default conflict resolution algorithm is ABORT, and the same list of algorithms is permitted for indexes as in the CREATE TABLE statement.


Views

A view is a convenient way of packaging a query into an object that can itself be used in the FROM clause of a SELECT statement.

Creating and Dropping Views

The syntax for CREATE VIEW is shown next.

CREATE [TEMP | TEMPORARY] VIEW view-name AS select-statement

The select-statement can be as simple or as complex as necessary; it could return the subset of a single table based on a conditional WHERE clause, or join many tables together to form a single object that can be more easily referenced in SQL.

To drop a view, simply use the DROP VIEW statement with the view-name given when it was created.

A view is not a table. You cannot perform an UPDATE, INSERT, COPY, or DELETE on a view, but if the data in one of the source tables changes, those changes are reflected instantly in the view.

Using Views

The following example shows a view based on the demo database tables employees and employee_rates using a query that returns the current rate of pay for each employee.

sqlite> CREATE VIEW current_pay AS
   ...> SELECT e.*, er.rate
   ...> FROM employees e, employee_rates er
   ...> WHERE e.id = er.employee_id
   ...> AND er.end_date IS NULL;

We can then query the new view directly, even adding a new condition in the process:

sqlite> SELECT * FROM current_pay
   ...> WHERE sex = 'M';
id    first_name  last_name   sex  email                     rate
----  ----------  ----------  ---  ------------------------  ------
101   Alex        Gladstone   M    alex@mycompany.com        30.00
103   Colin       Aynsley     M    colin@mycompany.com       25.00

The column names in a view are the column names from the table. Where an expression is used, SQLite will faithfully reproduce the expression as the column heading.

sqlite> CREATE VIEW veg_upper AS
   ...> SELECT upper(name), upper(color)
   ...> FROM vegetables;
sqlite> SELECT * FROM veg_upper LIMIT 1;
upper(name)|upper(color)
CARROT|GREEN

However, the column in the view cannot actually be called upper(name). As shown in the following example, SQLite will attempt to evaluate the upper() function on the nonexistent name column.

sqlite> SELECT upper(name) from veg_upper;
SQL error: no such column: name

Column aliases can be used to give an explicit name to a column so that they can be referenced within a subsequent query.

sqlite> CREATE VIEW veg_upper AS
   ...> SELECT upper(name) AS uppername, upper(color) AS uppercolor
   ...> FROM vegetables;
sqlite> SELECT * FROM veg_upper
   ...> WHERE uppercolor = 'ORANGE';
uppername|uppercolor
CARROT|ORANGE
PUMPKIN|ORANGE

Note

When a view includes two columns with the same namewhether it is the same column selected twice from one table, or once each from two tables that happen to share a column nameSQLite will modify the column names in the view unless aliases are used. A duplicate column will be suffixed with _1 the first time it appears, _2 the second time, and so on.

SQLite does not validate the select-statement SQL in CREATE VIEW. You will only know if there is an error in the SELECT when you come to query the new view. The view's SELECT statement is effectively substituted into the query at the point where view-name appears, so the errors displayed may not appear to reflect the query you typed.

The following example creates a view with a deliberate errorthere is no column entitled shape in vegetablesand shows that the error is not detected until you query the view.

sqlite> CREATE VIEW veg_error AS
   ...> SELECT shape FROM vegetables;
sqlite> SELECT * from veg_error;
SQL error: no such column: shape


Triggers

A trigger is an event-driven rule on a database, where an operation is initiated when some other transaction (event) takes place. Triggers may be set to fire on any DELETE, INSERT, or UPDATE on a particular table, or on an UPDATE OF particular columns within a table.

Creating and Dropping Triggers

The syntax to create a trigger on a table is as follows:

CREATE [TEMP | TEMPORARY] TRIGGER trigger-name
[BEFORE | AFTER] database-event ON [database-name .]table-name
trigger-action

The TRigger-name is user-specified and must be unique across all objects in the databaseit cannot share the same name as a table, view, or index.

The trigger can be set to fire either BEFORE or AFTER database-event; that is, either to pre-empt the transaction and perform its action just before the UPDATE, INSERT, or DELETE takes place, or to wait until the operation has completed and then immediately carry out the required action.

If the database-event is specified as UPDATE OF column-list, it will create a trigger that will fire only when particular columns are affected. The trigger will ignore changes that do not affect one of the listed columns.

The TRigger-action is further defined as

[FOR EACH ROW | FOR EACH STATEMENT] [WHEN expression]
BEGIN
 trigger-step; [trigger-step;] *
END

At present only FOR EACH ROW TRiggers are supported, so each trigger stepwhich may be an INSERT, UPDATE, or DELETE statement or SELECT with a function expressionis performed once for every affected row in the transaction that causes the trigger to fire. The WHEN clause can be used to cause a trigger to fire only for rows for which the WHEN clause is true. The WHEN clause is formed in the same way as the WHERE clause in a SELECT statement.

The WHEN clause and any trigger-steps may reference elements of the affected row, both before and after the trigger action is carried out, as OLD.column-name and NEW.column-name respectively. For an UPDATE action both OLD and NEW are valid. An INSERT event can only provide a reference to the NEW value, whereas only OLD is valid for a DELETE event.

An ON CONFLICT clause can be specified in a trigger-step; however, any conflict resolution algorithm specified in the statement that causes the trigger to fire will override it.

As you might expect, the syntax to drop a trigger is simply

DROP TRIGGER [database-name .] table-name

If you forget the name of a trigger, you can query sqlite_master using type = 'trigger' to find all the triggers on the current database.

Using Triggers

In the last chapter we mentioned that triggers could be used to implement a cascading delete, so that rows from a table that referenced a foreign key would also be deleted if the foreign key were deleted from its own table. The trigger in the following example shows how this could be implemented on the demo database to delete entries from the timesheets table if the foreign key project_code is deleted from the projects table.

sqlite> CREATE TRIGGER projcode_cascade
   ...> AFTER DELETE ON projects
   ...> BEGIN
   ...>   DELETE FROM timesheets WHERE project_code = OLD.code;
   ...> END;

Similarly, we could create a trigger that maintains data integrityif the project code changes in the projects table, the child records in timesheets will be updated to reflect the new foreign key value.

sqlite> CREATE TRIGGER projcode_update
   ...> AFTER UPDATE OF code ON projects
   ...> BEGIN
   ...>   UPDATE timesheets
   ...>   SET project_code = NEW.code
   ...>   WHERE project_code = OLD.code;
   ...> END;

A quick test verifies that this trigger is working as we want it to:

sqlite> UPDATE projects
   ...> SET code = 'NEWCODE'
   ...> WHERE code = 'ABCCONS';
sqlite> SELECT count(*)
   ...> FROM timesheets
   ...> WHERE project_code = 'NEWCODE';
count(*)
----------
3

Interrupting a Trigger

Within the trigger-steps it is possible to interrupt the command that caused the trigger to fire and execute one of the conflict resolution algorithms available in an ON CONFLICT clause. This is done using the RAISE() function, which can be invoked using a SELECT statement as one of the following:

RAISE (ABORT, error-message) |
RAISE (FAIL, error-message) |
RAISE (ROLLBACK, error-message) |
RAISE (IGNORE)

Issuing an ABORT, FAIL, or ROLLBACK within a trigger will cause the transaction to exit and take the relevant action, and the error-message parameter is returned to the user.

We could use this behavior to prevent a project code from being deleted from the projects table while rows exist in timesheets that use it as a foreign key, rather than the rather destructive cascading delete.

sqlite> CREATE TRIGGER projcode_rollback
   ...> BEFORE DELETE ON projects
   ...> WHEN OLD.code IN (
   ...>   SELECT project_code FROM timesheets
   ...> )
   ...> BEGIN
   ...> SELECT RAISE(ROLLBACK, 'Timesheets exist for that project code');
   ...> END;

An attempted DELETE will produce an error:

sqlite> DELETE FROM projects WHERE code = 'NEWCODE';
SQL error: Timesheets exist for that project code

We can also verify that the DELETE transaction was rolled back.

sqlite> SELECT * FROM projects
   ...> WHERE code = 'NEWCODE';
code        client_id   title                start_date  due_date
----------  ----------  -------------------  ----------  ----------
NEWCODE     502         Ongoing consultancy  20030601

Using RAISE(IGNORE) causes the current trigger to be abandoned; however, any changes made up to that point will be saved and if the trigger was fired as the result of another trigger, that outer trigger's execution will continue.

Creating a Trigger on a View

The syntax for using triggers on views is slightly different than with tables. This functionality is provided as a way of intercepting INSERT, UPDATE, and DELETE operations on a view, usually to simulate that action by executing the actual steps necessary to make the requested data change appear in the view. Because a view may join two or more tables, a number of steps may be required.

The syntax for creating a trigger on a view is

CREATE [TEMP | TEMPORARY] TRIGGER trigger-name
INSTEAD OF database-event ON [database-name .] view-name
trigger-action

As before, database-event may be DELETE, INSERT, UPDATE, or UPDATE OF column-list, and the trigger-action is one or more SQL operations contained between the keywords BEGIN and END.


Working with Dates and Times

In our sample database we have chosen to use integers for columns that store a date value, represented by the format YYYYMMDD. This format is fairly readable and, because the most significant part (the year) comes first, allows arithmetic comparisons to be performed. For instance just as February 29th 2004 is earlier than March 1st, 20040229 is a smaller number than 20040301.

This technique is not without its limitations. First, there is no validation on the values stored. Although February 29th is a valid date in the leap year 2004, it does not exist three years out of four and the value 20050229 is not a real date, yet could still be stored in the integer column or compared to a real date.

In fact even if you used a trigger to make the number eight digits long and also fall within a sensible year range, there are many values that could still be stored that do not represent dates on the calendar. Very strict checking would be required in your application program to ensure such date information was valid.

Similarly, you cannot perform date arithmetic using integer dates. Although 20040101 + 7 gives a date seven days later, 20040330 + 7 would give a number that looks like March 37th.

We have not even looked at a data type to store a time value yet, but the same limitations apply if a numeric field is used. SQLite contains a number of functions that allow you to work with both dates and times stored as character strings, allowing you to manipulate the values in useful ways.

Valid Timestring Formats

SQLite is fairly flexible about the format in which you can specify a date and/or time. The valid time string formats are shown in the following list:

  • YYYY-MM-DD

  • YYYY-MM-DD HH:MM

  • YYYY-MM-DD HH:MM:SS

  • YYYY-MM-DD HH:MM:SS.SSS

  • HH:MM

  • HH:MM:SS

  • HH:MM:SS.SSS

  • now

  • DDDD.DDDD

For the format strings that only specify a time, the date is assumed to be 2000-01-01. Where no time is specified, midday is used. Simply using the string now tells SQLite to use the current date and time.

The format string DDDD.DDDD represents a Julian day numberthe number of days since noon on November 24, 4714 BC, Greenwich Mean Time. SQLite uses Julian date format internally to manipulate date and time values.

Displaying a Formatted Date and Time

The core date and time function in SQLite is strftime(), which has the following prototype:

strftime(format, timestring, modifier, modifier, ...)

This function is based upon the C function strftime() and the format parameter will accept most, although not all, of the same conversion specifiers. The following example shows how a date can be reformatted to MM/DD/YY format using strftime().

sqlite> SELECT strftime('%m/%d/%Y', '2004-10-31');
10/31/2004

Table 3.3 lists the conversions that can be performed by SQLite on a timestring.

Table 3.3. Date and Time Conversion Specifiers

String

Meaning

%d

Day of month, 01-31

%f

Fractional seconds, SS.SSS

%H

Hour, 00-23

%j

Day of year, 001-366

%J

Julian day number, DDDD.DDDD

%m

Month, 00-12

%M

Minute, 00-59

%s

Seconds since 1970-01-01 (unix epoch)

%S

Seconds, 00-59

%w

Day of week, 0-6 (0 is Sunday)

%W

Week of year, 01-53

%Y

Year, YYYY

%%

% symbol

Date and Time Modifiers

Given one or more optional modifier arguments, strftime() can perform a calculation on the date given in timestring.

To add or subtract a period of time, the days, hours, minutes, seconds, months and years modifiers can be used, as shown in these examples:

sqlite> SELECT strftime('%Y-%m-%d', '2004-10-31', '+7 days');
2004-11-07

sqlite> SELECT strftime('%H:%M', '22:00', '+12 hours');
10:00

sqlite> SELECT strftime('%Y-%m-%d %H:%M:%S',
                        '2004-01-01 00:00:00', '-1 second', '+1 year');
2004-12-31 23:59:59

Note

The modifier keywords can be written as either singular or plural. In the last of the preceding examples, we used 1 second and 1 year rather than 1 seconds and 1 years for readability. SQLite does not understand English grammar, so either is always acceptable.

In these examples we have used the same output format as the original timestring to return the date information in a format that can be recognized by SQLite. You should only format the date differently when you want to display it in your application in a particular way.

To save having to enter the same format strings repeatedly when working with dates, SQLite provides four convenience functions that call strftime() with predefined formats.

Use date() to return a date with the format string %Y-%m-%d and time() to return a time as %H:%S. The function datetime() returns the date and time using these two formats combined. Finally julianday() uses the %J format specifier to return the Julian day number.

The arguments to all four functions are the same as strftime() except that the format argument is omitted. The following example uses datetime() to produce a more concise SQL statement:

sqlite> SELECT datetime('2004-01-01 00:00:00', '-1 second', '+1 year');
2004-12-31 23:59:59

Other modifiers allow you to adjust a date or time to the nearest significant value. Specifying start of month, start of year, or start of day will decrease the value given in timestring to midnight on the first of the month or year, or on that day respectively.

When executed on any day during 2004, the start of year modifier returns 2004-01-01, as shown in the following example:

sqlite> SELECT datetime('now', 'start of year');
2004-01-01 00:00:00

Modifiers are applied to timestring in the order they appear in the statement, as shown in the following example. Note that had the second statement been executed on the last day of the month, the result would have been differentthe start of the following month would have been returned.

sqlite> SELECT datetime('now', 'start of month', '+1 day');
2004-07-02 00:00:00

sqlite> SELECT datetime('now', '+1 day', 'start of month');
2004-07-01 00:00:00

Any number of modifiers can be combined, giving you considerable power when working with dates and times. For instance, the last day of the current month can be found using three modifiers in succession.

sqlite> SELECT date('now', '+1 month', 'start of month', '-1 day');
2004-07-31

Handling Different Time Zones

The locale settings of your system will determine which time zone is used when displaying dates and times; however, the underlying system clock will use Coordinated Universal Time (UTC), also known as Greenwich Mean Time (GMT)Greenwich Mean Time (GMT). Your time zone setting will specify a number of hours to be added to or subtracted from the UTC value to arrive at the correct local time.

For instance, to find the local time in New York you have to subtract five hours from UTC, or four hours during daylight savings time. Even in Greenwich, the local time is UTC + 1 hour during the summer months.

To convert between UTC and local time values when formatting a date, use the utc or localtime modifiers. The following examples were run on a system with the timezone set to Eastern Standard Time (UTC 5 hours).

sqlite> SELECT time('12:00', 'localtime');
2000-01-01 07:00:00

sqlite> SELECT time('12:00', 'utc');
2000-01-01 17:00:00


SQL92 Features Not Supported

We finish this chapter on SQLite's implementation of the SQL language by looking at features of the ANSI SQL92 standard that are not currently supported by SQLite.

  • Although the CREATE TABLE syntax permits an optional CHECK clause to be present, the CHECK constraint is not enforced.

  • The keywords FOREIGN KEY are allowable in a CREATE TABLE statement; however, this currently has no effect.

  • Subqueries must return a static data set, and they may not refer to variables in the outer queryalso known as correlated subqueries.

  • All triggers are currently FOR EACH ROW, even if FOR EACH STATEMENT is specified.

  • Views are read-only, even when they select only from one table. However, an INSTEAD OF trigger can fire on an attempted INSERT, UPDATE, or DELETE to a view and deal with the transaction in the desired manner.

  • INSTEAD OF triggers are allowed only on views, not on tables.

  • Recursive triggerstriggers that trigger themselvesare not supported.

  • The ALTER TABLE statement is not present; instead a table must be dropped and re-created with the new schema.

  • Transactions cannot be nested.

  • count(DISTINCT column-name) cannot be used. However, this can be achieved by selecting a count() from a subselect of the desired table that uses the DISTINCT keyword.

  • All outer joins must be written as LEFT OUTER JOIN. RIGHT OUTER JOIN and FULL OUTER JOIN are not recognized.

  • The GRANT and REVOKE commands are meaningless in SQLitethe only permissions applicable are those on the database file itself.


Chapter 4. Query Optimization

Now that you are familiar with the SQL syntax implemented by SQLite, let's take a look at the way in which your tables are created and how the way your queries are written can affect the performance of your database.


Keys and Indexes

In the preceding chapter you met the CREATE INDEX statement, used to add an index to one or more columns of a database table.

Indexes are used to preserve a particular sort order on a column within the database itself, enabling queries in which the WHERE clause references that column directly to operate much faster. Rather than scanning a table's data from top to bottom and pulling out just the rows that match the given criteria, the index tells SQLite exactly where to look in that table to find the matching rows.

What an Index Does

A database index is not an abstract concept; you certainly have come across one before. Let's consider the index in this book, for instance, and think about when you would use it.

If you were trying to find references to the API call sqlite_exec(), for example, you might already expect to find this in Chapter 6, "The C/C++ Interface"but bear in mind that a computer program cannot think for itself. So for a moment pretend you don't have the benefit of this knowledge.

To make sure you find every reference in the book, the systematic and thorough approach is to start from page one, line one and read each page a line at a time until the end of the book, noting each reference as you come across it. Clearly, this would be a time-consuming process.

Fortunately the production team behind this book has created a comprehensive index, with topics and terms used throughout the book listed alphabetically and page references given for each entry. You've seen an index before and know how to use one, so you already know that it's much quicker to look up sqlite_exec() in the index and jump to each page in turn to find the topic you are looking for than to scan every single printed word.

How Indexes Work in SQLite

A database index works much the same way as the index in a book. Suppose you have a table containing a list of names and telephone numbers, and the data has been built up over time as you come into contact with new people. It's very likely that some of your acquaintances will share a surname, but certainly your records were not added to the table in alphabetical order.

Imagine you want to find the details for someone called Brown. The SQL SELECT statement might look like this:

SELECT * FROM contacts
WHERE last_name = 'Brown';

Without an index, the database has to look at each record in the database in turn and compare it to the string Brown. The more popular you are, the larger your contacts database will be and the longer this query will take to complete. This process is known as a full table scan.

However, having an index on the last_name column means that SQLite will know how to sort the records in that table alphabetically by surname, and consequently it can pick out the matching values of last_name without even looking at the other rows in the table.

Such an index could be created on the contacts table as follows:

CREATE INDEX contacts_last_name ON contacts(last_name);

Remember that each index has to be given its own unique identifier, and that an index name cannot share the same name as a table, view, or trigger in the same database.

Note

Although the CREATE INDEX syntax allows you to specify a sort orderASC or DESCall indexes are currently created in ascending order. This feature is reserved for possible future use.

Indexing Multiple Columns

An index can be created on more than one column at a timeknown as a clustered index. The order in which the columns are specified in the CREATE INDEX statement determines the sort order for the indexrecords are sorted on the first column initially, then on the second column where values of the first column are the same.

Think of how a telephone directory is organized. People appear in order of their surname, but because there are many people with the same surname, the listings are further ordered by first name or initials.

In SQLite, this kind of index for the contacts table would look like this:

CREATE INDEX contacts_name ON contacts(last_name, first_name);

A WHERE clause with conditions on both columns would find results very quickly using this index. In fact, because the primary sort order is last_name, a query that only references last_name would also perform well using this index.

The ordering of the columns for a clustered index is important. A condition imposed on first_name only would be no better off using this index than it would doing a full table scan.

Unique Indexes

You can specify the optional keyword UNIQUE in the CREATE INDEX statement to indicate that each value in the indexed columnor each permutation of values in a clustered indexonly appears in the table once.

Using CREATE UNIQUE INDEX is the same as imposing a UNIQUE constraint in the table schema, but it can be done at any time after the table has been created. If you attempt to insert a duplicate value, SQLite will return an error and the insert will fail.

In fact, using the UNIQUE attribute in the CREATE TABLE statement causes SQLite to create a unique index automatically on this column as shown in the following example. Note that the sql column is empty because no CREATE INDEX command was entered.

sqlite> create table mytable (
   ...>   a INTEGER PRIMARY KEY
   ...> );
sqlite> select * from sqlite_master where type = 'index';
type        name                   tbl_name    rootpage    sql
----------  ---------------------  ----------  ----------  ----------
index       (mytable autoindex 1)  mytable     3

Note

If a column is declared as INTEGER PRIMARY KEY, it is used internally as the key for the table structure, so the records in the table are ordered on that column by default. Although it does not cause an index to show up in sqlite_master, the column will behave as if it were indexed. Explicitly creating an index on an INTEGER PRIMARY KEY column uses additional database resources without providing any extra benefits.

When to Create an Index

An index can be created or dropped at any time without affecting the data contained in that table. A number of factors determine when it becomes beneficial to create an index.

Usually you will create indexes on columns used in conditions in WHERE statements. Whether the purpose of the index is to restrict a dataset from one table or to perform a join, generally speaking SQLite will be able to do its work much faster if key fields are indexed.

However, very small tables are often not made any faster by adding an index and in some cases, the index can actually slow things down. It can be just as fast for SQLite to do a quick scan of all rows than to look up the sort order from the index.

If a table has data inserted frequently but not queried very often, use indexes sparingly. Although indexes can speed up queriesand UPDATE statements with a WHERE clause that references the indexed columninserting data is slower when an index is present. In addition to adding a record to the table, an INSERT requires the index to be updated to reflect the position of the new data within the sort order of each indexed column.

Think of the filing cabinet in a traditional office, with customer files arranged alphabetically in the drawers. When a new document is filed, it takes a moment to locate the correct drawer and file and make sure that the item is put away in the correct place. However, when it comes to accessing documents in the future, everything relating to a particular customer is in the same place and can be found very quickly.

The alternativea large stack of paperslets you file things away much faster by just placing them on top of the pile, but looking for specific information becomes a daunting task. Just as most offices use a filing cabinet rather than an unsorted pile, most database tables can benefit from indexes when used correctly.

Don't go overboard with the number of indexes you create. The biggest performance benefits come from careful index selection based on the nature of the data stored in tables and the ways in which the application interacts with it.

If you do have too many columns indexed, it is possible that SQLite will not pick the best one for any particular query. Only one index can be used on each table at a time, so a clustered index is the only way to apply the capabilities of an index across two columns simultaneously. We will see an example of this later.

When Indexes Can Be Used

You also need to bear in mind what kinds of conditions can and cannot make use of an index. Most operators will use an index if one is available provided that the column value is not modified before it is compared.

The following examples will all make use of an index on mytable.myfield if there is one.

SELECT * FROM mytable WHERE myfield = 'xyz';
SELECT * FROM mytable WHERE myfield > 2000;
SELECT * FROM mytable WHERE myfield IN ('val1', 'val2');
SELECT * FROM mytable WHERE myfield IN (SELECT somefield FROM anothertable)

However, the BETWEEN and LIKE operators will not use an index at any time. Such queries can usually be rewritten if using an available index would be beneficial. For instance

SELECT * FROM mytable WHERE myfield BETWEEN 10 and 20;

becomes

SELECT * FROM mytable WHERE myfield >= 10 AND myfield <= 20;

Similarly, a LIKE operator that matches the first few characters of a string can be written using two inequality operators.

SELECT * FROM mytable WHERE myfield LIKE 'sql%';

becomes

SELECT * FROM mytable WHERE myfield >= 'sql' AND myfield < 'sqm';

Note

A LIKE operator with characters after a wildcard symbol cannot be rewritten as a pair of inequalities. This would be like trying to find people whose surnames end with a "y" in the telephone directory.

These examples show that two conditions on the same column combined with an AND can still make use of an index. However, this is not true of the OR keywordthe IN operator should be used instead wherever possible.

SELECT * FROM mytable WHERE myfield = 'abc' OR myfield = 'xyz';

becomes

SELECT * FROM mytable WHERE myfield IN ('abc', 'xyz');

If some operation is performed on a column value in the condition, SQLite will not be able to use an index. For instance, the following queries will be evaluated using a full table scan, regardless of any indexes on myfield:

SELECT * FROM mytable WHERE myfield % 2 = 1;
SELECT * FROM mytable WHERE substr(myfield, 0, 1) = 'w';
SELECT * FROM mytable WHERE length(myfield) < 5;

Tweaking the Table List

To join two or more tables together, the rows have to be combined before the results are returned, and the SQLite engine does this using nested loops. The hierarchy of the loops is determined by the order in which table names are specified in the FROM list, so that the leftmost table becomes the outer loop and the rightmost table becomes the inner loop.

Knowing a little about the tables you are performing the SELECT on can help you use this feature of SQLite to increase the speed of the query. If you use the tables that will produce the smallest number of rowsafter any WHERE clause conditions that directly apply to that table have been appliedthe outer loop will be as small as possible, and conditions evaluated on the inner loops will be performed a smaller number of times.

Benchmarking

Listing 4.1 shows a simple benchmarking shell script that can be used to execute a query repeatedly within sqlite and using the Unix time command to time execution.

Listing 4.1. benchmark.sh
#!/bin/sh
if [ $# -ne 3 ]; then
  echo "Usage: $0 database-file sql-file num-loops"
  exit;
fi
dbname=$1;
sqlfile=$2;
numloops=$3
tmpfile=/tmp/sqlite_benchmark.sql
> $tmpfile
echo "Generating temporary SQL file"
for i in 'seq 1 $numloops'; do
  cat $sqlfile >> $tmpfile
done
echo "Executing query $numloops times"
time sqlite $dbname < $tmpfile > /dev/null
rm -f $tmpfile

This script is quite crude but will do enough to tell us how fast one query is in relation to another. A shell script has been used here so as not to tie it to any particular programming API, but after you have read the chapters in Part II of this book, "Using SQLite Programming Interfaces," you will be able to create a benchmarking process that suits the environment you work in.

Note

The script in Listing 4.1 uses the seq command, which is part of the GNU coreutils package, included on Linux and certain other Unix systems as standard. If your operating system does not include this command, the package can be obtained from http://ftp.gnu.org/pub/gnu/coreutils/.

The shell script version can be quite slow because it involves a lot of disk writes to generate the temporary SQL file. Using a programming API, the query could be executed repeatedly in a loop construct within the same SQLite connection.

However, to perform a loop in the shell with sqlite called multiple times to execute the same query over and over again would carry a large overhead as the program has to start up, and open and close the database file for each iteration. As we are only interested in the time taken to perform the query, the file containing multiple queries is built up before sqlite is invoked and this work is not timed.

Create a text file containing the query to be analyzed and execute the program using the following syntax. Note that because the SQL query is passed to sqlite many times from one file, the terminating semicolon must be present.

./benchmark.sh database-file sql-file num-loops

The output will look something like this:

$ ./benchmark.sh words somefile.sql 10000
Generating temporary SQL file
Executing query 10000 times
real    0m6.814s
user    0m2.360s
sys     0m1.220s

Ideally the number of loops should be as high as you can bear to wait for the result to obtain a reliable average execution time. This will depend on the actual execution time of your query. The preceding example looped 10,000 times and took just a few seconds, but for a slow query even one loop might take longer than this.

Some Examples

The examples that follow use some relatively large sample tables to demonstrate that indexes can be used effectively. You can download a sample of the database from the Sams Publishing website at http://www.samspublishing.com/. Enter this book's ISBN (without the hyphens) in the Search box and click Search. When the book's title is displayed, click the title to go to a page where you can download the code if you want to work through these examples.

The first example has three tables, each containing 10,000 rows. This is not large compared to the sizes of databases SQLite is capable of handling, but is big enough for us to see the difference that an index can make.

The tables each contain a numeric integer between 1 and 10,000 and one word pulled randomly from a dictionary file. Table t1 uses an INTEGER PRIMARY KEY and contains each value from 1 to 10,000 in the id column. The schema for t1 is as follows:

CREATE TABLE t1 (
  num INTEGER PRIMARY KEY,
  word TEXT NOT NULL
);

Though word is not defined as UNIQUE in t1, the sample table was created in such a way that values of word are not duplicated. Therefore this table contains a list of 10,000 random English words, numbered sequentially.

CREATE TABLE t2 (
  num INTEGER NOT NULL,
  word TEXT NOT NULL
);

Table t2 contains one record for each value of word in t1; however, the num column contains a random integer between 1 and 10,000. As there is no UNIQUE constraint on num, the random way in which the data was generated has resulted in some numbers being duplicated and some being omitted from this table.

CREATE TABLE t3 (
  num INTEGER NOT NULL,
  word TEXT NOT NULL
);

The final table, t3, has a similar schema to t1 but with no INTEGER PRIMARY KEY. In fact data has been inserted so far in such a way that num is unique and every value from 1 to 10,000 is present, but to start with we do not want this column to be indexed.

Values of word were inserted randomly from the same dictionary file, which actually contains more than 15,000 words. Therefore some words from t1 appear twice or more in t3, some are omitted, and some values that do not appear in either of the other tables have found their way into t3.

This data is trivial, but it allows us to create some reasonably large joins on both num and word to see how indexes could be used. In the queries that follow, we use count(*) as the selected output because the number of rows returned will usually be large and returning the count will not significantly affect the query's execution time.

Let's start by looking at how the INTEGER PRIMARY KEY in t1 makes things faster, even though there is no visible index. Because the records in t1 are stored in the order of the num column, a comparison on this column is much faster than on t2 where the ordering of num is random.

Listing 4.2 shows query t1.sql for performing a very simple benchmark on these tables using the equals operator.

Listing 4.2. t1.sql
SELECT count(*) FROM t1
WHERE num = 3500 AND 4000;

The result of executing this query 1000 times is

$ ./benchmark.sh words t1.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m0.007s
user    0m0.000s
sys     0m0.010s

File t2.sql performs the exact same query on table t2. Here are the results:

$ ./benchmark.sh words t2.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m11.261s
user    0m9.270s
sys     0m1.930s

As you can see, the difference the INTEGER PRIMARY KEY makes is vast because SQLite knows for certain what the order of the records will be.

When table t3 was created, records were actually inserted in order of column num, but without the column being indexed, SQLite still has to perform a full table scan to be sure that it does not leave out any rows meeting the condition.

$ ./benchmark.sh words t3.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m10.898s
user    0m8.880s
sys     0m1.970s

If we create an index on t3.num, the benchmark results show speeds comparable to the those of the original query using the INTEGER PRIMARY KEY:

sqlite> CREATE INDEX t3_num_idx ON t3(num);
$ ./benchmark.sh words t3.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m0.029s
user    0m0.030s
sys     0m0.000s

Joining any two of these tables produces a potential 100 million rows, so this is where indexes can make a real impact. Consider the query in Listing 4.3, which joins tables t1 and t2 on the num field. As t2.num is not indexed, a full table scan must be performed on t2 for each row in t1.

Listing 4.3. joint1t2.sql
SELECT count(*) FROM t1, t2
WHERE t1.num = t2.num;

This query is unacceptably slow, in fact so slow that it's not even worth running multiple times with the benchmark script. Just one execution took over a minute to complete.

$ time sqlite words < joint1t2.sql
10000
real    1m3.822s
user    1m1.000s
sys     0m0.170s

If we join t1 and t3 instead using the same condition, this is much faster. As each row in t1 is iterated, SQLite can find the corresponding record in t3 almost immediately using the index we just created.

$ ./benchmark.sh words joint1t3.sql 100
Generating temporary SQL file
Executing query 100 times
real    0m4.721s
user    0m4.400s
sys     0m0.310s

Remember that we said the order in which tables appear in the FROM list is significanthere's an example. Let's take Listing 4.3 and reverse the table order as shown in Listing 4.4. This time, each row in t2 is examined in turn and joined to t1 using the condition specified. Because t1.num is an INTEGER PRIMARY KEY, SQLite can jump straight to the row in question.

Listing 4.4. joint2t1.sql
SELECT count(*) FROM t2, t1
WHERE t1.num = t2.num;
$ ./benchmark.sh words joint2t1.sql 100
Generating temporary SQL file
Executing query 100 times
real    0m4.435s
user    0m3.960s
sys     0m0.380s

Let's see what difference an index can make to the word column, which contains a variable-length character string, using the query shown in Listing 4.5.

Listing 4.5. t1word.sql
SELECT count(*) FROM t1
WHERE word = 'tickles';
$ ./benchmark.sh words t1word.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m8.533s
user    0m6.370s
sys     0m2.070s

If we add an index on t1.word and rerun the same test, the performance is much, much better.

sqlite> CREATE INDEX t1_word_idx ON t1(word);
$ ./benchmark.sh words t1word.sql 1000
Generating temporary SQL file
Executing query 1000 times
real    0m0.185s
user    0m0.160s
sys     0m0.030s

Our sample query just looks for a single word; in t1 and t2 the word "tickles" appears exactly once, so we could use a unique index if we want to. However, in t3 it actually appears three times, so trying to create a unique index gives a warning:

sqlite> CREATE UNIQUE INDEX t3_word_idx on t3(word);
SQL error: indexed columns are not unique

Instead we have to use a nonunique index on t3.word.

sqlite> CREATE INDEX t3_word_idx on t3(word);


The EXPLAIN Statement

There is a way to compare how different indexes affect the speed of certain queries without having to perform endless benchmark tests. However, it requires a little knowledge of the inner workings of SQLite.

The EXPLAIN command can be used to find out how an SQL query is parsed by SQLite and from that you can determine the way in which it will actually be executed. The output from EXPLAIN is a series of opcodes from the Virtual Database Engine, which we'll look at in more detail in Chapter 10, "General Database Administration."

When using sqlite the .explain command can be used to make the format of the output of EXPLAIN more readable. The following example shows the opcodes used to process a query on t2 using a WHERE condition on the non-indexed num column.

sqlite> .explain
sqlite> EXPLAIN SELECT word FROM t2 WHERE num = 1234;
addr  opcode        p1          p2          p3
----  ------------  ----------  ----------  -----------------------------------
0     ColumnName    0           0           word
1     Integer       0           0
2     OpenRead      0           4           t2
3     VerifyCookie  0           2979
4     Rewind        0           11
5     Column        0           0
6     Integer       1234        0           1234
7     Ne            1           10
8     Column        0           1
9     Callback      1           0
10    Next          0           5
11    Close         0           0
12    Halt          0           0

To work out whether an index is used or not, knowing the actual meaning of all these opcodes is not necessary. If you compare the preceding output to that for a similar query on t3 where num is indexed, you'll see the name of the index in the p3 column alongside an OpenRead opcode.

sqlite> EXPLAIN SELECT word FROM t3 WHERE num = 1234;
addr  opcode        p1          p2          p3
----  ------------  ----------  ----------  -----------------------------------
0     ColumnName    0           0           word
1     Integer       0           0
2     OpenRead      0           5           t3
3     VerifyCookie  0           2979
4     Integer       0           0
5     OpenRead      1           1077        t3_num_idx
6     Integer       1234        0           1234
7     NotNull       -1          10
8     Pop           1           0
9     Goto          0           22
10    MakeKey       1           0           n
11    MemStore      0           0
12    MoveTo        1           22
13    MemLoad       0           0
14    IdxGT         1           22
15    RowKey        1           0
16    IdxIsNull     1           21
17    IdxRecno      1           0
18    MoveTo        0           0
19    Column        0           1
20    Callback      1           0
21    Next          1           13
22    Close         0           0
23    Close         1           0
24    Halt          0           0

Because table and column names from the query will also appear in the p3 column, it is a good idea to have a strong naming convention for your indexes so that they stand out in this output. In this case we used t3_num_idx, which tells us exactly which table and column the index applies to.

We can use EXPLAIN to see which index SQLite chooses to use where two are availableremember that only one index can be used per table in a query. In the following examples, only the lines of output from EXPLAIN that correspond to an index name are given.

sqlite> EXPLAIN SELECT * FROM t3
   ...> WHERE num = 4000 AND word = 'soccer';
...
6     OpenRead      1           1682        t3_word_idx
...

So of the two indexes available on table t3, SQLite has chosen the index on t3.word to use for this query.

Let's take a moment to recap what indexes are in place on these tables. You can always find this information by querying the sqlite_master table:

sqlite> SELECT name, sql FROM sqlite_master
   ...> WHERE type = 'index';
name              sql
----------------  ------------------------------------
t3_num_idx        CREATE INDEX t3_num_idx ON t3(num)
t1_word_idx       CREATE INDEX t1_word_idx ON t1(word)
t3_word_idx       CREATE INDEX t3_word_idx on t3(word)

In fact the index that SQLite will choose where more than one is equally suitable is the one that appears last in the sqlite_master tableusually the one most recently created. A brand new index will also take precedence over older indexes for the duration of the current SQLite session and the database schema has not been re-read. A clustered index is considered more suitable than an index on a single column if all the columns in the index form part of the WHERE clause.

There may be occasions when you want to prevent use of the index SQLite would otherwise choose in order to force another index to be used. This is achieved by using an operator that does not affect the column's value.

To make sure the index on t3.word is used, add zero to the num column:

sqlite> EXPLAIN SELECT * FROM t3
   ...> WHERE num+0 = 4000 AND word = 'soccer';
...
6     OpenRead      1           1695        t3_word_idx
...

Similarly, to force use of the index on t3.num, use the concatenation operator to append an empty string to word:

sqlite> EXPLAIN SELECT * FROM t3
   ...> WHERE num = 4000 AND word||'' = 'soccer';
...
6     OpenRead      1           1077        t3_num_idx
...

If multiple indexes can be used in a query, each one will appear in the output from EXPLAIN. The following example shows that indexes are also used for a LEFT OUTER JOIN.

sqlite> EXPLAIN SELECT count(*) FROM t1
   ...> LEFT OUTER JOIN t3 ON t1.num = t3.num
   ...> WHERE t1.word in ('apple', 'banana');
11    OpenRead      2           927         t1_word_idx
15    OpenRead      3           2042        t3_num_idx

Using Transactions

You saw in Chapter 3, "SQLite Syntax and Use," that every change to the database must take place within a transaction, and that if one is not started explicitly with the BEGIN TRANSACTION command, SQLite will begin an implicit transaction whenever an INSERT, UPDATE, or DELETE statement is processed.

However, taking the time to create your own transactions can be well worth the effort. Each transaction requires separate low-level operations to open and close the journal file, and this is an unnecessary overhead when a number of changes to the database can be made in a single transaction.

Therefore rather than letting SQLite start several implicit transactions for you when performing a series of changes to the database, get into the habit of using BEGIN TRANSACTION and COMMIT TRANSACTION wherever such statements can be grouped together.

Note

If you are writing to a temporary table, the impact of using transactions is not so drastic. Because temporary tables are not designed to be stored beyond the current session, the disk writes are not flushed to the database file as regularly.

The VACUUM Statement

Any kind of disk access is faster if the data being read is contiguous on the drive, and fetching rows from SQLite is no exception.

When records are deleted from a table, SQLite blanks out the space they used to occupy but does not remove it from the database file. The same applies when a table or index is dropped. Because additional disk space does not need to be reallocated to the database file, a future INSERT statement that can occupy this space is slightly quicker, but not all data records are the same size, and before long the database will become fragmented.

The VACUUM command is SQLite's defragmenter. It works by systematically copying the contents of the database to a temporary fileleaving no empty spaceand then reloading the original file from this copy.

The SQL syntax allows a table name or index name argument to be passed to VACUUM for backward compatibility with much older versions of SQLite, however any argument given is now simply ignored. Issuing a VACUUM command will always clean up the entire database.

Tuning the Database Itself

This chapter has mostly covered optimizations that can be performed on individual SQL statements. Another way to improve the performance of SQLite is to tune certain database parameters to suit your application.

We will examine system-wide performance considerations in Chapter 10.


Part II: Using SQLite Programming Interfaces

5 The PHP Interface

6 The C/C++ Interface

7 The Perl Interface

8 The Tcl Interface

9 The Python Interface


Chapter 5. The PHP Interface

PHP is a server-side embedded scripting language for writing dynamic web pages that supports communication with many different databases, and SQLite is no exception.

In this chapter we will look at how SQLite support is activated in PHP, how to communicate with a database through a web page, and how custom functions can be added to SQLite through the PHP interface.


Configuring PHP for SQLite Support

Modern versions of PHP make it very easy to add in SQLite support, so we'll briefly look at the steps required for both Linux/Unix and Windows versions of PHP.

For more general assistance with PHP configuration on a particular platform, refer to the online documentation at http://www.php.net/manual/en/installation.php.

Configuring PHP for Linux/Unix

From PHP 5, the SQLite extension is bundled with the PHP distribution and will be enabled at compile time unless it is disabled with the --without-sqlite option.

You do not need to install any SQLite components before building PHP, but you will need to download the sqlite tool separately if you want to use it to examine your databases outside of PHP.

For earlier PHP versions, the official extension can be obtained from http://pecl.php.net/package/SQLite. This also includes source for libsqlite, so again there are no other prerequisites for adding the module to PHP. With PHP 4.3.0 and later, you can use the pear utility to automatically download and install the SQLite extension:

# pear install sqlite
downloading SQLite-1.0.2.tgz ...
Starting to download SQLite-1.0.2.tgz (362,412 bytes)
.........................................done: 362,412 bytes
51 source files, building
running: phpize
[...]
install ok: SQLite 1.0.2

The full output is quite lengthy, showing that the latest version of the extension has been downloaded, compiled, and copied to your PHP extensions directory.

For earlier versions of PHP, you can download and compile the extension by hand after downloading the latest version by following these steps:

# gzip d SQLite-1.0.2.tar.gz
# tar xf SQLite-1.0.2.tgz
# cd SQLite-1.0.2
# phpize
# ./configure
# make
# make install

With the extension installed, all you need to do to activate the new extension is add the following line to your php.ini file and restart the web server.

[sqlite]
extension="sqlite.so"

Alternatively you can also load the SQLite extension dynamically in each script that requires it with the dl() function.

dl("sqlite.so");

There should be no need to specify the full path to sqlite.so; the shared object file will be installed to the extension_dir value in php.ini.

Configuring PHP for Windows

The Windows DLL version of the SQLite extension is bundled with PHP5, and for earlier versions it can be downloaded from http://snaps.php.net/win32/PECL_STABLE/php_sqlite.dll.

Save this file to the PHP extensions directoryusually C:\php\extensionsand then activate the module by adding the following lines to your php.ini file and restarting Apache.

[sqlite]
extension=php_sqlite.dll

Checking for SQLite Support

If the SQLite extension is loaded, you can see some information about the module in the relevant section of the page generated by the phpinfo() function. Figure 5.1 shows the typical output when SQLite support is present.

Figure 5.1. Output of phpinfo() showing SQLite support in PHP.

[View full size image]

In the command-line PHP, you can check which modules are loaded with the m switch, for instance:

$ php -m | grep sqlite
sqlite

You can also check whether the extension is loaded programmatically using the extension_loaded() function, as demonstrated in Listing 5.1. This listing will attempt to load the extension if it is not already present, and exit if the attempt fails.

Listing 5.1. Check to Find Whether SQLite Extension is Available to PHP
<?php

if (!extension_loaded("sqlite")) {
  if (!dl("sqlite.so")) {
    echo "The SQLite extension could not be loaded.  Exiting.";
    exit;
  }
}

// Continue with SQLite-dependent code here

?>

Getting Information About the SQLite Extension

When you are sure the SQLite extension is available in your version of PHP, there are two functions available to return information about the version and encoding of the attached library.

These are the values we saw in Figure 5.1 in the output of phpinfo(). The functions allow you to use the respective values programmatically if you want to perform conditional actions based on the library version, for instance.

A call to sqlite_libversion() requires no arguments and returns a text string containing the version number of the linked library. On the system that the screenshot in Figure 5.1 was taken on, this would return the string 2.8.11. The function sqlite_libencoding() would return iso8559 on the same system.

By default PHP builds libsqlite with ISO-8859-1 encoding. In fact, this is not strictly ISO-8859-1it operates according to your system local settings. The alternative, UTF-8, is not fully supported by PHP at the time of writing and may produce unexpected problems.


Using the PHP SQLite Extension

Now we know that our PHP supports the SQLite interface, let's look at the set of commands that are available for communicating with a SQLite database.

Opening a Database

The PHP function to open a SQLite database is sqlite_open() and its prototype is given next:

resource sqlite_open ( string filename [, int mode [, string &errmessage]])

The filename parameter specifies the database name, and if the file specified does not exist it will be created. A relative or absolute path to the database file can be provided; otherwise, PHP looks for filename in the same directory as the script being executed. To open an in-memory database, use :memory:.

Although PHP can execute scripts from the command line, this chapter assumes that you are mostly using PHP in a web environment, where file permissions can be problematic. In order for PHP to open or to select from a SQLite database, the user under which the web server is running must have read permissions on the database file.

To perform an UPDATE, INSERT, or DELETE, that web server userid must also not only have write permissions on the database file itself but also must be able to create files within the same directory so that the journal file can be written.

Apache often runs as the user nobody or apache and so care needs to be takenparticularly on a shared systemif the database file is made writable to this user or, worse, if the file is made world-writable.

To prevent other users of the same web server from accessing your databases, PHP should be run in safe mode. An alternative is to use suPHPavailable from http://www.suphp.org/so that PHP can run as a number of different users on the same server.

The mode parameter to sqlite_open() specifies the mode of the file, and is octal 0666 by default. The intended use of this parameter is to open a database file in read-only mode, using mode 0444 for example.

The resource returned is a database handle if sqlite_open() is successful. On failure, the function returns FALSE and if the optional errmessage parameter is passed by reference it will contain a descriptive error message explaining why the database cannot be opened.

In Listing 5.2 we use sqlite_open() to open a new database webdb. To follow the rest of the examples in this chapter, make sure this database is created with the correct file permissions to be opened by PHP without any error.

Listing 5.2. Script to Open a Database with Error Checking
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {
  echo "SQLite database opened";
}

?>

PHP allows for a persistent database connection to be used via the sqlite_popen() function, which takes the same parameter list as sqlite_open().

Persistent connections reduce the overhead of repeatedly connecting to the same database. Where a persistent handle has already been opened to the specified database, that handle is made available to the script. If no persistent connection is available, a new connection is created.

Note

Persistent connections in PHP take advantage of the nature of multiprocess or multithreaded web servers. As long as PHP is built into Apache as a module or loaded into the web server using a supported SAPI, persistent connections will be available. If you are running PHP as a CGI wrapper, there is no persistence between page requests and although the sqlite_popen() function is still available, it will behave no differently than sqlite_open(). For more information, see http://php.net/manual/en/features.persistent-connections.php.

To close a SQLite database session, use sqlite_close(). The only parameter is the database resource returned by the function that opened the connection. If sqlite_popen() was used, the persistent connection will be closed and will no longer be available to other processes.

Passing Queries and Commands to SQLite

To pass an SQL command to SQLite, use the sqlite_query() function. Two parameters are requireda database connection resource and the query text itself. The prototype is given here:

resource sqlite_query (resource dbhandle, string query)

In fact the parameters can be specified in reverse order to that shown in the preceding example for compatibility with other PHP database extensions. However, the preferred syntax is to give the dbhandle parameter first for consistency with other SQLite APIs.

Note

The terminating semicolon is not required when a single query is executed through sqlite_query(). However, the PHP extension does allow you to submit multiple commands at the same time by passing a single string of SQL statements with each terminated by a semicolon.

For a SELECT statement, resource will be a result handle that can be processed by further functions, as we will see shortly. The result will be FALSE if a SELECT query fails.

Where query does not return any rows, for instance an UPDATE, INSERT, or DELETE statement or a CREATE TABLE as shown in the example in Listing 5.3, the resource value returned is trUE if the statement is executed successfully; otherwise, it is FALSE.

To find the reason a query has failed, use the sqlite_last_error() and sqlite_error_string() functions to retrieve the error message from SQLite. The former returns a numerical error code, which the latter decodes into a human-readable error message, as demonstrated in Listing 5.3.

Listing 5.3. Executing a CREATE TABLE Statement Through the PHP Extension
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  $sql = "CREATE TABLE contacts (
           id INTEGER PRIMARY KEY,
           first_name CHAR,
           last_name  CHAR,
           email      CHAR
         )";

  $res = sqlite_query($db, $sql);
  if (!$res) {
    echo "Error with query:";
    echo "<PRE>$sql</PRE>";
    echo sqlite_error_string(sqlite_last_error($db));
  }
  else {
    echo "Command successful";
  }
}

?>

The first time Listing 5.3 is executed, the table contacts will be created. However, a second execution will produce an error message similar to that shown in Figure 5.2.

Figure 5.2. Error messages displayed when a SQLite query fails.

[View full size image]

The screenshot was taken on a system that has display_errors turned on in php.ini, and you can see that the feedback from the PHP extension is generally more useful than the SQLite error message returned.

Using Commands That Change the Database

The apostrophe or single quote character needs to be delimited before it can be stored in a SQLite column. Two apostrophes appearing together are treated as a single character within a string enclosed by single quotes.

For instance this statement is invalid:

INSERT INTO mytable (myfield) VALUES ('It's tricky');

It should be rewritten as follows, which will insert the string It's tricky containing a single apostrophe into the database:

INSERT INTO mytable (myfield) VALUES ('It''s tricky');

The function sqlite_escape_string() can be used to automatically make strings compatible with SQLite, by escaping single quote characters and encoding binary-unsafe characters. Listing 5.4 contains a PHP script that loops through an array of contact details and safely inserts each record into the database.

Note

The PHP function addslashes() is not suitable for escaping quotes in values intended for storage in SQLite.

Listing 5.4. Escaping Special Characters Before Inserting Data
<?php

$contacts = array(array("first_name" => "Ronnie",
                        "last_name"  => "O'Sullivan",
                        "email"      => "ronnie@worldsnooker.com"),
                  array("first_name" => "Shaquille",
                        "last_name"  => "O'Neil",
                        "email"      => "shaq@lakers.com"),
                  array("first_name" => "John",
                        "last_name"  => "O'Shea",
                        "email"      => "jos@manutd.com"));

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  foreach ($contacts as $c) {

    // Escape each value before building INSERT statement
    foreach ($c as $field => $value) {
      $c[$field] = sqlite_escape_string($value);
    }

    $sql = "INSERT INTO contacts (first_name, last_name, email)
            VALUES ('$c[first_name]', '$c[last_name]', '$c[email]')";

    $res = sqlite_query($db, $sql);
    if (!$res) {
      echo "Error with query.";
      echo "<PRE>$sql</PRE>";
      echo sqlite_error_string(sqlite_last_error($db));
    }
    echo "Inserted $c[first_name] $c[last_name]<br />\n";
  }
}

?>

In lines 3 to 11 we declare an array of the contact information that will be inserted. The outer loop, opened on line 20, extracts each of the inner arrays into $c, and then each value within $c is escaped before the query string is built.

As the contacts table includes an INTEGER PRIMARY KEY, a unique number is assigned to the id column each time an INSERT is performed. The value assigned can be retrieved using the sqlite_last_insert_rowid() function.

Replace line 36 in Listing 5.4 with the following code to display the value of the id column for each record inserted:

$id = sqlite_last_insert_rowid($db);
echo "Inserted $c[first_name] $c[last_name] as ID $id<br />\n";

Whenever an UPDATE, INSERT, or DELETE statement is issued, it's likely that one or more rows in a table have been modified in some way. The function sqlite_changes() will return the number of affected rows and can be used to confirm that the command has done what it was intended to do.

Listing 5.5 issues an UPDATE statement converting each first_name and last_name value to uppercase in the contacts table and shows the number of rows affected by this command.

Listing 5.5. Finding the Number of Rows Affected by an UPDATE Command
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  $sql = "UPDATE contacts
          SET first_name = upper(first_name),
              last_name  = upper(last_name)";

  $res = sqlite_query($db, $sql);
  if (!$res) {
    echo "Error with query.";
    echo "<PRE>$sql</PRE>";
    echo sqlite_error_string(sqlite_last_error($db));
  }
  $changes = sqlite_changes($db);
  echo "Updated $changes row(s)<br />\n";
}

?>

Note

The number reported by sqlite_changes() is the number of matching rows in the WHERE clause of an UPDATE statement, or the total number of rows in the table if there is no WHERE. It is not necessarily the number of rows changed by the command; where a row is updated to the same value it previously held, this is still counted in the total number of changes.

Working with Retrieved Data

The resource returned by sqlite_query() for a SELECT statement can be used in conjunction with a number of PHP functions to fetch the data returned by the query and to find information about the dataset itself.

The functions sqlite_num_rows() and sqlite_num_fields() both take a single resource parameter and return an integer value of the number of rows and columns in the returned dataset respectively.

To fetch a column name, use sqlite_field_name(). The function takes two parameters, the resource returned by sqlite_query() and a column identifier specified as an index number. Where the columns are selected in the FROM list of the query, the leftmost column in the list has index 0. If SELECT * FROM tablename is used, the columns are numbered from zero in the order that they appear in the database schema.

The function to fetch a column value is sqlite_column(), which takes the same two parameters as sqlite_field_name(), but this time the column identifier may be either an index number or the name of the column, if it is known.

For instance, if the query passed was

SELECT first_name, last_name
FROM contacts;

the following two statements are equivalent:

$value = sqlite_column($res, 1);

$value = sqlite_column($res, "last_name");

Immediately after sqlite_query() has been executed, the values returned by sqlite_column() are from the very first row of the dataset. To shift the result handle pointer to the next row, use sqlite_next().

The most common usage is to seek to each row in turn using sqlite_next() in a loop structure. To find out when the end of the dataset is approaching, use sqlite_has_more() with a single resource parameter, which returns trUE as long as there are still more rows to fetch. If you call sqlite_next() when there are no more rows to be fetched, a warning will be shown.

The example in Listing 5.6 puts together the functions we have briefly introduced in this section to execute a query and display the contents of the returned dataset in a formatted HTML table. The table column headings are the database column names, and each record is displayed in a table row in turn until there are no records left.

Listing 5.6. Displaying the Results of a Query in an HTML Table Using PHP
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  $sql = "SELECT * FROM contacts
          ORDER BY last_name, first_name";

  $res = sqlite_query($db, $sql);
  if (!$res) {
    echo "Error with query.";
    echo "<PRE>$sql</PRE>";
    echo sqlite_error_string(sqlite_last_error($db));
  }

  echo sqlite_num_rows($res)." row(s) found<br />\n";

  echo "<TABLE BORDER=1>\n";

  echo "<TR>\n";
  for ($i=0; $i<sqlite_num_fields($res); $i++) {
    echo "<TH>" . sqlite_field_name($res, $i) . "</TH>\n";
  }
  echo "</TR>\n";

  while (sqlite_has_more($res)) {
    echo "<TR>\n";
    for ($i=0; $i<sqlite_num_fields($res); $i++) {
      echo "<TD>" . sqlite_column($res, $i) . "</TD>\n";
    }
    echo "</TR>\n";
    sqlite_next($res);
  }

  echo "</TABLE>";
}

?>

The output should look similar to that shown in Figure 5.3.

Figure 5.3. Output from Listing 5.6, showing the result of a query displayed in an HTML table.

[View full size image]

Let's look at parts of this code in more detail. After we have passed the query to SQLite using sqlite_query(), we display a count of the number of rows that will be displayed in line 20.

echo sqlite_num_rows($res)." row(s) found<br />\n";

Next, after opening an HTML <TABLE> tag, we generate a row of table headings<TH>by counting through the number of fields in the dataset and outputting the name of each one in turn.

echo "<TR>\n";
for ($i=0; $i<sqlite_num_fields($res); $i++) {
  echo "<TH>" . sqlite_field_name($res, $i) . "</TH>\n";
}
echo "</TR>\n";

We then use the same for loop as in the preceding lines within another while loop to fetch a row at a time and display each field value in turn inside <TD> tags, to align with the column headings generated previously.

while (sqlite_has_more($res)) {
  echo "<TR>\n";
  for ($i=0; $i<sqlite_num_fields($res); $i++) {
    echo "<TD>" . sqlite_column($res, $i) . "</TD>\n";
  }
  echo "</TR>\n";
  sqlite_next($res);
}

In fact the outer while loop could also be performed as a for loop by counting up to the value returned by sqlite_num_rows(). The method you use will be a matter of preference.

Another way to fetch rows of data is using sqlite_current() or sqlite_fetch_array(). Both functions return an associative array of all the values from the next row using the column names as keys, with the latter also advancing the result handle pointer to the next row, as if sqlite_next() had been called.

After executing this command

$row = sqlite_current($res);

the following two statements are equivalent:

$value = sqlite_column($res, "first_name");

$value = $row["first_name"];

The default behavior of sqlite_fetch_array() and sqlite_current() is to return an associative array with columns indexed both by their field name and their numerical position beginning at zero. Therefore if first_name was the first column listed, the following statement is also equivalent:

$value = $row[1];

An optional second parameter can be passed to either function to define this behavior. The valid constants are SQLITE_ASSOC, SQLITE_NUM, and SQLITE_BOTH, which instruct PHP to create an associative array, a numerically indexed array or an array with both sets of indexes respectively.

For instance, if the command executed is

$row = sqlite_current($res, SQLITE_NUM);

$row["first_name"] will not contain any value.

Note

The configuration directive sqlite.assoc_case in php.ini determines the case of the keys used in the associative arrays returned by sqlite_current() and sqlite_fetch_array(). The default value is zero for mixed-case keys that match the column names exactly. When this setting is 1 or 2, the key names are converted to all uppercase or all lowercase respectively.

The following section of code could replace lines 30 to 37 of Listing 5.6, performing the same loop using do ... while:

while ($row = sqlite_fetch_array($res)) {
  echo "<TR>\n";
  for ($i=0; $i<sqlite_num_fields($res); $i++) {
    echo "<TD>" . $row[$i] . "</TD>\n";
  }
  echo "</TR>\n";
}

If you are interested only in the first row returned by a query, the function sqlite_fetch_single() returns an associative array of values from the very first row in the dataset. Because the result handle pointer is not moved and its position does not affect the behavior of this function, sqlite_fetch_single() is the most optimal way to return data to PHP when only the first row is required.

The PHP extension also provides a single function for returning the entire result of a query into a single array. The following code line gives the prototype for sqlite_array_query(), and as with sqlite_query(), the order of the query and dbhandle parameters can be reversed for compatibility with other PHP database extensions.

array sqlite_array_query ( resource dbhandle, string query [, int result_type])

This function returns a single, two-dimensional array made up of one array of associative arrays of values for each record in the dataset.

The same functionality could be implemented quite easily in your PHP script using a sqlite_query() and a loop on sqlite_fetch_array(), but as a built-in function of the PHP SQLite extension, it is significantly faster to use this function that builds the array in PHP. The following two pieces of code have the same result:

$data = sqlite_query($db, $sql);

$res = sqlite_query($db, $sql);
while($row = sqlite_fetch_array($res)) {
  $data[] = $row;
}

The contents of the $data array that was generatedby either methodfor the query SELECT * FROM contacts is shown in Listing 5.7, using the print_r() function to recursively output the array elements.

Listing 5.7. Array Returned by sqlite_array_query() Function
Array
(
    [0] => Array
        (
            [0] => 2
            [id] => 2
            [1] => Shaquille
            [first_name] => Shaquille
            [2] => O'Neil
            [last_name] => O'Neil
            [3] => shaq@lakers.com
            [email] => shaq@lakers.com
        )

    [1] => Array
        (
            [0] => 3
            [id] => 3
            [1] => John
            [first_name] => John
            [2] => O'Shea
            [last_name] => O'Shea
            [3] => jos@manutd.com
            [email] => jos@manutd.com
        )

    [2] => Array
        (
            [0] => 1
            [id] => 1
            [1] => Ronnie
            [first_name] => Ronnie
            [2] => O'Sullivan
            [last_name] => O'Sullivan
            [3] => ronnie@worldsnooker.com
            [email] => ronnie@worldsnooker.com
        )

)

The PHP extension allows random access of the rows in a dataset using the sqlite_seek() function, for which the prototype is given here:

bool sqlite_seek (resource result, int rownum)

The function attempts to set the result handle pointer to row rownum for the given resource, and returns trUE on success or FALSE if the rownum specified does not exist.

Listing 5.8 shows how sqlite_seek() can be used to jump to a random row within the dataset returned from our sample contacts database. Execute this script a few times to see that a random selection is made each time.

Listing 5.8. Using sqlite_seek() to Jump to a Specific Row
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  $sql = "SELECT * FROM contacts
          ORDER BY last_name, first_name";

  $res = sqlite_query($db, $sql);
  if (!$res) {
    echo "Error with query.";
    echo "<PRE>$sql</PRE>";
    echo sqlite_error_string(sqlite_last_error($db));
  }

  sqlite_seek($res, rand(0, sqlite_num_rows($res)-1));
  $row = sqlite_fetch_array($res);
  print "<PRE>";
  print_r($row);
  print "</PRE>";
}

?>

The function sqlite_rewind() takes a single resource parameter and resets the result handle pointer to the first record in the dataset. The following two commands are equivalent:

sqlite_rewind($res);

sqlite_seek($res, 0);

For these operations to work, the result handle must be buffered and seekable, and creating this data structure in memory is an overhead that can be avoided. The function sqlite_unbuffered_query() works just the same as sqlite_query() except that the result handle can only be accessed sequentially. As only one row is returned at a time, this gives much better performance and is recommended unless random access is a necessity.


Working with User-Defined Functions

A very powerful feature of the PHP SQLite extension is the ability to define functions in PHP that can be registered and executed within an SQL statement. It is therefore possible for PHP developers to extend the functionality of SQLite using a familiar language.

The example in Listing 5.9 defines a trivial function in PHP that reverses each word in a string. This differs from the usual behavior of strrev(), which reverses the entire string. The example uses sqlite_create_function() to register a user-defined function, word_reverse(), for use within SQLite.

Listing 5.9. Adding a User-Defined Function to SQLite in PHP
<?php

function reverse_words($string) {
  $w = explode(" ", $string);
  for($i=0; $i<count($w); $i++) {
    if ($i > 0)
      $ret .= " ";
    $ret .= strrev($w[$i]);
  }
  return($ret);
}

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {
  sqlite_create_function($db, "word_reverse", "reverse_words", 1);

  $sql = "SELECT word_reverse('The quick brown fox');";
  $res = sqlite_query($db, $sql);
  echo sqlite_column($res, 0);
}

?>P

First, we declare the function reverse_words() in PHP. Using explode() to break up the passed-in stringassuming words are separated by a space characterwe then call strrev() on each word in turn and piece the string back together:

function reverse_words($string) {
  $w = explode(" ", $string);
  for($i=0; $i<count($w); $i++) {
    if ($i > 0)
      $ret .= " ";
    $ret .= strrev($w[$i]);
  }
  return($ret);
}

After opening a SQLite connection to webdb as database resource $db, we use sqlite_create_function() to register the user-defined function.

sqlite_create_function($db, "word_reverse", "reverse_words", 1);

There are four arguments to sqlite_create_function(). First the database resource, in this case $db, is supplied. The second parameter is the function name that will be used by SQLite, and the third is the name of the PHP function from which it is derived.

In our example, we used word_reverse() for the SQLite function and reverse_words() for the PHP function to tell the two apart. There is actually no reason that they cannot share the same name if desired.

The fourth parameter is optional and indicates the number of parameters that are passed to the UDF. In this case, only a single string parameter is required, so the value is 1. This acts as a hint to the SQLite parser, and an error will be returned if the number of parameters passed when the function is called does not match this value. If the PHP function can accept a variable number of parameters, this value can be omitted.

The output from Listing 5.9 is as you might expect:

ehT kciuq nworb xof

A function registered with sqlite_create_function() does not exist outside of the process in which it is executed. Because the function is created within a given database resource, a user-defined function can be called within PHP functions that have access to the same resource, so the scope of a UDF within a script is not an issue. However, the function is not saved to the database permanently.

Where a persistent database connection has been used, sqlite_create_function() must still be called again before the function is available to SQLite. If PHP reattaches to a persistent SQLite connection in which a user-defined function has been declared and you attempt to call that function, the following warning will be displayed rather than a no such function error.

sqlite_query(): this function has not been correctly defined for this request

It is possible to use an existing function name as the name of a UDF in the second parameter to sqlite_create_function(), which will overload the behavior of that function without warning. All subsequent calls to that function will execute the PHP function specified rather than the built-in function.

The php() Function

When PHP first opens a SQLite database, it automatically registers the function php(), which provides a quick way to access any PHP function without having to add it as a UDF with sqlite_create_function(). Both built-in PHP functions and those defined in your script code can be referenced this way.

Listing 5.10 shows how the PHP function ucwords() can be called from within SQLite using php() to capitalize the first character of each word in a string.

Listing 5.10. Calling a PHP Function from SQLite Using php()
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {
  $sql = "SELECT php('ucwords', 'The quick brown fox');";
  $res = sqlite_query($db, $sql);
  echo sqlite_column($res, 0);
}

?>

The output from Listing 5.10 is as expected:

The Quick Brown Fox

Creating Aggregating Functions

The process to create an aggregating user-defined functionone that applies a calculation across all the rows returned by a query, or across similar rows indicated by a GROUP BY clauseis slightly different than a regular function.

The function sqlite_create_aggregate() requires two functions to manage the aggregate; one that is called for each row on the result set returnedthe step functionand another that is called once after all the rows have been processedthe finalize function.

A context variable is passed by reference so that both functions can access the data being aggregated.

The example in Listing 5.11 creates and demonstrates a new aggregating function to calculate the median average of a set of numbers. The median is a value such that half the values in the set are above the median and half are below it. By contrast, the built in avg() function computes the mean averagethe sum of all the values divided by the number of items in the dataset.

Listing 5.11. Adding an Aggregating UDF with sqlite_create_aggregate()
<?php

function median_step(&$context, $num) {
  $context[] = $num;
}

function median_finalize(&$context) {
  sort($context);
  $count = count($context);
  if ($count%2 == 1) {  // Odd number of values
    $median = $context[floor($count/2)];
  }
  else { // Even number of values
    $median = ($context[$count/2] + $context[($count/2)-1] ) / 2;
  }
  return($median);
}

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {

  sqlite_create_aggregate($db, "median", "median_step", "median_finalize");

  $sql = "SELECT median(num), avg(num) FROM numbers";
  $res = sqlite_query($db, $sql);

  echo "Median is ";
  echo sqlite_column($res, 0);
  echo "<br />\n";

  echo "Mean is ";
  echo sqlite_column($res, 1);
  echo "<br />\n";
}

?>

Before we can use our new function we need to create the table numbers and insert some values. The script in Listing 5.12 creates the table and inserts the squares of numbers one to nine in turn. Because of the geometric nature of this sequence, we can use it to show how the median and mean averages differ using our new function.

Listing 5.12. Inserting a Sequence of Square Numbers
<?php

if (!$db = sqlite_open("webdb", 0666, &$errmessage)) {
  echo "Could not open database:<br />\n";
  echo $errmessage;
  exit;
}
else {
  $sql = "CREATE TABLE numbers (num INTEGER)";
  $res = sqlite_query($db, $sql);

  for ($i=1; $i<=9; $i++) {
    $val = $i * $i;
    $sql = "INSERT INTO numbers (num) VALUES ($val)";
    $res = sqlite_query($db, $sql);
  }
}

?>

The following line of code shows the complete sequence of numbers inserted. From looking at this sequence in numerical order, we can see that the median will be 25, as there are four numbers on either side:

1 4 9 16 25 36 49 64 81

Let's look back through Listing 5.11 in more detail. The step function simply builds an array in PHP, adding each value in turn:

function median_step(&$context, $num) {
  $context[] = $num;
}

Notice however that $context is passed by reference in the function parameters. The value does not need to be returned.

The finalize function has to do a little more work. After the array has been created, we sort it into numerical order and find the number of items. Again, $context is passed by reference so that the array built by median_step() is available within this scope:

function median_finalize(&$context) {
  sort($context);
  $count = count($context);

If there are an odd number of values, the median is the middle value itself. If there are an even number of values, the median is the mid-point between the two middle values.

if ($count%2 == 1) {  // Odd number of values
  $median = $context[floor($count/2)];
}
else {  // Even number of values
  $median = ($context[$count/2] + $context[($count/2)-1] ) / 2;
}

Lastly, we return the computed median value.

  return($median);
}

To register the aggregating function median(), we call sqlite_create_aggregate() as follows to reference the two functions that we just defined:

sqlite_create_aggregate($db, "median", "median_step", "median_finalize");

To compare the median of the values in numbers with its average, this SQL statement is executed:

$sql = "SELECT median(num), avg(num) FROM numbers";
$res = sqlite_query($db, $sql);

The values displayed are

Median is 25
Mean is 31.6666666666667

Working with Binary Data in UDFs

If the data you are passing to or from a UDF may not be binary safe, you must encode it to avoid possible problems. A NUL byte is a string terminator, so if one could appear anywhere other than the end of a string you may get unexpected results, for instance.

The function sqlite_udf_encode_binary() takes a string parameter and returns a binary-safe encoded version. The reverse is sqlite_udf_decode_binary(). For performance reasons, PHP will not perform such encoding automatically itself.


Using the PEAR Database Class

In addition to the API we have already discussed, PHP also allows access to SQLite databases via the PEAR Database class. We will show briefly how to use this with SQLite.

The PEAR class provides a unified API for accessing SQL-based databases. It supports SQLite as well as many other popular databases such as Microsoft SQL Server, MySQL, Oracle, and PostgreSQL, as well as ODBC connectivity.

The database-abstraction approach allows you to create scripts that can work on a wide range of database engines simply by changing the connection parameters. The same set of functions is made to work with whatever database type you happen to be connected to at the timeright down to a function that delimits quotes and other special characters the correct way for that database. If you expect your PHP application to need porting to a different database system in future, the PEAR class is well worth considering.

Note

Database-specific features must be worked around when using a database abstraction layer. For instance the INTEGER PRIMARY KEY feature specific to SQLite is similar to an AUTO INCREMENT field in MySQL, but the same functionality can be only achieved in Oracle using CREATE SEQUENCE. In fact the PEAR Database class forces you to use a database-level sequence object for all autoincrementing fields for maximum compatibility.

To download and install the PEAR DB class automatically if you do not already have it, use the pear utility:

# pear install DB

There are two ways to define a database connection. The first uses a string to indicate the Data Source Name (DSN) and usually looks like this:

dbtype://username:password@hostname/dbname

For SQLite, however, the username, password, and hostname fields are not required, so the following DSNs are sufficient to identify a database file in the current directory or in the given path respectively.

sqlite:///dbfile
sqlite:////home/chris/sqlite/dbfile

The second way to define a DSN is by setting elements of an array of parameters, which is defined by the class as follows:

$dsn = array(
    'phptype'  => false,
    'dbsyntax' => false,
    'username' => false,
    'password' => false,
    'protocol' => false,
    'hostspec' => false,
    'port'     => false,
    'socket'   => false,
    'database' => false,
);

Only the phptype and database elements are required to identify a SQLite database, so the following array is a valid DSN:

$dsn = array(
    'phptype'  => 'sqlite',
    'database' => 'webdb'
);

Connecting to the database with a DSN defined by either method is as simple as the following statement:

$db = DB::connect($dsn);

To execute an SQL command that does not return any rows, use the query() method:

$sql = "DELETE FROM mytable WHERE myfield = 'somevalue'";
$res = $db->query($sql);

To test if a query was successful, use the isError() function on the returned result:

if (DB::isError($res)) {
  echo "An error occurred";
}

For a SELECT query that returns a single row, use the getrow() method to return that row as an arraysimilar to calling sqlite_fetch_single(). If the optional second parameter is passed the constant DB_FETCHMODE_ASSOC, an associative array using the column names as indexes will be created:

$sql = "SELECT * FROM mytable";
$row = $db->getRow($sql, DB_FETCHMODE_ASSOC);

To return all the rows in a query as an arraysimilar to sqlite_array_query()use the getAll() method.

$sql = "SELECT * FROM mytable";
$row = $db->getAll($sql, DB_FETCHMODE_ASSOC);

This chapter has only touched on a few of the many functions available in the PEAR Database class. There is comprehensive online documentation for the class at http://pear.php.net/manual/en/package.database.php.


Chapter 6. The C/C++ Interface

In this chapter we will look at how SQLite's C/C++ interface can be incorporated into your programs. We will discuss the different ways of querying a database and look at how you can add your own custom functions to SQLite.


Preparing to Use the C/C++ Interface

In order to develop software with an embedded SQLite database, you need to make sure your system has the SQLite development library installed. The latest version can always be downloaded from http://www.sqlite.org/download.html.

If you have already installed and used sqlite as a precompiled binary, you will not necessarily have the library yet. For Linux, the precompiled library is sqlite.so.gz and for Windows it is sqlitedll.zip. If you also want to use the TCL bindings, precompiled libraries are available that support this, named tclsqlite.so.gz and tclsqlite.zip respectively.

Users of Red Hat Linux and other Unix-like systems that use the RPM package format can use the sqlite-devel package to install the library along with header files and documentation.

If you compiled SQLite from source, the libraries were installed when you issued the make install command. A default source installation on Linux/Unix will put libsqlite in the /usr/local/lib directory and sqlite.h in /usr/local/include.


Using the C Language Interface

Now that you have made sure the library is available on your system, let's take a look at the functions that make up the C language interface.

Opening and Closing a Database

The function to open a SQLite database is sqlite_open(). The prototype is

sqlite *sqlite_open(const char *dbname, int mode, char **errmsg);

The dbname parameter is the name of the database file on the filesystem. If just a filename is given, it is assumed that the database is in the current directory at runtime. A relative or absolute path can be specified.

The mode parameter is intended for future use, to specify the file mode in which the database file is opened. Typical values would be 0777 for read/write access or 0444 for read-only. However, at the time of writing, this parameter is ignored by the library.

The sqlite structure to which a pointer is returned is an opaque structure that must be passed as the first parameter to all the other SQLite API functions. If the database cannot be opened, the return value will be NULL and errmsg will point to the location of the error message created by sqlite_open(). The memory allocated for errmsg should be freed using sqlite_freemem().

To close the database, use sqlite_close(), which is simply passed the open sqlite structure pointer.

The example in Listing 6.1 shows how to establish a connection to a database called progdb in the current directory, and displays an error message if the database cannot be opened.

Listing 6.1. Connecting to a SQLite Database Using C/C++
#include <stdio.h>
#include <sqlite.h>

main()
{

  char *errmsg;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  printf("Successfully connected to database\n");
  sqlite_close(db);
}

We compile listing6.1.c into an executable listing6.1 as follows:

$ gcc -o listing6.1 listing6.1.c lsqlite

Executing the new program should tell us that we have been able to connect to the progdb database. The first time you run the program, the database will be created and you will be able to see a file called progdb in the current directory:

$ ./listing6.1
Successfully connected to database

If there was a problem opening the database, for example, if you do not have write permission on the current directory, an error will be displayed:

$ cd /
$ /path/to/listing6.1
Could not open database: unable to open database: progdb

Getting Information About the SQLite Library

Two constants are defined in sqlite.h that allow you to find information about the version and encoding of the library:

const char sqlite_version[]

const char sqlite_encoding[]

The value of sqlite_version is a string representation of the library version number, for instance 2.8.13.

The encoding method for the library is determined at compile time and the default value of sqlite_encoding is iso8859 for ISO-8859-1 encoding. The alternative encoding method currently provided by SQLite is UTF-8. The encoding method used by a program cannot be changed without recompiling the library.

Executing SQL Statements

The function sqlite_exec() is used to send an SQL statement to the SQLite engine. Its prototype is

int sqlite_exec(
  sqlite *db,
  char *sql,
  int (*xCallback)(void*,int,char**,char**),
  void pArg,
  char **errmsg
);

The five parameters required are as follows:

  • db is a valid sqlite structure pointer.

  • sql is a null-terminated character string containing one or more SQL statements.

  • The third parameter is a pointer to a callback function or may be NULL if no callback is required.

  • pArg is a pointer to the first argument of the callback function.

  • If the query fails, the error message from SQLite will be pointed to by errmsg.

The callback function is invoked once for each row returned by the query, and we will examine this shortly. An SQL statement that is not a SELECT does not return any rows, so no callback function is required and the third and fourth parameters can be NULL.

Note

A semicolon is not required to terminate a single SQL query in the sql parameter. However, if multiple statements are passed in one sqlite_exec() call, they must be separated with semicolons.

The example in Listing 6.2 shows how a CREATE TABLE statement is executed using the C interface.

Listing 6.2. Executing a CREATE TABLE Statement with the C Library
#include <stdio.h>
#include <sqlite.h>

main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  ret = sqlite_exec(db,
          "CREATE TABLE contacts (   \n"
          "  id INTEGER PRIMARY KEY, \n"
          "  first_name CHAR,        \n"
          "  last_name  CHAR,        \n"
          "  email      CHAR)",  NULL, NULL, &errmsg);
  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }

  sqlite_close(db);
}

The SQL statement in Listing 6.2 is formatted using \n characters and indented for readability. This is not a requirement, but as the schema stored in the database is the exact CREATE TABLE statement executed, the formatting here will make the output of the schema more readablefor instance when using the .schema command in sqlite.

Running the compiled program should be silent, assuming there are no problems accessing progdb, and the table will be created. If you execute the program a second time the table name will already be in use, and the following error message from SQLite will be displayed.

$ ./listing6.2
SQL error: table contacts already exists

The constant SQLITE_OK is a zero value and is used to check for successful execution of an SQL statement. Any non-zero value indicates that an error has occurred. The full list of return code constants is shown in Table 6.1.

Table 6.1. Return Code Constants

Constant

Value

Description

SQLITE_OK

0

Successful result.

SQLITE_ERROR

1

SQL error or missing database.

SQLITE_INTERNAL

2

An internal logic error in SQLite.

SQLITE_PERM

3

Access permission denied.

SQLITE_ABORT

4

Callback routine requested an abort.

SQLITE_BUSY

5

The database file is locked.

SQLITE_LOCKED

6

A table in the database is locked.

SQLITE_NOMEM

7

A malloc() failed.

SQLITE_READONLY

8

Attempt to write to a read-only database.

SQLITE_INTERRUPT

9

Operation terminated by sqlite_interrupt().

SQLITE_IOERR

10

A disk I/O error occurred.

SQLITE_CORRUPT

11

The database disk image is malformed.

SQLITE_NOTFOUND

12

Table or record not found.

SQLITE_FULL

13

Insertion failed because database is full.

SQLITE_CANTOPEN

14

Unable to open the database file.

SQLITE_PROTOCOL

15

Database lock protocol error.

SQLITE_EMPTY

16

Database table is empty.

SQLITE_SCHEMA

17

The database schema changed.

SQLITE_TOOBIG

18

Too much data for one row of a table.

SQLITE_CONSTRAINT

19

Abort due to constraint violation.

SQLITE_MISMATCH

20

Data type mismatch.

SQLITE_MISUSE

21

Library used incorrectly.

SQLITE_NOLFS

22

Uses OS features not supported on host.

SQLITE_AUTH

23

Authorization denied.

SQLITE_ROW

100

sqlite_step() has another row ready.

SQLITE_DONE

101

sqlite_step() has finished executing.

More information on each of these return codes can be found in Appendix E, "C/C++ Interface Reference."

Note

If you want to validate a string to see if it forms a valid SQL statement, use the sqlite_complete() function. Taking a single character pointer argument, the function returns true when the string supplied forms a complete SQL statement or false if more text is required.

The sqlite_complete() function considers a statement to be complete if it ends with a semicolon, ignoring semicolons that separate statements within a CREATE TRIGGER statement. The function does not attempt to validate the syntax of the SQL statement itself.

Using Commands That Change the Database

Now that we have created a table, let's insert some rows. Listing 6.3 creates a program that reads three values from stdin that are used as the first_name, last_name, and email fields in a subsequent INSERT statement.

Listing 6.3. Inserting Records into the Database
#include <stdio.h>
#include <sqlite.h>

#define MAXLEN 32
#define MAXQRY 200

main()
{

  char *errmsg;
  int ret;
  char first_name[MAXLEN], last_name[MAXLEN], email[MAXLEN];
  char qry[MAXQRY];

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  printf("Enter first name: ");
  scanf("%s", first_name);

  printf("Enter last name: ");
  scanf("%s", last_name);

  printf("Enter email: ");
  scanf("%s", email);

  sprintf(qry, "INSERT INTO CONTACTS (first_name, last_name, email) \n"
               "VALUES ('%s', '%s', '%s')", first_name, last_name, email);

  ret = sqlite_exec(db, qry, NULL, NULL, &errmsg);
  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n%s\n", errmsg, qry);
  }
  else
  {
    printf("%s %s (%s) was inserted as ID %d\n",
            first_name, last_name, email, sqlite_last_insert_rowid(db));
  }

  sqlite_close(db);
}

As the contacts table includes an INTEGER PRIMARY KEY field, a unique number is assigned to the id column each time an INSERT is performed. The value assigned can be retrieved using the sqlite_last_insert_rowid() function. This statement is used to confirm that the insert took place successfully and shows the assigned id value.

printf("%s %s (%s) was inserted as ID %d\n",
        first_name, last_name, email, sqlite_last_insert_rowid(db));

Compile and run Listing 6.3 and enter some details:

$ ./listing6.3
Enter first name: Chris
Enter last name: Newman
Enter email: chris@lightwood.net
Chris Newman (chris@lightwood.net) was inserted as ID 1

Be aware that no validation is being done in this program to make sure a valid SQL statement is being submitted to SQLite. The apostrophe or single quote character is the most common cause of problems, as you can see if you attempt to enter a contact name containing an apostrophe.

[chris@python c]$ ./listing6.3
Enter first name: Paddy
Enter last name: O'Brien
Enter email: paddy@irish.com
SQL error: unrecognized token: "@"
INSERT INTO CONTACTS (first_name, last_name, email)
VALUES ('Paddy', 'O'Brien', 'paddy@irish.com')

The SQL query displayed in the error message shows the problem caused by the apostrophe. In this case the @ symbol causes an error, which appears strange at first but look at the pairs of single quotes in the VALUES list. Before SQLite even attempts to perform the insert, it reads three strings from the line'Paddy', 'O' and ', '. The @ sign is not contained in quotes and causes a parser error.

Fortunately the library provides an easy way around this using sqlite_mprintf().

This function works in a similar way to sprintf() except that a pointer to a string buffer is returnedthe memory assigned to the buffer by malloc() should be freed using sqlite_freemem() when it is no longer needed. Using sqlite_mprintf(), the format code %q works like %s, but any single quotes are delimited, so they can be inserted as literal characters.

We could rewrite the statement that builds the SQL query in Listing 6.3 like this:

char *qry;
...
qry = sqlite_mprintf("INSERT INTO CONTACTS (first_name, last_name, email) \n"
               "VALUES ('%q', '%q', '%q')", first_name, last_name, email);

For convenience, the function sqlite_exec_printf() works like sqlite_exec() but accepts a printf-style format string with its arguments given after the usual five parameters. Again the %q format code can be used to make sure quotes are delimited correctly.

We can use this function to combine the sprintf() and sqlite_exec() calls in Listing 6.3 while also protecting against problems with single quotes, as the following example demonstrates.

ret = sqlite_exec_printf(db,
             "INSERT INTO CONTACTS (first_name, last_name, email) \n"
             "VALUES ('%q', '%q', '%q')", NULL, NULL, &errmsg,
                                          first_name, last_name, email);

Replace lines 33 to 36 with the preceding statement, and then recompile and execute the program to attempt to insert the contact name that caused us problems before:

Enter first name: Paddy
Enter last name: O'Brien
Enter email: paddy@irish.com
Paddy O'Brien (paddy@irish.com) was inserted as ID 2

The INSERT statement in the preceding example used an implicit transaction to execute a single change to the database. Where an explicit transaction is required around a group of statements, it can be started simply by executing a BEGIN TRANSACTION command through the SQLite C Library.

sqlite_exec(db, "BEGIN TRANSACTION");

Issuing a COMMIT TRANSACTION or ROLLBACK TRANSACTION instruction is done the same way. Be aware that if sqlite_close() is called during an open transaction, an implicit ROLLBACK will take place.

Whenever an UPDATE or DELETE operation is performed, records in the database will be changedassuming of course that the WHERE clause does find some rows that should be affected. The function sqlite_changes() allows us to see just how many matching rows were changed or removed as a result of an SQL command.

Listing 6.4 issues an UPDATE statement converting each first_name and last_name value to uppercase in the contacts table and displays the number of rows affected by this command. Because there is no WHERE clause in the SQL statement, the number of rows affected will be the total number of rows in the contacts table.

Listing 6.4. Performing an UPDATE on the Database
#include <stdio.h>
#include <sqlite.h>

main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  ret = sqlite_exec(db,
          "UPDATE contacts "
          "SET first_name = upper(first_name), \n"
          "    last_name  = upper(last_name)", NULL, NULL, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }
  else
  {
    printf("%d row(s) were changed\n", sqlite_changes(db));
  }

  sqlite_close(db);
}

When we run this program, the response tells us how many rows were affected:

$ ./listing6.4
2 row(s) were changed

Note

The number returned by sqlite_changes() is the total number of rows that could be altered by the UPDATE regardless of whether or not the new value is different from the old value. Therefore if this program is run a second time SQLite still reports the same number of changes, even though the data effectively remains the same.

Callback Functions for SELECT Queries

In order to process the results of a query from sqlite_exec(), a callback function is used that is invoked once for each row in the dataset. The callback function must have this prototype:

int callback(void *pArg, int argc, char **argv, char **columnNames)

The pArg parameter to the callback function is a copy of pArg from the sqlite_exec() call. This can be used to pass arbitrary data to the callback, but may be NULL if not required.

The value of argc is the number of columns returned by the query. The array argv contains pointers to strings for each resultthe size of the array is argc. The fourth parameter is also an array in which the first argc entries are the column names from the query.

If you turn on the SHOW_DATATYPES pragma, the columnNames array will also contain the column types from the database schema, in the next argc elements onwards. The PRAGMA command is discussed in detail in Chapter 10, "General Database Administration," but for now this particular setting is turned on by issuing the following statement.

PRAGMA SHOW_DATATYPES=ON;

Listing 6.5 contains how a callback function is used to output the entire contents of a dataset.

Listing 6.5. Displaying the Results of a SELECT Query Using a Callback Function
#include <stdio.h>
#include <sqlite.h>

int callback(void *pArg, int argc, char **argv, char **columnNames)
{
  int i;
  for (i=0; i<argc; i++) {
    printf("%-10s %-8s %s\n",
                columnNames[i], columnNames[i+argc], argv[i]);
  }
  return(0);
}

main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  ret = sqlite_exec(db, "PRAGMA SHOW_DATATYPES=ON", NULL, NULL, NULL);

  ret = sqlite_exec(db, "SELECT * FROM contacts", callback, NULL, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }

  sqlite_close(db);
}

Let's take a closer look at the callback function. We specify callback as the third parameter to sqlite_exec() on line 32, which causes the function callback() defined in lines 4 to 12 to be invoked for each row returned by the query.

for (i=0; i<argc; i++) {
  printf("%-10s %-8s %s\n",
              columnNames[i], columnNames[i+argc], argv[i]);
}

The value in argc is the number of columns returned by the query. In the query from Listing 6.5 this is actually 4the total number of columns in contacts. This loop displays one line for each column, using a format string to printf() to specify a fixed-width column output. We show each column's name (columnNames[i]), type (columnNames[i+argc]), and value (argv[i]).

The callback function ends with

return(0);

The return value from the callback function is importanta non-zero value returned from callback() will cause the calling sqlite_exec() function to return an error.

Compiling and executing this program outputs the contents of the database, as shown here:

$ ./listing6.5
id         INTEGER  1
first_name CHAR     CHRIS
last_name  CHAR     NEWMAN
email      CHAR     chris@lightwood.net
id         INTEGER  2
first_name CHAR     PADDY
last_name  CHAR     O'BRIEN
email      CHAR     paddy@irish.com

Executing SQL Without a Callback Function

Historically, if you wanted to retrieve the results of an SQL query using the C interface, a callback function was the only method. However, most programmers found this way of coding inconvenient, so a new method became available.

The new method allows you to work with SQLite at a lower level by manipulating a SQLite virtual machine. It uses three functions to replace sqlite_exec(), as follows.

  • Compile a single SQL statement with sqlite_compile()

  • Execute sqlite_step() once for each row of output

  • Call sqlite_finalize() to clean up when execution has finished

Compilation is the process that takes an SQL statement and generates a SQLite virtual machine capable of running that statement. The workings of the Virtual Database Engine (VDBE) are discussed in Chapter 10.

The prototype for sqlite_compile() is

int sqlite_compile(
  sqlite *db,
  const char *zSql,
  const char **pzTail,
  sqlite_vm **ppVm,
  char **pzErrmsg
);

As before, the db is an open database resource pointer and the second parameteridentified here as zSqlis the SQL statement itself. The return value is SQLITE_OK if compilation succeeded; otherwise, another error code is returned and pzErrmsg will point to a textual error message.

The behavior of sqlite_compile() differs from sqlite_exec() in that only one SQL statement can be compiled at a time. If two or more statements are passed in zSql, the third parameter pzTail will point to the first character following the end of the compiled statement so that any remaining SQL commands can be handled.

After successful compilation, the pointer passed as the fourth parameter will point to the virtual machine, stored in an opaque sqlite_vm type structure. With a valid virtual machine pointer, sqlite_step() can be called once or more times to return a single row of results. This is the prototype for sqlite_step():

int sqlite_step (
  sqlite_vm *pVm,
  int *pN,
  const char ***pazValue,
  const char ***pazColName
);

The pVm parameter should be a valid pointer to a virtual machine created with sqlite_compile(). The number of columns in the result set is pointed to by pN. The pazValue and pazColName arguments each point to an array of pointers, which are used to reference the column values and their names and data types respectively.

Note

The data type information is always included in pazColName along with the column names when using sqlite_step() regardless of whether the SHOW_DATATYPES pragma has been turned on.

The return value from each sqlite_step() call is an integer code from a subset of those shown in Table 6.1. Only the following codes will be used:

  • SQLITE_BUSY

  • SQLITE_ROW

  • SQLITE_DONE

  • SQLITE_ERROR

  • SQLITE_MISUSE

A SQLITE_BUSY code indicates that the database file is locked by another thread or process and the calling function should handle this. Sleeping for a short amount of time to wait for the lock to clear is an acceptable action to take, after which sqlite_step() can be called again. The function sqlite_busy_timeout can be used to specify a number of milliseconds for which SQLite should wait for a lock to clear before returning SQLITE_BUSY.

Note

A custom handler for locked databases can be registered with the sqlite_busy_handler() instruction, to specify a function that will be executed whenever SQLite attempts to access a busy database file. A non-zero return code will cause the interface to attempt to open the file again and the cycle continues until the handler function returns zero.

If a row from the data set has been made available in pazValue and pazColumn, the return code will be SQLITE_ROW. If the last row has already been retrieved, a subsequent call will return SQLITE_DONE, so sqlite_step() will be called in total once more than the actual number of rows in the data set.

Should an error occur, the SQLITE_ERROR code is returned. After either SQLITE_DONE or SQLITE_ERROR it is a misuse of the library to attempt to invoke sqlite_step() a further time, and doing so will generate the SQLITE_MISUSE return code.

Note

The SQLITE_MISUSE error code is intended as a debugging aid, but because of the nature of the problems it attempts to detect, you should not rely on it. In some instances such misuse of the library may result in a program crash.

When all processing is complete, sqlite_finalize() should be called to clean up any memory allocated to the virtual machine. This function can also be used to find details of an error that caused sqlite_step() to return a SQLITE_ERROR. Here's the prototype:

int sqlite_finalize(
  sqlite_vm *pVm,
  char **pzErrmsg
);

The return code of sqlite_finalize() indicates the overall success or failure of the SQL statement and is the same code that would have been returned by sqlite_exec() if the same statement had been executed using that method.

Though it is a requirement to call sqlite_finalize() when either SQLITE_DONE or SQLITE_ERROR is returned, the virtual machine can be deleted at an earlier stage. A premature sqlite_finalize() instruction will interrupt processing immediately and return the database to its previous state.

Listing 6.6 shows how the same query we executed using the callback method in Listing 6.5 can be done with the non-callback interface.

Listing 6.6. Using the Non-Callback Interface to Execute a Query
#include <stdio.h>
#include <sqlite.h>

main()
{

  char **values, **columnNames;
  char *errmsg;
  int ret, cols, i;
  sqlite_vm *vm;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  ret = sqlite_compile(db, "SELECT * FROM contacts", NULL, &vm, &errmsg);

  while(sqlite_step(vm, &cols, &values, &columnNames) == SQLITE_ROW) {
    for (i=0; i<cols; i++) {
      printf("%-10s %s\n", columnNames[i], values[i]);
    }
  }

  ret = sqlite_finalize(vm, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }

  sqlite_close(db);
}

Fetching a Whole Dataset with sqlite_get_table()

The sqlite_get_table() function is provided for convenience and acts as a wrapper around the core API functions that we have already seen. This function executes an SQL query and returns a pointer to a data structure that contains the entire result at once.

The prototype for sqlite_get_table() is as follows:

int sqlite_get_table(
  sqlite *db,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg
);

In addition to the usual db and sql parameters, the function also requires passing four pointers of the types shown. The result created is a one-dimensional array of character pointers, and so in order to work out which element corresponds to which data item, the nrow and ncolumn values reference the total number of rows and columns returned respectively.

The result array contains one element for each column of every row in the dataset, plus one additional element for each of the column headings. The column headings are stored first, in result[0] to result[ncol-1]. The value of the first column in the first row returned will be result[ncol], and so on.

The example in Listing 6.7 performs a simple loop to output the contents of the contacts table using sqlite_get_table().

Listing 6.7. Using sqlite_get_table to Fetch an Entire Dataset at One Time
#include <stdio.h>
#include <sqlite.h>

main()
{

  char *errmsg;
  char **result;
  char buf[80];
  int ret, rows, cols, i, j;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  ret = sqlite_get_table(db, "SELECT * FROM contacts",
                         &result, &rows, &cols, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }
  else {
    printf("The query returned %d row(s) and %d column(s)\n", rows, cols);

    for (i=0; i< cols * (rows+1); i++) {
      printf("result[%d] = %s\n", i, result[i]);
    }

  }

  sqlite_close(db);
}

The output from this program helps to show how the elements in result are organized.

$ ./listing6.7
The query returned 2 row(s) and 4 column(s)
result[0] = id
result[1] = first_name
result[2] = last_name
result[3] = email
result[4] = 1
result[5] = CHRIS
result[6] = NEWMAN
result[7] = chris@lightwood.net
result[8] = 2
result[9] = PADDY
result[10] = O'BRIEN
result[11] = paddy@irish.com

When you are done with the result of a call to sqlite_get_table(), the memory allocated should be freed using sqlite_free_table(result).

Another function combines the convenience of sqlite_get_table() with sqlite_mprintf() to enable automatic delimiting of single quote characters using the %q format code. sqlite_get_table_printf() takes the same first six arguments as sqlite_get_table(), with the ability to use a printf-style format string in the sql parameter, which uses values from the seventh argument onwards.


Adding New SQL Functions

The SQLite library allows the SQL language to be extended with new functions implemented as C code. In fact all of SQLite's built-in functions are now written this way.

Creating a regular function begins with a call to sqlite_create_function(), and this is the prototype:

int sqlite_create_function() {
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xFunc)(sqlite_func *, int, const char **),
  void *pUserData
);

The example in Listing 6.8 defines a trivial function in C that capitalizes every alternate letter in a string. The example uses sqlite_create_function() to register the function defined in capitalize_alternate() for use within SQLite under the name altcaps().

Listing 6.8. Creating a Custom Function Using sqlite_create_function()
#include <stdio.h>
#include <sqlite.h>

void capitalize_alternate(sqlite_func *context, int argc, const char **argv)
{
  int i;
  static char str[80];

  for (i=0; i<strlen(argv[0]); i++) {
    if (i%2 == 0)
      str[i] = toupper(argv[0][i]);
    else
      str[i] = tolower(argv[0][i]);
  }
  str[i] = '\0';

  sqlite_set_result_string(context, str, -1);
}

main()
{

  char *errmsg;
  char **result;
  char str[80];
  int ret, rows, cols, i, j;

  sqlite *db = sqlite_open("progdb", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    exit(1);
  }

  sqlite_create_function(db, "altcaps", 1, capitalize_alternate, NULL);

  ret = sqlite_get_table(db, "SELECT altcaps('this is a test')",
                                              &result, &rows, &cols, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }
  else {
    for (i=0; i< cols * (rows+1); i++) {
      printf("result[%d] = %s\n", i, result[i]);
    }
  }

  sqlite_close(db);
}

First of all we declare the function capitalize_alternate() in C, which will do the work we want to include in SQL.

The prototype for a function that can be used with sqlite_create_function() is fixed; the context argument can be used to pass arbitrary data to the function. The argc and argv arguments tell the C function how many parameters have been passed when it is called from SQL.

void capitalize_alternate(sqlite_func *context, int argc, const char **argv)

The loop cycles through each item in the string passed inargv[0] will contain the only parameter in this caseand uses toupper() and tolower() in turn to alternate the case of each character, adding to str one character at a time.

for (i=0; i<strlen(argv[0]); i++) {
  if (i%2 == 0)
    str[i] = toupper(argv[0][i]);
  else
    str[i] = tolower(argv[0][i]);
}
str[i] = '\0';

Finally, we tell the interface that str is our return value using sqlite_set_result_string():

sqlite_set_result_string(context, str, -1);

Three different functions enable return values of different data types to be used:

sqlite_set_result_string(sqlite_func *, const char *, int)
sqlite_set_result_int(sqlite_func *, int);
sqlite_set_result_double(sqlite_func *,double);

Alternatively, the sqlite_set_result_error() function can be used to return an error code to SQLite from the custom function:

sqlite_set_result_error(sqlite func *, const char *, int);

The third int parameter to sqlite_set_result_string() and sqlite_set_result_error() is the number of characters to take from the string. If the value is negative, as in our example, all characters up to and including the first \0 are returned.

So with the custom function defined in C, a call is made to sqlite_create_function() to add it to the SQL language.

sqlite_create_function(db, "altcaps", 1, capitalize_alternate, NULL);

The zName parameter here tells SQLite that the function is called altcaps(). The name used in SQL does not have to match the C function name, although it is possible to do so.

To test that the function has been registered correctly, sqlite_get_table() is called to execute a simple query using altcaps().

ret = sqlite_get_table(db, "SELECT altcaps('this is a test')",
                                            &result, &rows, &cols, &errmsg);

If the query is successful, the contents of the returned array are displayed:

for (i=0; i< cols * (rows+1); i++) {
  printf("result[%d] = %s\n", i, result[i]);

Running the compiled program will produce the following output, showing that the function has indeed capitalized every other letter of the string.

$ ./listing6.8
result[0] = altcaps('this is a test')
result[1] = ThIs iS A TeSt

Note

Although sqlite_create_function() takes a database resource parameter and registers the function only to that database, the function is not stored in the database file. Therefore when the program that calls sqlite_create_function() has terminated, the user-defined function is no longer available to SQLite.

If required within a function's code, the data contained in the pointer pUserData in the sqlite_create_function() call can be referenced using the sqlite_user_data() function, which has this prototype:

void *sqlite_user_data(sqlite_func *)

Creating Aggregating Functions

The process to create an aggregating functionone that applies a calculation across all the rows returned by a query or across similar rows indicated by a GROUP BY clauseis slightly different from a regular function.

The interface call sqlite_create_aggregate() takes two function pointer arguments. One function is called for each row in the result setthe step functionand one is called once all the rows have been processedthe finalize function. The prototype is as follows:

int sqlite_create_aggregate (
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xStep) (sqlite func *, int, const char **),
  void (*xFinalize)(sqlite_func*),  void pUserData
);

To see an example, let's take a look at how SQLite actually implements the sum() aggregating function. The source code for this and many other built-in functions can be found in func.c in the SQLite source distribution.

The step function sumStep() is as follows:

static void sumStep(sqlite_func *context, int argc, const char **argv){
  SumCtx *p;
  if( argc<1 ) return;
  p = sqlite_aggregate_context(context, sizeof(*p));
  if( p && argv[0] ){
    p->sum += sqliteAtoF(argv[0], 0);
    p->cnt++;
  }
}

SumCtx is a data structure suitable for storing the values needed to compute the return value; in this case a running total is added to each step to produce the sum. In fact because the avg() function is computed from the sum divided by the number of elements, the SumCtx structure also includes a running count of the number of terms and is used for both functions.

typedef struct SumCtx SumCtx;
struct SumCtx {
  double sum;     /* Sum of terms */
  int cnt;        /* Number of elements summed */
};

The sqlite_aggregate_context() function provides access to an area of memory that is available only to the particular instance of a function called from SQL, and the memory is automatically freed after the related finalize function has been called.

The work done by the step function for sum() is actually quite simple. The running total is incremented by the current value and the running count is incremented by one:

p->sum += sqliteAtoF(argv[0], 0);
p->cnt++;

The function sqliteAtoF() is a SQLite-safe version of the regular atof() function to convert an ASCII string to a floating-point number and is defined in util.c. If the locale setting requires it, sqliteAtoF() will accept a comma as a decimal point instead of a period character.

The sumFinalize() function references the context structure containing the totals computed in the step function and uses sqlite_set_result_double() to return a floating-point result containing the sum of all the values:

static void sumFinalize(sqlite_func *context){
  SumCtx *p;
  p = sqlite_aggregate_context(context, sizeof(*p));
  sqlite_set_result_double(context, p ? p->sum : 0.0);
}

We mentioned before that the avg() function can reuse much of the code for the sum() function, and in fact the same step function is used to create avg() in SQLite. The following code shows the avgFinalize() function.

static void avgFinalize(sqlite_func *context){
  SumCtx *p;
  p = sqlite_aggregate_context(context, sizeof(*p));
  if( p && p->cnt>0 ){
    sqlite_set_result_double(context, p->sum/(double)p->cnt);
  }
}


Chapter 7. The Perl Interface

In this chapter we will look at the Perl interface to SQLite. Databases are accessed from Perl scripts using the Database Interface (DBI) and a Database Driver (DBD) for SQLite. If you have used DBI/DBD to interface Perl with other database systems, much of the procedure to communicate with SQLite will be familiar to you.

This chapter gives an overview of Perl's DBI in general and also details the attributes and methods specific to the SQLite DBD.


Preparing to Use the SQLite Interface

To access a SQLite database from Perl you need to install both the DBI module and the DBD module for SQLite.

If you do not already have DBI installed, use cpan to download it from the Comprehensive Perl Archive Network and install it.

# cpan
cpan> install DBI

On Windows platforms using the ActivePerl distribution, use the ppm.bat script to install Perl modules.

C:\perl\bin> ppm.bat
ppm> install DBI

Note that it can take a while to download, configure, compile, and install CPAN modules. If you are accessing CPAN for the first time, a set of modules will also be downloaded first to update your system.

To add the SQLite DBD module to your system, install DBD::SQLite2 from CPAN.

# cpan
cpan> install DBD::SQLite2

The DBD::SQLite2 package includes as much of SQLite as it needs, so there is no need to have the SQLite libraries on your system before installing the DBD moduleyou can add SQLite support to Perl on a fresh system without needing to download anything from sqlite.org, although you will probably want to install the sqlite monitor tool too.

Note

As we have chosen to stick with the existing, stable, and well-supported SQLite 2 engine throughout this book, you should install DBD::SQLite2 for a compatible Perl database driver. The original DBD::SQLite module uses the latest SQLite libraryversion 3which uses a different database file format than the previous version.

Though the Perl DBI abstracts the actual database back end and the examples in this chapter will work with whatever SQLite version you use, you also need to be using the appropriate sqlite tool for your SQLite library version in order to read your databases. You can read about the changes in SQLite 3 in Appendix I, "The Future of SQLite."


About the Perl DBI

Perl's Database Interface uses a database abstraction model, so it doesn't really matter what the underlying database is. The DBI module calls the appropriate Database Driver module and passes off the SQL commands for execution.

There are many more DBD modules than you are ever likely to use, supporting all the major database systems and many of the minor ones. The naming convention is DBD::dbname, for example DBD::Oracle for Oracle or DBD::Sybase for Sybase (and SQL Server). There's also DBD::ODBC for other ODBC-enabled databases that don't have their own specific DBD module and even DBD::CSV to interact with comma-separated-values files using DBI.

The DBD module we are interested in is called DBD::SQLite2, and the way in which queries from your Perl script interact with DBI and the SQLite DBD driver is shown in Figure 7.1.

Figure 7.1. The Perl Database Interface model.

The DBI module is loaded into your script with this simple command:

use DBI;

You do not need to use any specific DBD module as DBI will take care of that when it is needed. This makes it easy to write highly portable database applications in Perl because only a single instruction needs to be changed to tell DBI to work with a different database. The instruction looks like this for SQLite:

$dbh = DBI->connect("DBI:SQLite2:dbname=dbfile", "", "");

The prototype for DBI->connect() has three parametersa data source, username, and password. As SQLite does not use user-based authentication, the second and third parameters are always blank.

The data source contains three parts separated by colons: the keyword DBI, the database typein this case SQLiteand an expression indicating the name of the database. The filename given as dbfile can include an absolute or relative path or will be opened from the current working directory if no path is given.

Getting Information About the DBI

The DBI module will report back its supported DBD drivers through the available_drivers() method. The script in Listing 7.1 grabs the available drivers as an array and loops through to print each entry to screen.

Listing 7.1. Using available_drivers() to Check What DBD Modules Are Installed
#!/usr/bin/perl -w

use DBI;
use strict;

my @drivers = DBI->available_drivers();
foreach my $driver (@drivers) {
  print "$driver \n";
}

The output will look similar to this:

# ./listing7.1.pl
CSV
DBM
ExampleP
File
Proxy
SQLite
Sponge

Your output may be different if you have additional DBD modules already installed on your system. Of those shown in the preceding example, only CSV, DBM, and SQLite are actual database drivers. ExampleP is a simple example of a driver that can be used to help write new DBD drivers. The others are drivers that perform internal operations shared by other DBD drivers.

DBI also provides the installed_versions method, which will give a formatted report of both the supported DBD modules and their installed versions as well as details of the Perl and operating-system versions. This method can easily be called from the command line as follows:

$ perl -MDBI -we 'DBI->installed_versions()'
  Perl            : 5.008003    (i386-linux-thread-multi)
  OS              : linux       (2.4.21-4.elsmp)
  DBI             : 1.43
  DBD::Sponge     : 11.10
  DBD::SQLite2    : 0.32
  DBD::Proxy      : 0.2004
  DBD::File       : 0.31
  DBD::ExampleP   : 11.12
  DBD::DBM        : 0.02
  DBD::CSV        : 0.21

Using DBD Drivers

Let's look at how the same query can be performed using two different DBD modules by simply changing the DBI->connect() instruction. Listing 7.2 shows a script that will connect to a SQLite database, create a very simple table, and insert a few rows of data.

Listing 7.2. Creating Some Simple Data Records Using Perl DBI
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "");

my $sql = "CREATE TABLE mytable (mynumber INTEGER)";
$dbh->do($sql);

for (my $i=1; $i<=5; $i++) {
  $sql = "INSERT INTO mytable VALUES ($i)";
  $dbh->do($sql);
}

The DBI->connect() call opens up a database called perldb from the current working directory. Unless you already have a database by this name in that directory, a new file will be created.

$dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "");

To send an SQL statement to be processed by DBI, we use the do() method on a database handle.

$dbh->do($sql);

Running the script in Listing 7.2 will create a new database file called perldb (unless one already existed, of course), and we can view the schema of the new table and view the rows that were inserted.

$ sqlite perldb
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE mytable (mynumber INTEGER);
sqlite> SELECT * FROM mytable;
1
2
3
4
5

Now let's try the same thing using a different DBD module. One of the simplest, although crudest, ways to store data to the filesystem is in a comma-separated-values file, and Perl can use a DBD module to communicate with CSV files using an SQL interface. If your system does not already have it, install the DBD::CSV module using cpan.

cpan> install DBD::CSV

To adapt the script from Listing 7.2 to save data to comma-separated files rather than SQLite, simply change the DBI->connect() instruction so that it reads as follows:

$dbh = DBI->connect("DBI:CSV:f_dir=/tmp", "", "");

The data source for the CSV driver has an optional assignment, f_dir, for the output directory and uses the current directory if this is not given. One CSV file is written per table, rather than one file for the entire database in SQLite. The operation is successful in creating a file called mytable containing the same data rows as before, as you can see by examining the new file created in /tmp.

$ cat /tmp/mytable
mynumber
1
2
3
4
5

Though the way the data files are written is different with the CSV driver, the same SQL code that we wrote to create tables in SQLite will also create a CSV if the data source is changed. The same procedure can be used for any supported DBD module, whether it is a filesystem-based database or a client/server RDBMS.

For the remainder of this chapter we will look at examples with DBD::SQLite2, but do remember that the Perl interface for SQLite is not specific to SQLite. Indeed you may already have an existing Perl application that uses DBI with a different database that could easily be ported to SQLite by simply changing the DBD module in the data source.


Using the SQLite DBD

In this section we will look at the methods and attributes available for a DBI object with examples specific to the DBD::SQLite2 module.

Opening and Closing the Database

As we saw before, the DBI->connect() function is used to open a database, and for a SQLite database no username or password arguments are required. The usage is always as follows:

$dbh = DBI->connect("DBI:SQLite2:dbname=dbfile", "", "");

Some basic error trapping is useful in case the connection to the database fails. Because SQLite will create a new database file with the given name if one does not exist, DBI->connect() will only fail if there is an I/O error, for instance file permissions not allowing the file to be opened or created in a particular directory, or no more disk space on the device.

The errstr property contains the most recent database error. The following example could be used to exit with an error message if SQLite is unable to open the specified dbfile:

$dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
       or die("Error: " . DBI::errstr);

To close the connection to a database, simply invoke the disconnect() method on your DBI object.

$dbh->disconnect();

Executing SQL Statements

We have already seen that the do() method can be used to pass SQL to the DBI module in order to execute a command. In fact, do() is good only for non-SELECT statements and the preferred method to send SQL commands to the DBI module is using a two-step process. do() is a convenience function that effectively combines these two steps.

First the query needs to be prepared by the SQL engine using prepare, after which it can be executed using the execute method.

Listing 7.3 creates the contacts table in SQLite with error trapping at every stage. The errstr property returns the most recent error message generated by the DBD, so any cause of error that causes the script to exit prematurely will be displayed to the screen.

Listing 7.3. Creating a Table Using DBI
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $sql = "CREATE TABLE contacts (   ".
          "  id INTEGER PRIMARY KEY, ".
          "  first_name TEXT,        ".
          "  last_name TEXT,         ".
          "  email TEXT)             ";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

my $ret = $sth->execute()
          or die("Cannot execute: " . DBI::errstr() );

The prepare() method returns a statement handle object, which in turn can invoke the execute() method to perform the query.

$sth = $dbh->prepare($sql);
...
$ret = $sth->execute();

Note

The prepare() method instructs SQLite to get ready to execute the statementthe database engine will tokenize the statement and work out how it will be executed when the time comesbut it does not parse the SQL statement. Any syntax errors in the SQL will only be reported when execute() is called.

Using Bind Variables

A powerful feature of DBI is that an SQL statement still needs to be prepared only once, even if it is to be executed repeatedly with different values in the statement. This is achieved using placeholders in the SQL and an array of bind values passed as an argument to execute() and gives a significant performance saving over preparing a query with static values each time it is executed.

A placeholder is indicated by the question mark symbol. For example, the following query could be prepared to create an SQL INSERT statement where the placeholders indicate that we will supply the values to be inserted at execution time. Note that no quotes are required around a placeholder even if a string value is the expected substitution.

INSERT INTO contacts (first_name, last_name, email) VALUES (?, ?, ?)

Listing 7.4 shows how a record is inserted using this technique.

Listing 7.4. Inserting a Record Using Bind Values
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $sql = "INSERT INTO contacts (first_name, last_name, email) ".
          "VALUES (?, ?, ?)";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

$sth->execute('Chris', 'Newman', 'chris@lightwood.net')
       or die("Cannot execute: " . DBI::errstr() );

The execute() method takes one argument for each placeholder in the order specified in the query. We can verify that the record was inserted as expected using the sqlite tool.

$ sqlite perldb
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> SELECT * FROM contacts;
id  first_name  last_name   email
--  ----------  ----------  --------------------
1   Chris       Newman      chris@lightwood.net

Let's extend this a little further to create a script that allows you to populate the contacts table by entering records from the keyboard.

The script in Listing 7.5 creates a loop that reads user input that is expected to be the three elements first_name, last_name, and email separated by commas. If the format of the input is valid, the record is inserted into the database; otherwise, the script will exit.

Listing 7.5. Inserting Records from User Input with a Single Prepared SQL Statement
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr);

my $sql = "INSERT INTO contacts (first_name, last_name, email) ".
          "VALUES (?, ?, ?)";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr);

while(<>) {
  chomp;
  my ($first_name, $last_name, $email) = split /,/;

  if (!$email) {
    print "Required format: first_name, last_name, email\n";
    next;
  }

  $sth->execute($first_name, $last_name, $email)
         or die("Cannot execute: ". DBI::errstr() );

  print "Inserted: $first_name, $last_name, $email\n";

}

The key part of this script is the while loop, which reads from standard input. Input is taken from stdin a line at a time, and we use chomp to strip the trailing newline characters.

while(<>) {
  chomp;

The expected data format is a comma-separated list of the three elements to be inserted into contacts. The split instruction breaks up the read line of input and the first three comma-separated elements are assigned to $first_name, $last_name, and $email respectively.

my ($first_name, $last_name, $email) = split /,/;

To validate the data record we test that a value has been assigned to $email. If there are less than three comma-separated values in the input, $email will be empty. If there are more than three elements, the fourth and subsequent elements are simply discarded.

if (!$email) {
  print "Required format: first_name, last_name, email\n";
  next;
}

If everything is okay, a row is inserted by calling the execute() method on the prepared SQL statement with the assigned variables.

$sth->execute($first_name, $last_name, $email)
       or die("Cannot execute: ". DBI::errstr );

Note

The placeholders in an SQL statement can only represent items that do not alter the execution plan. Therefore you cannot use a placeholder to substitute a table or column name.

The bind_param() function is used to set bind parameter values individually and provides a little more functionality than supplying a list of bind values to the execute() method by allowing you to give a data type hint.

In fact when bind parameters are passed to the execute() method, the DBI effectively calls bind_param() for each value in the list. The data type is assumed to be SQL_VARCHAR.

The syntax of bind_param() is as follows:

$sth->bind_param($p_num, $bind_value, $bind_type)

The parameters are numbered in the order they appear in the SQL starting at 1, and the number of the parameter you want to assign is passed in the $p_num argument.

The only data type constants appropriate for SQLite are SQL_VARCHAR and SQL_INTEGER. In order to use these constants in your script you must import them with the use command:

use DBI qw(:sql_types);

To produce the same result we saw in Listing 7.4 using bind_param(), instead of passing the bind values to execute(), the following lines would be required:

$sth->bind_param(1, $first_name, SQL_VARCHAR);
$sth->bind_param(2, $last_name, SQL_VARCHAR);
$sth->bind_param(3, $email, SQL_VARCHAR);
$sth->execute() or die "Cannot execute: ". DBI::errstr;

Bind values are safe when passed into an SQL statement. For example, the following execution of Listing 7.5 shows that a surname containing an apostrophe does not conflict with the implied quotation marks around that string value.

$ ./listing7.5.pl
Paddy,O'Brien,paddy@irish.com
Inserted: Paddy, O'Brien, paddy@irish.com

Had we performed the same INSERT command using static values in the script, some kind of delimiting would be required to avoid the query looking like this:

INSERT INTO contacts (first_name, last_name, email)
VALUES ('Paddy', 'O'Brien', 'paddy@irish.com')

This query would cause an error upon execution as the parser would not be able to determine that the apostrophe in O'Brien was not in fact a closing quotation mark around the string 'O'.

In Perl we can call the DBI method quote() to put a value into the right type of quotes for the underlying database (single quote for DBD::SQLite2) while adding the necessary delimiters to make them safe for use within an SQL command. Listing 7.6 performs the quote method on each of the values in an INSERT statement.

Listing 7.6. Using the quote Method to Safely Delimit String Values
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $first_name = "Paddy";
my $last_name = "O'Brien";
my $email = "paddy\@irish.com";

my $sql = sprintf("INSERT INTO contacts (first_name, last_name, email) \n".
                                "VALUES (%s, %s, %s)",
                  $dbh->quote($first_name),
                  $dbh->quote($last_name),
                  $dbh->quote($email));

print $sql . "\n";

The output from running Listing 7.6 displays the SQLite-safe query text.

$ ./listing7.6.pl
INSERT INTO contacts (first_name, last_name, email)
VALUES ('Paddy', 'O''Brien', 'paddy@paddy.com')

Remember that SQLite delimits the apostrophe character by doubling itthe occurrence of '' in a string enclosed by single quotes means that the string contains one apostrophe.

Note

Because quote() encloses the string in quotes as well as delimiting any awkward characters, it should not be used with placeholders and bind values. As we have already seen, apostrophes in bind values do not need to be delimited.

Using Commands That Change the Database

When an UPDATE or DELETE operation is performed, it is usually the intention to alter one or more database rows. The number of rows affected by the statement can be found by looking at the rows property of the statement handler.

$sth->execute();
print $sth->rows;

The number of affected rows is also the return value from execute, which provides a handy shortcut to this information. Because the do() method is a convenience function that prepares and executes an SQL command in one go, you can also look at the return value from do() to find the number of affected rows.

For example, after the following statement is executed, $num will contain the number of rows that has been deleted:

$num = $dbh->do("DELETE FROM contacts WHERE first_name = 'Chris'");

When an INSERT is performed on a table with an INTEGER PRIMARY KEY fieldas in our previous examplesSQLite assigns the next numerical value to that field if the inserted value is NULL. The SQLite DBD module extends DBI to provide a private method for retrieving this value. It is called as follows:

$id = $dbh->func("last_insert_rowid").

Note

The INTEGER PRIMARY KEY property of a database column is specific to SQLite, so last_insert_rowid is implemented by DBD::SQLite2, not the DBI module. Therefore, be aware that scripts that use this private method cannot be expected to work with other DBD modules.

Listing 7.7 performs an UPDATE followed by a INSERT on the contacts table, and displays the number of affected rows and the assigned INTEGER PRIMARY KEY field respectively.

Listing 7.7. Finding Affected Rows and the Assigned Value of a Primary Key
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $sql = "UPDATE contacts ".
          "SET first_name = upper(first_name)";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

my $num = $sth->execute or die("Cannot execute: ". DBI::errstr() );

print "Updated " . $sth->rows ." rows \n";

$sql = "INSERT INTO contacts (first_name, last_name, email) ".
       "VALUES ('Bill', 'Williams', 'bill\@williams.com')";

$sth = $dbh->prepare($sql)
       or die("Cannot prepare: " . DBI::errstr() );

$sth->execute() or die("Cannot execute: ". DBI::errstr() );

print "Inserted with ID " . $dbh->func("last_insert_rowid"). "\n";

The following output will be generated, depending on the actual number of records already in the contacts table:

$ ./listing7.7.pl
Updated 4 rows
Inserted with ID 5

Transactions

Three DBI functions are provided to begin and end SQLite transactions. They map directly to the underlying SQL commands and each is equivalent to issuing the corresponding command shown in Table 7.1.

Table 7.1. Transaction Functions in Perl DBI

begin_work()

BEGIN TRANSACTION

commit()

COMMIT TRANSACTION

rollback()

ROLLBACK TRANSACTION

Fetching Records from the Database

We have already seen how an INSERT or UPDATE command returns the number of affected rows as the result of the execute method. Now let's take a look at how the result of a SELECT statement is processed using the Perl DBI.

The fetchrow_array() method can be used on an executed statement handler to return a list of the selected column values. The first time it is called, the first row from the query is returned, and on subsequent calls the record pointer is advanced to return the next row in sequence.

Listing 7.8 shows how fetchrow_array() can be used in a loop to output every row from a table.

Listing 7.8. Using fetchrow_array() to Output the Contents of a Table
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $sql = "SELECT first_name, last_name FROM contacts";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

$sth->execute() or die("Cannot execute: ". DBI::errstr() );

while( my ($first_name, $last_name) = $sth->fetchrow_array() ) {
  print "$first_name $last_name \n";
}

The while loop assigns two variables from the elements of the array returned by fetchrow_array() for each iteration.

while( my ($first_name, $last_name) = $sth->fetchrow_array() ) {

After the last row of data has been fetched, the while condition is false and the loop exits. The output generated by Listing 7.8 is as follows:

$ ./listing7.8.pl
CHRIS Newman
PADDY O'Brien
BILL Williams

As an alternative to fetchrow_array(), you can use the fetchrow_arrayref() method, which returns a reference to an array containing each row's elements. A while loop to achieve the same result as Listing 7.8 using this method would look like this:

while (my $ref = $sth->fetchrow_arrayref()) {
  print "@{$ref} \n";
}

The data rows can also be returned as a hash, using the fetchrow_hash() method. A reference to a hash is returned where the hash contains column name and value pairs.

while (my $ref = $sth->fetchrow_hashref()) {
  print "$$ref{'first_name'} $$ref{'last_name'}\n";
}

Note

Where values in the table are NULL, they are seen in Perl as undef, regardless of which method you choose to fetch the data.

The functions that begin fetchrow_ also have corresponding functions that begin fetchall_. The difference, as you might expect, is that the entire dataset returned from the query is fetched all at once, rather than a row at a time.

Listing 7.9 shows how the result fetched by fetchall_arrayref() can be used in Perl.

Listing 7.9. Using fetchall_arrayref to Fetch an Entire Dataset
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

my $sql = "SELECT * FROM contacts ORDER BY first_name";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

$sth->execute() or die("Cannot execute: ". DBI::errstr() );

my $ref = $sth->fetchall_arrayref();

foreach my $row (@{$ref}) {
  print "@$row\n";
}

The output looks like this, with the columns displayed in the order they appear in the table schema:

$ ./listing7.9.pl
3 BILL Williams bill.com
1 CHRIS Newman chris@lightwood.net
2 PADDY O'Brien paddy.com

An optional argument to fetchall_arrayref() can be used to fetch only a subset of the columns returned by the query into the array, known as a slice. The argument should be a hash reference of the column numbers to be returned, where columns are numbered from zero starting with the first in the selected list. Negative numbers can be used to count from the end of the column list.

For example, the following statement would cause only the second and last columns from the query to be put into the array:

$ref = $sth->fetchall_arrayref([1,-1]);

An optional second argument allows the maximum number of rows that will be created in the array to be specified. A non-zero value will restrict the dataset if it is larger than the total number of rows returned by the query, including the LIMIT clause if there is one.

If either argument is not required, it can be specified as undef. The following statement does not slice the array but limits the number of rows returned to 2:

$ref = $sth->fetchall_arrayref(undef, 2);

The fetchall_hashref() method requires an argument that gives the name of the field to be used as the key of the returned hash. The key field argument may be the column name or number.

The following statement calls fetchall_hashref() on contacts using the email column as the key:

$ref = $sth->fetchall_hashref('email');

Given an email address, we can now use this hash to find other information:

$first_name = $ref->{'chris@lightwood.net'}->{'first_name'};
$last_name = $ref->{'chris@lightwood.net'}->{'last_name'};

The finish method that can be called on a statement handle is used to destroy the handle when it is no longer needed. It is simply called as follows:

$sth->finish();

However, finish() should only be called when you are done with reading data from the statement handle and know that there is more data still to come. If any of the fetch instructions have come to the end of the dataset, there is no need to call finish().

Error Checking

The error checking we have come across so far has tested the return code of a function and taken action to display the error and exit if necessary. This might look like the following example using a traditional if condition to check for success:

$ret = $sth->execute();
if (!$ret) {
  print DBI::rrstr;
  exit;
}

Or it can be done in the more concise syntax using or die.

$ret = $sth->execute()
       or die("Cannot execute: " . DBI::errstr);

There is an even more concise way to apply error checking to all your database function calls. The Perl DBI implements automatic error checking using two built-in exceptions, PrintError and RaiseError, enabled by setting attributes with those names against the database handle.

The values of PrintError and RaiseError can be set at the time the database connection is established via the attr argument, using 1 to turn that feature on and 0 to turn it off.

To create a database connection with automatic error checking handled using RaiseError, you would use a statement similar to the following:

 $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "", {
  PrintError => 0,
  RaiseError => 1
} );

The PrintError exception causes the warn() function to be called with the error stringthe value of the errstr propertyas its argument. The error message will be displayed, but program execution will not stop.

The RaiseError exception causes the die() function to be called, halting program execution. If both PrintError and RaiseError are enabled, warn() is called first, then die().

The status of each exception can be changed mid-script simply by reassigning the value of that attribute. To turn off RaiseError, for instance, you would do the following:

$dbh->{RaiseError} = 0;

Tracing

The Perl DBI includes a powerful tracing mechanism that allows you to follow a query's execution to aid debugging.

Tracing is enabled using the trace() method, which can be called on a database handle or on the DBI itself to alter the default settings for every database handle opened in the script. Its argument is a trace level from 0 to 15; the values for which are shown in Table 7.2.

Table 7.2. Trace Level Values and Their Meanings

0

Trace disabled.

1

Trace DBI method calls returning with results or errors.

2

Trace method entry with parameters and returning with results.

3

As level 2, with some additional high-level information from the DBD and some internal information from the DBI.

4

As level 3, with more detailed information from the DBD.

515

Each higher level adds more obscure tracing information.

Trace level 1 should give a good overview of what is going on in your script. Level 2 will give some extra information that can be useful when debugging, and the higher levels are really only any use if you are trying to trap a specific problem.

To perform a simple trace of your script using trace level one, use the following command:

DBI->trace(1);

Trace output is written to stderr by default; however, an optional second parameter to trace() allows this to be changed.

Listing 7.10 creates a new table, numbers, and performs a simple loop to insert the first few square numbers into the num column. This is a trivial example to show tracing in action, but we will use the data in this listing later in this chapter.

Listing 7.10. Inserting a Series of Square Numbers
#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

$dbh->trace(1);

my $sql = "CREATE TABLE numbers (num INTEGER)";
$dbh->do($sql);

$sql = "INSERT INTO numbers (num) VALUES (?)";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

for (my $i=1; $i<=9; $i++) {
  $sth->execute($i * $i)
         or die("Cannot execute: " . DBI::errstr() );
}

Tracing is set to level 1 at the DBI level. Running this script produces trace output similar to the following:

$ ./listing7.10.pl
    DBI::db=HASH(0x9b88b90) trace level set to 0x0/1 (DBI @ 0x0/0)
in DBI 1.43-ithread (pid 2509)
    <- do('CREATE TABLE numbers (num INTEGER)')= '0E0' at
listing7.10.pl line 11
    <- prepare('INSERT INTO numbers (num) VALUES (?)')= DBI::st=HASH(0x9b8bd0c)
at listing7.10.pl line 15
    <- execute(1)= 1 at listing7.10.pl line 19
    <- execute(4)= 1 at listing7.10.pl line 19
    <- execute(9)= 1 at listing7.10.pl line 19
    <- execute(16)= 1 at listing7.10.pl line 19
    <- execute(25)= 1 at listing7.10.pl line 19
    <- execute(36)= 1 at listing7.10.pl line 19
    <- execute(49)= 1 at listing7.10.pl line 19
    <- execute(64)= 1 at listing7.10.pl line 19
    <- execute(81)= 1 at listing7.10.pl line 19
!   <- DESTROY(DBI::db=HASH(9ae4248))= undef during global destruction

Tracing can also be enabled from the shell without needing to amend your scripts using the DBI_TRACE environment variable. If DBI_TRACE is assigned a non-zero numeric value it will set the trace level for the DBI in Perl scripts to that value. If the number is followed by =filename, trace output will be redirected to that file.

For example, to run a script in trace level 2 with output written to TRace.log, the following would work in the Bourne shell:

$ DBI_TRACE=2=trace.log perl myscript.pl


Adding New SQL Functions

A powerful feature of the SQLite library is the ability to add user-defined functions to the SQL language. Because this feature is specific to SQLite, it is not part of the Perl DBI.

Instead, Perl provides the facility to create user-defined functions through two private methods in the SQLite DBD.

Creating Functions

You can register a new function using the private method create_function, called as follows:

$dbh->func( $name, $argc, $func_ref, "create_function" )

The three arguments required are the function name, the number of arguments the function will take, and a reference to the Perl function that is to be used whenever the SQL function is called.

The simplest way to see how a user-defined function is used is to register a built-in Perl function in the SQL language. The following statement uses a minimal inline function, such as $func_ref, which registers the Perl function rand() as the SQL function rand(). No arguments are required for the function, so $argc is zero.

$dbh->func( "rand", 0, sub { return rand() }, "create_function" );

SQLite's built-in random() function returns a random signed 32-bit integer, whereas Perl's rand() function returns a decimal between 0 and 1. You could see the difference by preparing and executing the following statement from a Perl script after the rand() function has been registered in SQL:

SELECT rand(), random();

User-defined functions have to be registered for each database connection where they are to be used. Such functions are available in SQL only for the duration of that connectionthey are not saved to the database itself or made available to other connection objects within the same script. Of course, user-defined functions are designed to allow you to add custom functions to SQL, so let's look at a more complex example (see Listing 7.11).

Listing 7.11. Creating a User-Defined Function in Perl
#!/usr/bin/perl -w

sub altcaps {
  my $str = $_[0];
  my $newstr = "";
  for ($i=0; $i<length($str); $i++) {
    $char = substr($str, $i, 1);
    if ($i%2) {
      $newstr .= uc($char);
    }
    else {
      $newstr .= lc($char);
    }
  }
  return $newstr;
}

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
       or die("Cannot connect: " . DBI::errstr() );

$dbh->func( "altcaps", 1, "altcaps", "create_function" );

my $sql = "SELECT altcaps(last_name) FROM contacts";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

$sth->execute() or die("Cannot execute: ". DBI::errstr() );

while ( my $first_name = $sth->fetchrow_array() ) {
  print "$first_name \n";
}

Listing 7.11 creates a new function in Perl called altcaps(), which converts the case of a string so that it contains alternating upper- and lowercase characters. The function is then registered as an SQL function with the same name and is used to select a modified version of the last_name column from contacts.

The output from running this script is as follows:

nEwMaN
o'bRiEn
wIlLiAmS

In this example we specified a $argc value of 1 as only a single string argument is required. SQLite will return an error code if the function is passed too many or too few arguments.

For example, if the SQL statement is changed to

$sql = "SELECT altcaps(first_name, last_name) FROM contacts";

The following error is produced:

Cannot execute: wrong number of arguments to function altcaps()
at ./func.pl line 30.

However, you can call the user-defined function as many times as you like in the same query. For example, this is a valid usage of the custom function:

$sql = "SELECT altcaps(first_name), altcaps(last_name) FROM contacts";

Note

If $argc is 1, any number of arguments can be passed to the function. This simply disables validation of the number of arguments in the SQL parser; you must still deal with the passed-in arguments in a sensible way in your Perl function.

Creating Aggregating Functions

SQLite also allows you to create new aggregating functions through the Perl interface. An aggregating function is one that applies a calculation across all the rows returned by a query or across similar rows indicated by a GROUP BY clause.

The private method in DBD::SQLite2 that is used to create an aggregating function is called create_aggregate.

$dbh->func( $name, $argc, $pkg, "create_aggregate" );

This is called in much the same way as create_function, but the $pkg argument is a reference to a package containing the steps necessary to implement an aggregating function.

Three steps are required:

  • The new() method is called once to initialize an aggregator object upon which the step() and finalize() methods will be called.

  • The step() method is called once for each row in the aggregate.

  • The finalize() method is called once after all the rows in the aggregate have been processed and returns the computed value.

In Listing 7.12 we create a new aggregating function, median(), to calculate the median of a set of numbers. The median is a value such that half the values in the set are above the median and half are below it. The SQLite built-in avg() function computes the mean averagethe sum of all values divided by the number of values.

Listing 7.12. Creating an Aggregating Function Using the Perl Interface
#!/usr/bin/perl -w

package median;

sub new {
  bless [];
}

sub step {
  my ( $context, $value ) = @_;
  push @$context, $value;
}

sub finalize {

  my $context = $_[0];

  if (!@$context) {
    return undef;
  }

  my @sorted = sort { $a <=> $b } @$context;
  my $count = @sorted;

  my $median;
  if ($count%2 == 1) { # Odd number of elements
    $median = $sorted[int($count/2)];
  }
  else {               # Even number of elements
    $median = ($sorted[$count/2] + $sorted[($count/2)-1] ) / 2;
  }

  return $median;
}

use strict;
use DBI;

my $dbh = DBI->connect("DBI:SQLite2:dbname=perldb", "", "")
          or die("Cannot connect: " . DBI::errstr() );

$dbh->func( "median", 1, "median", "create_aggregate" );

my $sql = "SELECT median(num), AVG(num) FROM numbers";

my $sth = $dbh->prepare($sql)
          or die("Cannot prepare: " . DBI::errstr() );

$sth->execute() or die("Cannot execute: ". DBI::errstr() );

my ($median, $mean) = $sth->fetchrow_array();
print "Median: $median \n";
print "Mean: $mean \n";

Let's look at the three methods that make up the median package, beginning with new().

sub new {
  bless [];
}

This simple function is all that's needed to initialize an aggregator.

The step() method is also quite simple. Each $value that is to be aggregated is read in turn and pushed onto context. We do all the hard work in the finalize() function, so step() is just used to gather the required data into a workable format.

sub step {
  my ( $context, $value ) = @_;
  push @$context, $value;
}

The finalize() method begins by reading context and making sure it contains at least one element.

my $context = $_[0];

if (!@$context) {
  return undef;
}

If there are no elements from which to find the median, undef is returnedtranslated to a NULL result in SQL.

To find the median value we need to look at the elements in numerical order, so a sort is performed on context.

@sorted = sort { $a <=> $b } @context;

The rule to find the median where there is an odd number of values is simply to find the value of the middle element in the list.

$count = @sorted;

if ($count%2 == 1) { # Odd number of elements
  $median = $sorted[int($count/2)];
}

For an even number of values, the median is the midpoint between the two middle values.

else {               # Even number of elements
  $median = ($sorted[$count/2] + $sorted[($count/2)-1] ) / 2;
}

Finally we return $median, which will become the result of the SQL aggregating function.

return $median;

We will use the data added by Listing 7.10 to test this new function, so we need some data to work from. This script created a table named numbers and inserted a sequence of square numbers into the num column.

The square numbers that were inserted, in order, are

1 4 9 16 25 36 49 64 81

We can see from looking at this sequence that the median will be 25, as there are four numbers on either side. The mean average, on the other hand, will be a bigger value as the later numbers are relatively large numbers compared to the earlier numbers in the sequence.

Listing 7.12 fetches both the median and the mean average of the rows using the new user-defined function and the SQLite built-in:

$sql = "SELECT median(num), AVG(num) FROM numbers";

Running the script shows the calculated values as follows:

$ ./listing7.12.pl
Median: 25
Mean: 31.6666666666667


Chapter 8. The Tcl Interface

Tcl (Tool Command Language) is a flexible, portable scripting language interpreter that is ideal for rapid application prototyping. With its simple syntax and in conjunction with Tk, a graphical user interface toolkit shipped as standard with Tcl, Tcl makes it possible to create powerful GUI applications very quickly.

In this chapter we look at how to add database functionality to your application using SQLite's Tcl interface.


Preparing to Use the Tcl Interface

The Tcl interface is included with the SQLite source distribution. Compiling from source on a Unix system creates the library file libtclsqlite.so, which should be installed to a valid library location.

A binary distribution of the TCL library can be obtained from http://www.sqlite.org/download.html for both Linux and Windows, in compressed formats named tclsqlite.so.gz and tclsqlite.zip respectively.

The library file should go in a location from which Tcl can find the package. On Linux this would usually be a subdirectory of /usr/share/tcl; on Windows, sqlite.dll is typically extracted to C:\tcl\lib.

Assuming the subdirectory is called TclSqlite, running the following command from Tcl will generate pkgIndex.tclwhich is required in order for Tcl to be able to locate the packageon a Linux system:

% pkg_mkIndex -direct /usr/share/tcl/TclSqlite *.so

On Windows, the equivalent command would be

% pkg_mkIndex -direct C:\tcl\lib\TclSqlite *.dll

If you installed using the RPM distribution on a compatible Linux system, tclsqlite.so will already have been installed to a directory from which it can be imported into tclsh or wish.


Using the Tcl Interface

Now that you have the SQLite Tcl library installed, let's look at how the interface is used within a Tcl script. In this section you'll see how to open and close a database and to issue commands that insert, update, or delete rows, and query the database. You'll also learn how to handle the resulting dataset in your script.

Opening and Closing a Database

To begin using the SQLite library from Tcl, your scripts first have to import the package.

package require sqlite

Run interactively, tclsh will display the version number of the SQLite library.

$ tclsh
% package require sqlite
2.0

You can access SQLite via Tcl by using a single command named sqlite, which you call as follows:

sqlite dbcmd database-name

The database file given in the database-name argument is openedor created if it does not exist. SQLite will attempt to open database-name from the current directory unless a path is specified.

A new composite command with the name given for dbcmd is then created for use within Tcl. The new command interfaces with an open SQLite database using a number of methods, similar to the way in which Tk widgets are created.

The first method we'll look at is the simplestthe close method instructs SQLite to close an open database, after which it will destroy the dbcmd command.

The first Tcl command in the following example would open a database called tcldb from the current directory and bind it to the command db1 within Tcl. The second command then closes the database, after which the command db1 will no longer be available.

$ tclsh
% package require sqlite
2.0
% sqlite db1 tcldb
0x95c7eb0
% db1 close

As usual, the error messages Tcl displays are helpful and generally no further error-trapping code is required. For instance, if you had tried to execute the preceding code in a directory that you did not have permission to write to, the error displayed would look like this:

$ tclsh
% package require sqlite
2.0
% sqlite db1 tcldb
unable to open database: tcldb

Executing SQL Statements

SQL statements are passed to SQLite from Tcl via the eval method. In its simplest form, the syntax is as follows:

% dbcmd eval sql-statement

The script in Listing 8.1 shows how a simple table called contacts can be created from a Tcl script.

Listing 8.1. Executing a CREATE TABLE Statement Using the TCL Interface
package require sqlite

sqlite db1 tcldb
db1 eval {
  create table contacts (
    id INTEGER PRIMARY KEY,
    first_name CHAR,
    last_name  CHAR,
    email      CHAR UNIQUE
  )
}

The SQL command in Listing 8.1 is formatted using spaces and newlines to indent and separate the columns within the CREATE TABLE statement. There is no need to format the SQL statements passed to the eval method in any particular way; however, this formatting is stored in the sqlite_master database and will be retrieved exactly as it was typed when the schema is subsequently queried.

Executing the script should be silent unless there is an error, for instance if the tcldb database cannot be written to or the table cannot be created. If the script in Listing 8.1 is run a second time, for instance, the following error message will be displayed:

$ tclsh listing8.1.tcl
table contacts already exists
    while executing
"db1 eval {
  create table contacts (
    id INTEGER PRIMARY KEY,
    first_name CHAR,
    last_name  CHAR,
    email      CHAR UNIQUE
  )
}
"
    (file "listing8.1.tcl" line 4)

As usual, the error message produced is verbose enough that no further error trapping is required to find out what went wrong.

Using Commands That Change the Database

The script in Listing 8.2 prompts for three pieces of user input before generating an SQL statement to insert a new record into the contacts table.

Listing 8.2. Inserting a New Record with the TCL Interface
package require sqlite

puts "Enter first name:"
gets stdin first_name

puts "Enter last name:"
gets stdin last_name

puts "Enter email:"
gets stdin email

sqlite db1 tcldb

set qry "INSERT INTO CONTACTS (first_name, last_name, email)
         VALUES ('$first_name', '$last_name', '$email')"

db1 eval $qry

In this listing, we have taken an extra step to assign the SQL query to a new variable, qry, which is then passed to the eval database method. The following output shows successful execution of this script.

$ tclsh listing8.2.tcl
Enter first name:
Chris
Enter last name:
Newman
Enter email:
chris@datasnake.co.uk

We can verify that the record has been inserted using the sqlite tool:

sqlite> select * from contacts;
id   first_name  last_name   email
---  ----------  ----------  ---------------------
1    Chris       Newman      chris@datasnake.co.uk

This program terminates silently if there are no errors, or will return a fairly readable error message if there is a problem. As the email column has the UNIQUE attribute, we can see how an error from an INSERT statement is reported by trying to insert another record with the same email address.

...
Enter email:
chris@datasnake.co.uk
column email is not unique
    while executing
"db1 eval $qry
"
    (file "listing8.2.tcl" line 17)

The first line of the output contains the error column email is not unique and tells us the command that caused the error and the line it appeared on. The SQL command itself is not shown on screen and you would need to output the contents of qry to the screen to see it verbatim; although, in this case it's fairly obvious what the query would have been.

Another reason this script might fail is if there are single quote characters in the user input. When evaluating the query, the strings read from stdin are replaced literally, so if we try to add a name containing an apostrophe (the same ASCII character as the single quote in SQL), it causes a syntax error as shown in the following example:

Enter first name:
Paddy
Enter last name:
O'Brien
Enter email:
paddy@irish.com
unrecognized token: "@"
    while executing
"db1 eval $qry
"
    (file "listing8.2.tcl" line 17)

The error message this time is not so helpful, but if we consider the query that has actually been passed to the eval method we can see where the problem arises.

INSERT INTO CONTACTS (first_name, last_name, email)
VALUES ('Paddy', 'O'Brien', 'paddy@irish.com')

The apostrophe has skewed the pairing of single quotes in the querynow 'O' and ', ' are contained in quotes, and the SQL statement is nonsense. The first error the SQLite parser comes across is the @ sign occurring outside of quotes, so that is the error reported.

To avoid this problem you should escape single quotes before they are used inside an SQL statement. This can be done easily using string map. The following example shows how to modify the code from Listing 8.2 to grab user input for the last_name field to eliminate any problems with apostrophes in the name:

puts "Enter last name:"
gets stdin last_name
set last_name [string map {' ''} $last_name]

The string map command finds any occurrence of one single quote in last_name and replaces it with two quotes. The resulting string is assigned back to last_name. SQLite uses two adjacent single quotes to represent a single quote within a string, so with this change in place the SQL generated by our script for the previous example now looks like this:

INSERT INTO CONTACTS (first_name, last_name, email)
VALUES ('Paddy', 'O''Brien', 'paddy@irish.com')

For this example only the last_name field was a problem, but you should consider all of your user input to ensure that such problems are avoided.

The INSERT statement we've used in the preceding example creates an implicit transaction in which the single new record is added to the database. Where an explicit transaction is required around a group of statements, you can invoke a BEGIN TRANSACTION command through the SQLite Tcl interface, and similarly end the transaction with a COMMIT TRANSACTION or ROLLBACK TRANSACTION instruction.

Listing 8.3 shows how several INSERT commands can be executed in a single transaction using SQLite's Tcl interface.

Listing 8.3. Using a SQLite Transaction from Tcl to Insert Multiple Rows
package require sqlite

sqlite db1 tcldb

db1 eval { BEGIN TRANSACTION }

db1 eval {
  INSERT INTO CONTACTS (first_name, last_name, email)
  VALUES ('Tom', 'Thomas', 'tom@thomas.com')
}

db1 eval {
  INSERT INTO CONTACTS (first_name, last_name, email)
  VALUES ('Bill', 'Williams', 'bill@williams.com')
}

db1 eval {
  INSERT INTO CONTACTS (first_name, last_name, email)
  VALUES ('Jo', 'Jones', 'jo@jones.com')
}

db1 eval { COMMIT TRANSACTION }

puts "Last rowid inserted: [db1 last_insert_rowid]"

In Listing 8.3 we begin by passing the BEGIN TRANSACTION command to SQLite and end the transaction by saving changes to the database with the COMMIT TRANSACTION command.

At the very end of the script, we use a new method on the db1 command. The last_insert_rowid method returns the most recently assigned value of the INTEGER PRIMARY KEY fieldin this example, the id field on the contacts table. Running the script will produce output similar to the following:

$ tclsh listing8.3.tcl
Last rowid inserted: 5

When an UPDATE or DELETE operation is performed on a database, a number of rows will be changed or removedassuming of course that the WHERE clause finds some matching rows that the operation is to be performed upon. To find how many rows were affected, we can use the changes database method.

Listing 8.4 performs an UPDATE on the contacts table that converts the first_name and last_name fields to uppercase using the SQL function upper(). As there is no WHERE clause, every row in the table will be affected and the value returned by the changes method will be the total number of rows in the table.

Listing 8.4. Using the changes Method to Find How Many Rows Were Affected by an UPDATE
package require sqlite

sqlite db1 tcldb

db1 eval {
  UPDATE CONTACTS
  SET first_name = upper(first_name),
      last_name  = upper(last_name)
}

puts "Rows updated: [db1 changes]"

Running this script will result in the changes being made to the database and the number of rows that have been changed will be displayed.

$ tclsh listing8.4.tcl
Rows updated: 5

Note

The number of affected rows returned by the changes method is the total number of rows that could be affected by the operation regardless of whether or not the record is changed by itin other words the number of rows matched by the WHERE clause or the total number of rows in the table if no WHERE clause is given. If you were to run the script in Listing 8.4 a second time, it will still show the same number of changes having been made, even though the strings are already in uppercase and their values are not changed.

Fetching Records from the Database

We have already seen how to pass an SQL statement to SQLite using the eval database method. Now let's look at how the result of a SELECT statement can be processed in Tcl.

To simply assign the entire result set from a SELECT statement to a variable in Tcl, you can assign the return value of the eval method as shown in Listing 8.5.

Listing 8.5. Assigning the Output of a SELECT Statement to a Variable
package require sqlite

sqlite db1 tcldb

set result [db1 eval { SELECT first_name, last_name FROM contacts } ]

puts $result

When the script in Listing 8.5 is executed, the output will be similar to the following:

$ tclsh listing8.5.tcl
CHRIS NEWMAN PADDY O'BRIEN TOM THOMAS BILL WILLIAMS JO JONES

This could be more useful. Although everything we asked for in the SELECT has indeed been assigned to result, there is nothing to indicate what is what. To the eye it's fairly clear that each pair of words output makes up a full name, but the data is not in a format that we can work with in a program. This method of assigning the result of a query is really only suitable for simple, single-row results.

Fortunately the Tcl interface allows you to process the data returned by a SELECT query one row at a time by specifying the name of an array variable and a script immediately after the SQL code. The value of each column will be inserted into the named array and the supplied code will be executed once for each row in the query result.

Listing 8.6 shows how the same query on the contacts table can be processed one row at a time.

Listing 8.6. Processing the Result of a SELECT Query One Row at a Time
package require sqlite

sqlite db1 tcldb

db1 eval {
  SELECT first_name, last_name FROM contacts
} result {
  parray result
}

The code to be executed for each row returned is simply to dump the contents of the result array to screen using parray. By doing so we can see exactly what elements are created in this array, as shown in the following output:

$ tclsh listing8.6.tcl
result(*)                 = first_name last_name
result(first_name)        = CHRIS
result(last_name)         = NEWMAN
result(typeof:first_name) = CHAR
result(typeof:last_name)  = CHAR
result(*)                 = first_name last_name
result(first_name)        = PADDY
result(last_name)         = O'BRIEN
result(typeof:first_name) = CHAR
result(typeof:last_name)  = CHAR
...

Not only are the selected values made available as elements of result using their column names as the keys, but their respective data types are also fetched into an element with a key named typeof:column_name.

Additionally, for each row returned, result(*) will contain a list of columns returned by the query. If your query was SELECT * FROM tablename, the value of result(*) is, usefully, the list of actual column names returned and not just the * character.

If the name of the array variable given is an empty string, the value of each column will be stored in a scalar variable with the same name as the column itself and the subsequent code executed only once. This property of the SQLite Tcl interface is demonstrated in Listing 8.7.

Listing 8.7. Processing the Result of a SELECT Query One Row at a Time
package require sqlite

sqlite db1 tcldb

db1 eval { SELECT * FROM contacts } {} {
  puts "$first_name $last_name"
}

This time the values of all the columns in the contacts table are fetched into variables with the same names as the columns, so we can output the values from the first_name and last_name columns by referencing the first_name and last_name variables. The following output shows the result of running Listing 8.7:

$ tclsh listing8.7.tcl
CHRIS NEWMAN
PADDY O'BRIEN
TOM THOMAS
BILL WILLIAMS
JO JONES

The onecolumn Database Method

If you want to select only a single value from the database, the onecolumn database method can be used in place of eval. The behavior is identical except that only the first column of the first row of the result is returned, regardless of the number of columns and rows in the result of the query.

This is a convenience function only; the same behavior can be achieved with the eval method and using lindex in Tcl to fetch the first column of the first row. The following two pieces of code behave in the same way:

set result [db1 onecolumn { SELECT * FROM contacts }]

set result [lindex [db1 eval { SELECT * FROM contacts }] 0]

Validating an SQL Statement

The SQLite Tcl interface provides a database method that allows you to check whether or not an SQL statement is complete before attempting to execute it using eval or onecolumn. This method is called complete and is called as follows:

if [ dbcmd complete sql-statement ] {
  # Action if statement is complete
} else {
  # Action if statement is not complete
}

Note

The complete method is not a check for valid SQL syntax. Its underlying function is sqlite_complete() from the C/C++ library, for which the logic is little more than to make sure an SQL statement ends with a semicolon. It is a little cleverer than thisfor instance, semicolons can appear within a CREATE TRIGGER commandbut the statement is not complete until the semicolon following the keyword END.

The script in Listing 8.8 reads one line of input at a time and continues until a complete SQL statement has been entered.

Listing 8.8. Checking Whether an SQL Statement Is Complete
package require sqlite

sqlite db1 tcldb

gets stdin qry
while { ![ db1 complete $qry ] } {
  gets stdin line
  set qry "$qry $line"
}

puts "Query is complete:"
puts $qry

A typical execution of this script might look like this:

$ tclsh listing8.8.tcl
SELECT *
FROM contacts
WHERE email like '%.co.uk';
Query is complete:
SELECT * FROM contacts WHERE email like '%.co.uk';
result(*)                 = id first_name last_name email
result(email)             = chris@datasnake.co.uk
result(first_name)        = CHRIS
result(id)                = 1
result(last_name)         = NEWMAN
result(typeof:email)      = CHAR
result(typeof:first_name) = CHAR
result(typeof:id)         = INTEGER
result(typeof:last_name)  = CHAR

Adjusting the Database Timeout Action

By default, SQLite will not wait for a lock to clear on the database file before giving up on a database transaction. SQLite can support multiple simultaneous processes reading from a database, but everything must stop when a write action takes placeno reading or other writing can be done at the same time.

There will be situations where you want to adjust the length of time that SQLite will wait for a lock to clear before giving up. This can be done with the timeout method, called like this:

dbcmd timeout ms

The value ms is the maximum length of time to wait, in milliseconds. Therefore a value of 1000 would mean the maximum delay is one second.

The busy method is similar to the timeout method, but offers you more control over what action should be taken when a lock cannot be obtained within the specified amount of time.

The busy method specifies a Tcl procedure to be used as a callback whenever a locked database cannot be opened. The callback can perform any action you want before returning to the main program to try to obtain a lock again. The callback procedure takes a single count parameter, which tracks how many times it has been called, and must return zero to force the calling script to retry the database transaction. A non-zero value will cause the operation to be interrupted.

Listing 8.9 shows how the busy callback can be used. This example simply dumps off to the callback if the database is locked and prints a message to the screen. The script will allow the callback to be called five times before giving up on the lock.

Listing 8.9. Using a Busy Callback to Handle a Locked Database
package require sqlite

sqlite db1 tcldb

proc mycallback {file count} {
  puts "This is the callback function, try $count"
  if {$count < 5} {
    # Do something useful whilst waiting
    return 0
  } else {
    return 1
  }
}

db1 busy mycallback

db1 eval { UPDATE contacts SET first_name = upper(first_name) }

Unless the tcldb database is already locked, this script will execute silently. To test the busy callback, lock the database for writing by issuing a BEGIN TRANSACTION command from another session. The following output is produced when the script fails to obtain a lock on five successive tries:

$ tclsh listing8.9.tcl
This is the callback function, try 1
This is the callback function, try 2
This is the callback function, try 3
This is the callback function, try 4
This is the callback function, try 5
database is locked
    while executing
"db1 eval { UPDATE contacts SET first_name = upper(first_name) }"
    (file "listing8.9.tcl" line 17)

Adding New SQL Functions

The SQLite Tcl extension allows the SQL language to be extended with new functions implemented as Tcl code. This is a very powerful feature that means you can extend SQL on demand using a familiar language, without having to learn C or recompile SQLite.

Note

Although the Tcl extension supports regular user-defined functions, aggregating functionswhich are supported by some other language interfacescannot be created in Tcl.

The function database method is used to specify a Tcl function that is to be registered as an SQL function. It takes two arguments, the name of the Tcl function itself and the name with which it is to be referenced in SQLthe names can be different.

Listing 8.10 shows how a user-defined function that reverses the characters in a string can be added to SQL.

Listing 8.10. Registering a User-Defined SQL Function Using the Tcl Library
package require sqlite

sqlite db1 tcldb

proc reverse_string {str} {

  set len [string length $str]
  set rev ""

  for {set x [expr $len - 1]} {$x>=0} {incr x -1} {
    set rev $rev[string index $str $x]
  }
  return $rev
}

db1 function reverse reverse_string

puts [db1 onecolumn {SELECT reverse('Hello, world')}]

The Tcl function in this example is called reverse_string and takes a single string argument. The first thing we do is find the length of this string to set up a loop.

set len [string length $str]

Then we perform a loop taking one character at a time from the end of the string and building a new string in reverse order. Because the first character of a string is found at index 0, the loop actually runs from len-1 to 0.

for {set x [expr $len - 1]} {$x>=0} {incr x -1} {
  set rev $rev[string index $str $x]
}

Finally, with the reversed string stored in rev, we return the result of the operation.

return $rev

We register this function in SQL using the function method. Listing 8.10 assigns a different name to the SQL function than the Tcl functionthe SQL function is to be called simply reverse.

db1 function reverse reverse_string

Finally, to test our custom function we perform a SELECT on a fixed string. Of course, now that the SQL function is available, it can be applied to a database column or expression.

puts [db1 onecolumn {SELECT reverse('Hello, world')}]

Executing the sample script gives the following output, as expected:

dlrow ,olleH


Chapter 9. The Python Interface

Python is an interpreted object-oriented programming language that is portable across different platforms. Its popularity is due to a combination of its powerful features and the clarity and agility of its syntax. It is an ideal language for prototyping while still managing to keep the code readable and manageable.

You can access SQLite from Python by using the PySQLite module, available from http://pysqlite.sourceforge.net/. PySQLite provides an interface to SQLite compliant with the Python Database API Specification 2.0, so the programming interface should be familiar if you have used Python with another database in the past and should still be intuitive if not.

The Python Database API Specification can be found at http://www.python.org/peps/pep-0249.html.


Preparing to Use the Python Interface

Before you can install PySQLite, you need to be sure your system has the Python interpreter and the development libraries installed, as well as a C compiler. You also need to have a SQLite development librarylibsqlite.so or sqlite.dllinstalled, as PySQLite does not include it.

Some Linux distributions package the Python interpreter and development libraries separately, and simply selecting Python at install time may not have included the development package. On Red Hat Linux, for example, you need to make sure the python-devel package has been installed.

Download the PySQLite source from http://sourceforge.net/projects/pysqlite/ and extract it. The version number in the filename may be different from that shown.

$ gunzip -c pysqlite-0.5.1.tar.gz | tar xf -

Enter the pysqlite directory that has been created and issue the following command to compile PySQLite:

$ cd pysqlite
$ python setup.py build
running build
running build_py
creating build
creating build/lib.linux-i686-2.3
creating build/lib.linux-i686-2.3/sqlite
copying sqlite/__init__.py -> build/lib.linux-i686-2.3/sqlite
copying sqlite/main.py -> build/lib.linux-i686-2.3/sqlite
running build_ext
building '_sqlite' extension
[...]

The following step installs the PySQLite extension into system directories and so you will need to become the root user, if you are not already, using the su command.

# python setup.py install
running install
running build
running build_py
running build_ext
running install_lib
[...]

To test that PySQLite has installed correctly, run the test suite that is shipped with the source code. The following output indicates a successful installation:

# python tests/all_tests.py
................................................................................
..........................................
----------------------------------------------------------------------
Ran 122 tests in 0.289s

OK


Using the Python Interface

Now that you have the PySQLite interface installed, let's look at how the interface is used from a Python script. In this section we'll see how to open and close databases and issue commands to SQLite.

Opening and Closing a Database

In order to use the PySQLite functionality, Python must first import the sqlite module. Then you can call the sqlite.connect() constructor method in order to open a database. The following is an example with Python running in interactive mode:

$ python
Python 2.3.3 (#1, May  7 2004, 10:31:40)
[GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite
>>> cx = sqlite.connect("pydb")
>>>

In its simplest form, the constructor takes a single database name parameter. The filename given can include a relative or absolute path and will be read from the current working directory if no path is specified. If that file exists, that database is opened; otherwise, a new empty database is created with the given filename. A connection object is returned, assigned to cx in this example.

If you followed the preceding example and no error occurred, exit Python and you'll see that the file pydb has been created in the current directory.

To close the connection to a database, simply call the .close() method on the connection object.

>>> cx.close()

After the connection has been closed, cx becomes unusable and an Error exception will be raised if you attempt to perform any operation on it.

Executing SQL Commands

To perform any database operation you must first create a cursor object using a valid database connection object. SQLite does not use the concept of cursors internally; however, PySQLite emulates this behavior in its interface in order to provide compliance with the Python Database API Specification. This is done using the .cursor() method, as follows:

>>> cx = sqlite.connect("pydb")
>>> cu = cx.cursor()

Note

You can actually create as many cursor objects as you like on the same database connection, although there is usually little point in doing so. Separate cursors do not isolate datachanges made in one cursor are immediately reflected in other cursors created from the same connection.

With a cursor object, you can use the .execute() method to execute an SQL command. Listing 9.1 shows a simple Python script that opens the pydb database and issues a CREATE TABLE statement to create the contacts table.

Listing 9.1. Executing a CREATE TABLE Statement Using the Python Interface
import sqlite

sql = """
CREATE TABLE contacts (
  id INTEGER PRIMARY KEY,
  first_name CHAR,
  last_name CHAR,
  email CHAR UNIQUE
)
"""

cx = sqlite.connect("pydb")
cu = cx.cursor()
cu.execute(sql)
cx.commit()
cx.close()

Run this script to create the table, and verify that it has been created using the sqlite tool.

$ python listing9.1.py
$ sqlite pydb
SQLite version 2.8.15
Enter ".help" for instructions
sqlite> .tables
contacts
sqlite> .schema
CREATE TABLE contacts (
  id INTEGER PRIMARY KEY,
  first_name CHAR,
  last_name CHAR,
  email CHAR UNIQUE
);

Error Handling

If for any reason the database file cannot be opened or the query cannot be executed, Python will raise an error. The following output was generated when an attempt was made to create a new SQLite database in a write-protected directory:

>>> cx = sqlite.connect("pydb")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib/python2.3/site-packages/sqlite/__init__.py", line 61,
  in connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.3/site-packages/sqlite/main.py", line 445,
  in __init__
    self.db = _sqlite.connect(database, mode)
_sqlite.DatabaseError: unable to open database: pydb

This is pretty ugly, so you'll probably want to trap any such errors and handle them in a nice way. The StandardError error class can be used to trap an error from PySQLite, but the sqlite package also includes its own Error class, with subclasses for more specific error detection.

Listing 9.2 performs the same CREATE TABLE statement as before, but uses the sqlite.Error exception so that the error is handled within the script.

Listing 9.2. Using sqlite.Error to Trap Errors
import sys;
import sqlite;

sql = """
CREATE TABLE contacts (
  id INTEGER PRIMARY KEY,
  first_name CHAR,
  last_name CHAR,
  email CHAR UNIQUE
)
"""

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

try:
        cu = cx.cursor()
        cu.execute(sql)
        cx.commit()
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

cx.close()

When this script is run from a location in which it does not have permission to open the database file, the output will look like the following:

$ ./listing9.2.py
Could not open the database file: unable to open database: pydb

If file permissions are okay but you run the script after the table has already been created, this will be the error produced:

$ ./listing9.2.py
Could not execute the query: table contacts already exists

PySQLite provides the subclasses of Error specified by the Database API specification in order to handle specific types of error message. The different error classes are shown in Table 9.1.

Table 9.1. Error Subclasses Implemented by PySQLite

InterfaceError

Error returned by the database interface.

DatabaseError

Error returned by the database engine when a database cannot be opened, or when SQLite returns SQLITE_ERROR, SQLITE_READONLY, SQLITE_CORRUPT, SQLITE_FULL, SQLITE_CANTOPEN, SQLITE_SCHEMA, or an unknown error code.

DataError

Indicates a problem with data values when SQLite returns SQLITE_TOOBIG, for example division by zero.

OperationalError

Error related to the operation of the database that is out of the developer's control, when SQLite returns SQLITE_PERM, SQLITE_ABORT, SQLITE_BUSY, SQLITE_LOCKED, SQLITE_INTERRUPT, SQLITE_IOERR, or SQLITE_PROTOCOL.

IntegrityError

Error due to a problem with the referential integrity of the database, when SQLite has returned SQLITE_CONSTRAINT or SQLITE_MISMATCH.

InternalError

An internal error in the database engine, raised when SQLite returns SQLITE_INTERNAL, SQLITE_NOTFOUND, or SQLITE_EMPTY.

ProgrammingError

Indicates a programming error, caused by an error in a user-defined function or an operation being attempted on a closed database.

NotSupportedError

Raised if a non-supported operation is attempted.

Connection Parameters

Several optional arguments can be supplied to sqlite.connect() after the name of the database file. The full prototype is

def connect (database, mode=0755, converters={}, autocommit=0,
             encoding=None, timeout=None, command_logfile=None)

The mode argument is currently not used but was intended to allow you to supply a file mode in which the database will be opened. Mode 0644 could be used, for instance, to open a database read-only.

The converters argument can be used to supply a mapping from SQL to Python data types. We will discuss the issues surrounding Python and SQLite data types in the section "Working with Data Types."

The autocommit feature is off by default but can be turned on for a connection object by setting this argument to 1. We will discuss this feature later in this chapter in the "Transactions" section.

Use the encoding argument to tell PySQLite what encoding method you want to use to save Unicode strings to the database. For example, pass the value utf-8 as the encoding argument to use UTF-8 encoding.

The timeout argument takes a value in seconds, which is the amount of time you want SQLite to continue trying to obtain a lock on the database file before giving up with a DatabaseError exception. The default value is None, meaning that no special action will be taken, and only one attempt will be made to open the database.

You can use the command_logfile argument to specify a file object that all the raw SQL statements passed to the SQLite engine through PySQLite will be logged to.

Using Commands That Change the Database

The script in Listing 9.3 prompts for three pieces of user input before generating an SQL statement to insert a new record into the contacts table.

Listing 9.3. Inserting a Database Record Using Data from User Input
import sys
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")

sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES ('%s', '%s', '%s')
""" % (fn, ln, em)

try:
        cu = cx.cursor()
        cu.execute(sql)
        cx.commit()
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

cx.close()

The three calls to raw_input() gather information from standard input that we'll use to build the SQL INSERT statement. The first name, last name, and email are stored to fn, ln, and em respectively.

fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")

We then use these values to build up a string, sql, which contains the command to be passed to SQLite.

sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES ('%s', '%s', '%s')
""" % (fn, ln, em)

The record is then inserted when we open a cursor on the SQLite connection and execute the SQL statement. The commit() method is called to ensure that the record is saved before closing the database connection.

Executing the script to add a record to the contacts table looks like this:

$ python listing9.3.py
Enter first name : Chris
Enter last name : Newman
Enter email address: chris@lightwood.net

We can then verify that the record has been inserted using the sqlite tool.

$ sqlite pydb
sqlite> SELECT * FROM contacts;
id    first_name  last_name   email
----  ----------  ----------  --------------------
1     Chris       Newman      chris@lightwood.net

Care must be taken to ensure that records read from user input are safe for insertion into SQLite. The following example shows how a name containing an apostrophe character can break the script in Listing 9.3:

Enter first name : Paddy
Enter last name : O'Brien
Enter email address: paddy@irish.com
Could not execute the query: unrecognized token: "@"

If you were to output the value of sql before it was executed, it would look like this:

INSERT INTO contacts (first_name, last_name, email)
VALUES ('Paddy', 'O'Brien', 'paddy@irish.com')

SQLite is unable to determine that the apostrophe in O'Brien is not the closing quote around the second data value. It considers only values within pairs of single quotes to be the data items.

In this case, the values to be inserted will be seen as 'Paddy', 'O' and ', ', and so the SQL statement is nonsense. It is the @ symbol that first causes a tokenizer error and the script to raise a DatabaseError exception.

Quotes are delimited in SQLite by another quote characteryou have to use two single quotes wherever you want one to appear. The preceding SQL statement would work if it were written as follows:

INSERT INTO contacts (first_name, last_name, email)
VALUES ('Paddy', 'O''Brien', 'paddy@irish.com')

However, to save having to validate user input and delimit apostrophes yourself, the execute() method can be used to substitute values into the query in a safe way. Rather than substituting values into sql, consider the following changes.

If we alter the SQL statement so that values are not enclosed in quotes at all, we can make PySQLite do the hard work of determining whether values require quoting and, if necessary, how special characters need delimiting.

sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES (%s, %s, %s)
"""

Then we execute the statement by passing a list of substitution values as the second argument to .execute().

cu.execute(sql, (fn, ln, em) )

The modified version can be seen in full in Listing 9.4.

Listing 9.4. Using .execute() to Substitute Values into a Query
import sys
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

fn = raw_input("Enter first name : ")
ln = raw_input("Enter last name : ")
em = raw_input("Enter email address: ")

sql = """
INSERT INTO contacts (first_name, last_name, email)
VALUES (%s, %s, %s)
"""
try:
        cu = cx.cursor()
        cu.execute(sql, (fn, ln, em) )
        cx.commit()
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

cx.close()

We can now insert records using strings that contain apostrophes into SQLite without having to worry about delimiting quotes first.

$ python listing9.4.py
Enter first name : Paddy
Enter last name : O'Brien
Enter email address: paddy@irish.com
$ sqlite pydb
sqlite> SELECT * FROM contacts;
id    first_name  last_name   email
----  ----------  ----------  --------------------
1     Chris       Newman      chris@lightwood.net
2     Paddy       O'Brien     paddy@irish.com

When a database operation is performed that affects rows in the databasenamely UPDATE and DELETE operationsyou can find the number of rows affected by the command by using the .rowcount attribute of the cursor object.

In Listing 9.5 we perform an UPDATE command that changes the case of the first_name field in contacts to uppercase across all database rows. To show how many rows were actually affected, the .rowcount attribute is displayed.

Listing 9.5. Finding the Number of Rows Affected by an SQL Statement
import sys
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

sql = "UPDATE contacts SET first_name = upper(first_name)"
try:
        cu = cx.cursor()
        cu.execute(sql)
        cx.commit()
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

print str(cu.rowcount) + " row(s) were affected"

cx.close()

When you execute this script, every row in the database is updated and the number of rows is printed to screen.

$ python listing9.5.py
2 row(s) were affected

Note

The number of affected rows returned by .rowcount is the total number of rows that the UPDATE or DELETE statement considersin other words, the number of rows matching the WHERE clause, if one is given. If a row is updated to the same value it held before the statement was executed, it is still counted by .rowcount even though no change to the data has been made.

After a SELECT statement, .rowcount also contains the number of rows returned by the query. In fact it returns the number of rows returned or affected by the WHERE clause in any SQL statement.

Transactions

PySQLite contains some built-in optimization to make the best use possible of SQLite's transactions. When a number of INSERT, UPDATE, or DELETE statements are issued in succession in a script, they are all executed within a single transaction.

The default behavior, if the SQL commands were entered through sqlite or via the underlying C/C++ interface, would be to open an implicit transaction for each statement and commit each transaction before moving on to the next command. There is significant performance benefit to be gained from grouping multiple statements into a single transaction, and so PySQLite attempts to handle the grouping automatically.

Whenever the .execute() method is passed an SQL statement that changes the database, PySQLite automatically sends a BEGIN TRANSACTION before that command is executed if a transaction has not already been started. A .commit() therefore needs to be issued before the changes will be saved to the database.

Note

If you look back to Listing 9.1, you'll see that we included a .commit() instruction before closing the database. Note that calling .close() while a transaction is open will roll back any unsaved changes.

PySQLite opens a transaction whenever necessary but is optimized to leave it as late as possible before the BEGIN TRANSACTION command is sent so that the database file is locked for as little time as possible. It is therefore the type of statement processed by .execute() that determines whether a transaction is started.

SELECT statements do not need to be run inside a transaction but if the result of a query is subsequently used in another statement that does change the database, you should make sure that both SQL commands are executed within the same transaction.

This is the default behavior of PySQLite, although it can be changed if desired by turning on autocommit mode by using a value of 1 for the autocommit argument in sqlite.connect().

cx = sqlite.connect("pydb", 755, {}, 1)

Because autocommit is the fourth argument, the mode and converters arguments must be supplied. In this example we have simply given them their default values.

An alternative syntax that is more compact is given in the following example. PySQLite recognizes that the second parameter is not a file mode and applies the autocommit setting:

cx = sqlite.connect("pydb", autocommit=1);

When autocommit is turned on, PySQLite will not attempt to do anything clever with transactions. An INSERT, UPDATE, or DELETE command issued outside of a transaction will be executed within its own implicit transaction. If you want to start a transaction manually, you must pass BEGIN TRANSACTION to the .execute() method.

Fetching Records from the Database

Let's take a look at how the result of a SELECT query is processed using PySQLite. The query is submitted through .execute() just as other types of SQL statement are executed.

To fetch the data retrieved by the SELECT, call .fetchone() on the cursor object. Data is returned one row at a time each time .fetchone() is called, and returns None if there are no more rows to be fetched.

Listing 9.6 shows a simple example that selects first_name and last_name from the contacts table. Each data record is fetched into row, which is then used to output the two column values to screen.

Listing 9.6. Fetching Selected Data Using .fetchone()
import sys
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

sql = "SELECT first_name, last_name FROM contacts"

try:
        cu = cx.cursor()
        cu.execute(sql)
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

row = cu.fetchone()
while row:
        print row.first_name + " " + row.last_name
        row = cu.fetchone()

cx.close()

The while loop simply prints the contents of row and, assuming there are more records to come, fetches the next row and repeats. The names of the selected columns are also the names of the elements of rowwe reference row.first_name and row.last_name respectively.

To comply with the Python Database API Specification, fetched columns can also be referenced using the index number of their position in the list, starting at zero. Take this example:

print row.first_name + " " + row.last_name

This could also be written as

print row[0] + " " + row[1]

The following is an example of output from running Listing 9.6:

$ python listing9.6.py
CHRIS Newman
PADDY O'Brien
Bill Williams

Complimentary to .fetchone(), PySQLite also implements the .fetchmany() and .fetchall() methods to retrieve the multiple rows or even the entire data set returned by the SELECT all at once.

The .fetchmany() method takes a single argument to specify the number of rows to fetch. If the value is smaller than the number of rows remaining, the cursor position is advanced so that the next call to .fetchone() or .fetchmany() will begin at the next row in sequence.

The .fetchall() method, as you have probably guessed, is equivalent to calling .fetchmany() with the argument equal to the total number of rows returned by the query. If .fetchone() or .fetchmany() has been called on that cursor object previously, only the remaining rows will be fetched.

Listing 9.7 performs the same query on the contacts table using .fetchmany() to grab the result two records at a time.

Listing 9.7. Fetching Several Rows at a Time Using .fetchmany()
import sys
import sqlite

try:
        cx = sqlite.connect("pydb", 755)
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

sql = "SELECT first_name, last_name FROM contacts"

try:
        cu = cx.cursor()
        cu.execute(sql)
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

print "Query returns " + str(cu.rowcount) + " row(s)"

rows = cu.fetchmany(2)
while rows:
        for row in rows:
                print row.first_name + " " + row.last_name
        rows = cu.fetchmany(2)

cx.close()

This time, we use two nested loops in order to display all the records. The outer while loop fetches rows two at a time, and the inner for loop iterates each of the fetched rows into a single row.

The .rowcount attribute for SELECT statements contains the total number of rows returned by the query. We output this value in Listing 9.7 to show the number of records that will be fetched.

$ python listing9.7.py
Query returns 3 row(s)
CHRIS Newman
PADDY O'Brien
Bill Williams

If no value is given, the number of rows is taken from the .arraysize attribute, which will contain the optimum number of rows to fetch at a time.

For example, the following alternative could have been used in Listing 9.7:

cu.arraysize = 2
rows = cu.fetchmany()

Working with Data Types

As we saw in Chapter 2, "Working with Data," SQLite is typeless. However, Python uses strong data typing and although most of the time it is able to infer what type it needs to convert a string from SQLite to, sometimes you need to give a mapping to ensure that a data value is converted to the correct type in Python.

Although the column types given to SQLite in the CREATE TABLE statement are mostly ignoredother than to determine whether numeric or text-based sorting is performed on that columnPySQLite will look at the column type to try to determine how a column should be converted to a Python data type.

Table 9.2 shows the default mappings, which should be kept in mind when creating the schema for a database that is to be interfaced from PySQLite.

Table 9.2. Default Mappings of SQLite Columns to Python Data Types

Column Type

Converter Function

CHAR, TEXT

str()

INT

int()

FLOAT, NUMERIC, NUMBER, DECIMAL, REAL, and DOUBLE

float()

UNICODE

UnicodeConvertor(self.encoding)

BINARY, BLOB

sqlite.decode()

DATE

DateTime.DateFrom()

TIME

DateTime.TimeFrom()

TIMESTAMP

DateTime.DateTimeFrom()

INTERVAL

DateTime.DateTimeDeltaFrom()

Note

The column types shown in Table 9.2 can be substrings of the actual type specified in the table schema. For example, an INTEGER column will also be converted using int(). Column types are not case-sensitive. The closest match found is used and sometimes you must be careful; for example, INT is closer to INTEGER than INTERVAL. To be sure of using the correct column type, always give the name in full.

To override the default mapping converter functions, the converter name can be passed to .execute() before a query is executed, in the following format:

cu.execute("-- types type1, type2, ...")

In this example, col1, col2, and so on are the converter functions to use for each column in turn. For example, to tell SQLite to convert the first two columns returned from the next query executed as string and integer respectively, you would use the following command:

cu.execute("-- types str, int")

The following built-in types can be passed in a -- types command:

  • str

  • int

  • long

  • float

  • unicode

  • binary

  • date

  • time

  • timestamp

  • interval

Adding New SQL Functions

Though not part of the Python Database API Specification, PySQLite provides an interface to SQLite's powerful user-defined function capabilities for both simple and aggregating functions.

This feature allows you to create your own functions in Python that can be registered as SQLite functions and therefore called within an SQL statement. There is therefore no need to know C in order to extend the SQL language when using PySQLite.

User-defined functions are registered through the .create_function() method on a database connection object. Functions are available in SQL only for the duration of that connectionthey are not saved to the database itself or made available to other connection objects within the same script.

In Listing 9.8 we create a function, altcaps(), in Python and then register it as an SQL function. The function takes a single string argument and returns it with alternating capitalization. This function is fairly useless but should serve as an example of how user-defined functions are implemented via PySQLite.

Listing 9.8. Creating a User-Defined SQL Function
import sys
import string
import sqlite

def altcaps (word):
        altstr = ""
        for i in range(len(word)):
                if i%2:
                        altstr += string.lower(word[i])
                else:
                        altstr += string.upper(word[i])
        return altstr

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

cx.create_function("altcaps", 1, altcaps)

sql = "SELECT altcaps(first_name) FROM contacts"

try:
        cu = cx.cursor()
        cu.execute('-- types str')
        cu.execute(sql)
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

row = cu.fetchone()
while row:
        print row[0]
        row = cu.fetchone()

cx.close()

The altcaps() function uses range() to create a loop that will run once for each character in word.

for i in range(len(word)):

We then use the mod operator (%) to test whether the loop index is odd or even. A new string altstr is built up a character at a time, with every odd-numbered character added in uppercase and every even position in lowercase.

if i%2:
       altstr += string.lower(word[i])
else:
       altstr += string.upper(word[i])

This function is then registered in SQLite:

cx.create_function("altcaps", 1, altcaps)

There are three arguments to .create_function()the name of the SQL function, the number of arguments, and a reference to the Python function that is to be executed when the SQL function is called.

It is important to tell PySQLite what data type to expect to be returned from a user-defined function; otherwise, float will be assumed. We set a string type for this function with the following line :

cu.execute('-- types str')

Note

PySQLite assumes that user-defined functions return a NUMERIC unless told otherwise. Therefore the default converter function is float().

Executing Listing 9.8 produces the following output, consisting of the first_name values from contacts with alternating upper- and lowercase letters:

$ python listing9.8.py
ChRiS
PaDdY
BiLl

In this example we gave the SQL function the same name as the Python function, although this is not a requirement. In Listing 9.9 we reference Python's random() function from the standard library and create an SQL function called pyrandom() so as not to conflict with SQLite's built-in random() function.

Note

If you use .create_function() to create an SQL function with the same name as an existing function, the user-defined function will simply overload the existing function. No error or warning is raised.

Listing 9.9. Registering a Python Library Function in SQL
import sys
import random
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

cx.create_function("pyrandom", 0, random.random)

sql = "SELECT pyrandom()";

try:
        cu = cx.cursor()
        cu.execute(sql)
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

row = cu.fetchone()
print row[0]

cx.close()

Adding Aggregating Functions

PySQLite also allows you to add custom aggregating functions to SQL. An aggregating function is one that applies a calculation across many rows at a timeeither the entire dataset returned by a query or rows grouped together by a GROUP BY clause.

Aggregating functions require a class to be defined containing four functions:

  • A step function to be executed for each row, named step()

  • A function to be executed after each row has been processed, named finalize()

  • A function to reset data stored in the class, named reset() when a new aggregate calculation begins

We will create a class definition that can be used to register an aggregating function to compute the median of a set of numbers. First, we need some data to try the function on.

Listing 9.10 creates a simple table, numbers, and loops from 1 to 9 inserting the square of each value. Because the sequence of square numbers does not increase proportionally, the mean and median averages will be different.

Listing 9.10. Creating a Table Containing a Sequence of Square Numbers
import sys
import sqlite

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

cu = cx.cursor()

sql = "CREATE TABLE numbers (num INTEGER)"
cu.execute(sql)

for i in range(1,10):
        sql = "INSERT INTO numbers VALUES (%d)" % (i*i)
        cu.execute(sql)

cx.commit()

cx.close()

The sequence inserted looks like this:

1 4 9 16 25 36 49 64 81

The median is a value such that half the values in the set are above the median and half are below it. From looking at the sequence we can see that the median will be 25.

Listing 9.11 contains the class definition for our median() aggregating function and executes an SQL statement to find both the median and the meanusing the SQLite built-in AVG() functionon the sequence of square numbers.

Listing 9.11. Creating a Table Containing the Mean and Median Averages
import sys
import sqlite

class median:
        def __init__(self):
                self.reset()

        def reset(self):
                self.values = []
        def step(self, val):
                self.values.append(float(val))

        def finalize(self):
                self.values.sort()
                n = len(self.values)
                if n%2 == 1:
                        return self.values[n//2]
                else:
                        return (self.values[n//2-1] + self.values[n//2])/2

try:
        cx = sqlite.connect("pydb")
except sqlite.Error, errmsg:
        print "Could not open the database file: " + str(errmsg)
        sys.exit()

cx.create_aggregate("median", 1, median)

sql = "SELECT median(num), avg(num) FROM numbers";

try:
        cu = cx.cursor()
        cu.execute(sql)
except sqlite.Error, errmsg:
        print "Could not execute the query: " + str(errmsg)
        sys.exit()

row = cu.fetchone()
print "Median is " + str(row[0])
print "Mean is " + str(row[1])

cx.close()

The class contains the following member functions:

def __init__(self):
        self.reset()

The constructor calls reset() to initialize the local array in which the values from which the median is calculated will be stored.

def reset(self):
        self.values = []

The step() function simply appends each value to the local array. Nothing more is required here; the hard work is done in finalize().

def step(self, val):
        self.values.append(float(val))

First of all we sort the array into numerical order.

def finalize(self):
        self.values.sort()

The rule to find the median is slightly different depending on whether there is an odd or even number of elements.

If there are an odd number of elements, the median is simply the middle value from the list. The index of the midpoint of the array is half the length of the array, rounded down to the nearest whole number.

n = len(self.values)
if n%2 == 1:
        return self.values[n//2]

If there is an even number of elements, the median is the midpoint between the two middle values.

else:
        return (self.values[n//2-1] + self.values[n//2])/2

The aggregating function is registered in SQLite using .create_aggregate().

cx.create_aggregate("median", 1, median)

The syntax is similar to .create_function() except that the argument is the name of the class that contains the reset(), step(), and finalize() methods required to implement that aggregate.

Listing 9.11 executes the following query to find both the median and mean averages from the test data:

SELECT median(num), avg(num) FROM numbers

The following output is produced to show that the median and mean are indeed different values:

$ python listing9.11.py
Median is 25.0
Mean is 31.6666666667


Part III: SQLite Administration

10 General Database Administration


Chapter 10. General Database Administration

This chapter deals with SQLite administration topics. We will look at how the database parameters can be adjusted with the PRAGMA command and discuss ways to back up and restore your database. We will also examine some of the database internals that will help you to gain a better understanding of the SQLite engine.


The PRAGMA Command

The PRAGMA command provides an interface to modify the operation of the SQLite library and perform low-level operations to retrieve information about the connected database.

Fetching Database Information

This section describes the set of PRAGMA directives that allows you to find information about the currently attached databases.

PRAGMA database_list;

One row is returned for each open database containing the path to the database file and the name that the database was attached with. The first two rows returned will be the main database and the location in which temporary tables are stored.

A PRAGMA command can be executed from the sqlite program as well as through any of the language APIsafter all, the sqlite program is just a simple front end that passes typed commands to the C/C++ interface and uses a choice of callback function, selected using .mode, to output the rows to screen.

The following example shows the result of executing PRAGMA database_list through the sqlite program after a second database has been attached to the session:

sqlite> ATTACH DATABASE db2 AS db2;
sqlite> PRAGMA database_list;
seq   name  file
0     main  /home/chris/sqlite/admin/db1
1     temp  /var/tmp/sqlite_6x5rZ2drAEtVpzj
2     db2   /home/chris/sqlite/admin/db2

If executed from a language interface, the pseudo-table returned by this pragma contains columns named seq, name, and file.

To find information about a database table, use the table_info pragma with the following syntax:

PRAGMA table_info(table-name);

The table-name argument is required and can refer only to a table in the main database, not one attached using ATTACH DATABASE. One row is returned for each column in that table, containing the columns shown in Table 10.1.

Table 10.1. Columns in the Pseudo-Table Returned by PRAGMA table_info

cid

An integer column ID, beginning at zero, that shows the order in which columns appear in the table.

name

The name of the column. The capitalization used in the CREATE TABLE statement is retained.

type

The data type of the column, taken verbatim from the CREATE TABLE statement.

notnull

If the column was declared as NOT NULL, this column will contain a non-zero value; otherwise, it will be zero.

dflt_value

Contains the DEFAULT value for the column if one has been specified, otherwise NULL.

pk

Will be 1 for the column(s) making up the PRIMARY KEY on the table and 0 for all other columns.

To find information about the indexes on a table, use the index_list and index_info pragmas as follows:

PRAGMA index_list(table-name);
PRAGMA index_info(index-name);

PRAGMA index_list causes one row to be returned for each index on the given table-name with the columns shown in Table 10.2.

Table 10.2. Columns in the Pseudo-Table Returned by PRAGMA index_list

seq

An integer beginning at zero that indicates the order in which the indexes were added to the table.

name

The name of the index. This is the identifier given in a CREATE INDEX statement, or the assigned name if the index was created with a PRIMARY KEY or UNIQUE constraint in the CREATE TABLE statement.

unique

Will be 1 if the index was declared as UNIQUE; otherwise, it will be 0.

When you know the name of an index, PRAGMA index_info can be used to find out which column or columns make up that index. A single column index will return a single row, whereas a clustered index will return one row for each indexed column, in the relevant order. The columns returned are shown in Table 10.3.

Table 10.3. Columns in the Pseudo-Table Returned by PRAGMA index_info

seqno

An integer beginning at zero that indicates the sequence of the columns in the index

cid

The cid number of the indexed column, from table_info

name

The name of the indexed column

When an index is created implicitly by the use of a UNIQUE or PRIMARY KEY constraint in the CREATE TABLE statement, the index will take an automatically assigned name of the form (table-name autoindex num). The following example shows these pragmas being run for a table with both explicit and implicit index names:

sqlite> CREATE TABLE mytable (
   ...>   col1 INTEGER NOT NULL,
   ...>   col2 CHAR NOT NULL,
   ...>   col3 INTEGER UNIQUE,
   ...>   col4 CHAR,
   ...>   PRIMARY KEY (col1, col2)
   ...> );

sqlite> CREATE INDEX col4_idx ON mytable(col4);

sqlite> PRAGMA index_list(mytable);
seq   name                            unique
----  ------------------------------  ------
0     col4_idx                        0
1     (mytable autoindex 2)           1
2     (mytable autoindex 1)           1

sqlite> PRAGMA index_info('(mytable autoindex 2)');
seqno  cid   name
-----  ----  ----------
0      0     col1
1      1     col2

Note

If the table-name or index-name argument is not found in the current database, no error is returned. The result of the PRAGMA is simply silent.

Altering Database Parameters

This section describes the set of PRAGMA directives that enables you to adjust certain database parameters for the purposes of performance tuning at the overall database level. The first parameter that can be adjusted is cache_size.

PRAGMA cache_size;

This instruction will return the maximum number of database disk pages that will be held in memory at once. The default value of 2000 pages equates to around 3000KB of memoryeach page taking up approximately 1.5KB.

The same PRAGMA can be used to set a new cache size by assigning a value as follows:

PRAGMA cache_size = num-pages;

Increasing the cache size can improve performance if the SQL operations you are executing would use a larger amount of memory than the size of the cache. For instance, an UPDATE operation on a large number of rows would benefit from a larger cache size. Keeping a limit on the cache size ensures that SQLite's memory usage remains under your control.

The cache_size can be altered dynamically, so if one query in your application in particular can benefit from a larger number of pages being cached, you can increase the size for that query and drop it back down afterwards.

The temp_store pragma allows the location used for the temporary database to be queried or changed.

PRAGMA temp_store;
PRAGMA temp_store = value;

The value of temp_store can be one of the constants DEFAULT, MEMORY, or FILE, which have the values 0, 1, and 2 respectively. The values indicate whether SQLite should store temporary database objects in memory or to disk. The DEFAULT value is set at compile time and unless you have changed it, it will be FILE.

Using in-memory databases for temporary tables can produce significant performance savings if your application can afford the additional memory usage.

The synchronous pragma allows the synchronous flag for the current database session to be queried or changed.

PRAGMA synchronous;
PRAGMA synchronous = value;

The value can be OFF, NORMAL, or FULL, which have the values 0, 1, and 2 respectively.

In synchronous mode FULL or NORMAL, SQLite will pause periodically to make sure data has been written to the disk before carrying on with the next operation. FULL mode is very safe and ensures that after a system crash for any reason, even unrelated to SQLite, the database will not be corrupt.

Because FULL synchronous is very slow, NORMAL mode is used more often. SQLite still pauses to check at critical moments, but there is a small chance that an operating-system crash or power failure at the wrong time could cause corruption of the database file.

When synchronous is OFF, SQLite does not pause to check that data has been written to disk. This mode is very fastsome operations have been clocked at 50 times faster than NORMAL modehowever, the chance of data loss is greater because it relies on the operating system to ensure that data has been written. Setting synchronous=OFF can give some major performance benefits, but you should give careful consideration to the potential consequences.

A good time to turn synchronous OFF is when populating a large database for the first time. In this situation, the risk of corruption is not critical because the population process could be restarted from a known point in the event of a failure.

Note

NORMAL mode is used by default, even though there is a small chance of data loss, for performance reasons. In reality, the kind of system crash that would cause your database to become corrupt is likely to carry with it other, more serious problems such as hardware failure, so NORMAL mode is considered acceptable for everyday use.

Altering Database Parameters Permanently

Changes made in the way shown in the preceding section will persist only until the current database session is closed. To change a database parameter permanently, the same PRAGMA directives can be called with a default_ prefix to query and set the value that is stored in the database file.

To query the current default cache size of the attached database, use the following instruction:

PRAGMA default_cache_size;

To increase the cache size for that database to 4000 pages permanently, the command would be

PRAGMA default_cache_size = 4000;

The same prefix can be applied to give default_temp_store and default_synchronous pragmas to query or set the permanent values of temp_store and synchronous respectively.

Altering Query Parameters

Though it can now be done using the sqlite_changes() function call, the traditional method of counting changes made to the database as a result of an INSERT, UPDATE, or DELETE operation was to use the count_changes PRAGMA. This PRAGMA is likely to be removed from future versions of SQLite in favor of sqlite_changes(), so it is included here only for historical value.

PRAGMA count_changes = ON;
PRAGMA count_changes = OFF;

When count_changes is set to ON, a single row is returned whenever an operation is performed that modifies the database containing the number of affected rows. The following example shows this in action using the sqlite program:

sqlite> PRAGMA count_changes = ON;
sqlite> UPDATE contacts SET first_name = upper(first_name);
4
sqlite> DELETE FROM contacts WHERE first_name = 'CHRIS';
1

Unless you are working with legacy code that uses count_changes, there is a small performance gain to be had by leaving this pragma turned off.

Note

The examples in this chapter use ON or OFF as the Boolean argument where one is required. ON, trUE, and YES are equivalent to a value of 1, whereas OFF, FALSE, and NO are equivalent to zero.

The column names passed to the callback function in the C/C++ interface as columnNamesalso used in other language APIs to identify the name of the column returnedare usually just the names of the columns, with a table prefix used only where a join has taken place. Use the full_column_names pragma to force SQLite to report a column name as table.column, even when only one table has been queried.

PRAGMA full_column_names = ON;

The following output shows an example of this in the sqlite monitor programthe column headings now include the table name:

sqlite> PRAGMA full_column_names = ON;
sqlite> SELECT first_name, last_name FROM contacts;
contacts.first_name  contacts.last_name
-------------------  ------------------
CHRIS                NEWMAN
PADDY                O'BRIEN
TOM                  THOMAS
BILL                 WILLIAMS
JO                   JONES

The columnNames parameter to the callback function will always contain the names of the columns selected in elements numbered from zero to argc1, but this array can be extended to also include the data types specified in the CREATE TABLE statement.

PRAGMA show_datatypes = ON;

Where no data type was specified in the CREATE TABLE statement, it is reported as NUMERIC. Expressions evaluated in a query are reported as either NUMERIC or TEXT depending on the expression itself.

The example in Listing 10.1 shows how the show_datatypes pragma can be used to view the data types of both database columns and evaluated expressions.

Listing 10.1. Using PRAGMA show_datatypes to Find the Data Types of Columns and Expressions
#include <stdio.h>
#include <sqlite.h>

int callback(void *pArg, int argc, char **argv, char **columnNames)
{
  int i;
  for (i=0; i<argc; i++) {
    printf("%-20s %-8s %s\n",
                columnNames[i], columnNames[i+argc], argv[i]);
  }
  return(0);
}

int main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("db1", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    return(1);
  }

  ret = sqlite_exec(db,
          "SELECT id, first_name, id+2, upper(first_name) "
          "FROM contacts LIMIT 1", callback, NULL, &errmsg);

  if (ret != SQLITE_OK)
  {
    fprintf(stderr, "SQL error: %s\n", errmsg);
  }

  sqlite_close(db);
  return(0);
}

The resulting output shows that the column types of the database columns are as they were declared in the CREATE TABLE statement, but the expressions revert to one of the internal data types, NUMERIC or TEXT.

$ ./listing10.1
id                   INTEGER  1
first_name           CHAR     Chris
id+2                 NUMERIC  3
upper(first_name)    TEXT     CHRIS

Analyzing the Database

An integrity check can be performed on the entire database with the integrity_check pragma. It is simply issued as follows:

PRAGMA integrity_check;

The check searches for corrupt indexes, malformed records, and other such problems. Any issues are returned as a single string or if there are no problems, a single string ok is returned. The response you would hope to see is

sqlite> PRAGMA integrity_check;
ok

The parser_trace pragma turns ON or OFF internal tracing of the SQLite parser stack and, when ON, returns one row for each step of the parser trace. To use this feature, SQLite must have been compiled without the NDEBUG compile time option.

The following example shows the first few lines of output from the parser trace for a simple SELECT statement. As you can see, it is not desirable to have this output turned on unless you are looking for a particular problem with the parser codethere were 108 lines of output in total, just for this simple query!

sqlite> PRAGMA parser_trace=ON;
sqlite> SELECT first_name, last_name FROM contacts;
parser: Input SELECT
parser: Reduce [explain ::=].
parser: Shift 3
parser: Stack: explain
parser: Shift 73
parser: Stack: explain SELECT
parser: Input ID
parser: Reduce [distinct ::=].
parser: Shift 74
parser: Stack: explain SELECT distinct
parser: Reduce [sclp ::=].
parser: Shift 289
parser: Stack: explain SELECT distinct sclp
parser: Shift 64
parser: Stack: explain SELECT distinct sclp ID
parser: Input COMMA
parser: Reduce [expr ::= ID].
parser: Shift 290
[...]

Similarly, tracing can be enabled at the Virtual Database Engine level using the vdbe_trace pragma. One row is returned for each VDBE opcode in the trace, which gives a result very similar to that obtained by the EXPLAIN command.

sqlite> PRAGMA vdbe_trace=ON;
sqlite> DELETE FROM contacts;
   0 Transaction     0    0
   1 VerifyCookie    0    3
   2 Transaction     1    0
   3 Clear           3    0
   4 SetCounts       0    0
   5 Commit          0    0
   6 Halt            0    0

We will examine the VDBE opcodes in more detail later in this chapter.


Backing Up and Restoring Data

It is, of course, essential to back up your data regularly. SQLite stores databases to the filesystem, so backing up your databases can be as easy as taking a copy of the database files themselves and putting them somewhere safe.

However, if you are copying the database file, can you be sure that it is not being written to at the time you issue the copy command? If there's even a small chance that a SQLite write operation could happen during the copy, you should not use this method to back up your database to ensure that data corruption cannot occur.

SQLite implements locking at the database level using a lock on the database file itself, which works to the extent that other processes using the SQLite library know when the database is being written to. Many processes can read from a database at the same time; however, a single writer process locks the entire database, preventing any other read or write operation taking place.

On Windows platforms, if SQLite has locked the file the operating system will acknowledge the lock and will not allow you to copy the file until SQLite is done with it. Likewise the copy operation will lock the database file so that SQLite cannot access it. However, on Unix systems, file locks are advisoryin other words they tell you that the file is locked if you care to check, but do not otherwise prevent access. Therefore it is entirely possible to initiate a cp command while SQLite has a database file locked for writing.

The .dump Command

The .dump command in sqlite provides a simple way to create a backup file in the form of a list of SQL statements that can be used to re-create the database from scratch. The CREATE TABLE statements are preserved and one INSERT command is written for each row of data in the table.

Note

Because the .dump command is part of an application that uses the SQLite library, there are no issues with file locking to worry about. If the database is locked at the moment you attempt to fetch the rows from the database, the busy handler will be invoked, which, in sqlite, is to wait for the specified timeout before returning an error message.

A backup operation can therefore be performed as follows:

$ sqlite dbname .dump > backup.sql

Future restoration from the generated backup.sql is simple:

$ sqlite dbname < backup.sql

You can optionally pass a table name to .dump to extract only the schema and records for that table. The following example shows the dump file produced for the contacts table:

$ echo '.dump contacts'| sqlite db1
BEGIN TRANSACTION;
CREATE TABLE contacts (
  id INTEGER PRIMARY KEY,
  first_name CHAR,
  last_name  CHAR,
  email      CHAR);
INSERT INTO contacts VALUES(1,'Chris','Newman','chris@lightwood.net');
INSERT INTO contacts VALUES(2,'Paddy','O''Brien','paddy@irish.com');
INSERT INTO contacts VALUES(3,'Tom','Thomas','tom@tom.com');
INSERT INTO contacts VALUES(4,'Bill','Williams','bill@bill.com');
INSERT INTO contacts VALUES(5,'Jo','Jones','jo@jojones.com');
COMMIT;

Note that the INSERT statements generated do not contain a column list before the VALUES keyword. If you are using the .dump output to restore a table exactly as it used to be, this is no problem; however, you cannot alter the table schema in the CREATE TABLE statement without also modifying every INSERT in the extract file.

The .dump command can act as a handy workaround for the lack of ALTER TABLE command in SQLite if you are able to adjust the INSERT commands accordingly. The following example uses sed to change the SQL commands to full inserts so that you can go ahead and add columns to the CREATE TABLE statement to re-create the database from this file with the new schema.

$ echo .dump | sqlite dbname | \
     sed e 's/VALUES/(id,first_name,last_name,email) VALUES/g' \
     > backup.sql

Whether you decide to use .dump for regular backups or to simply copy the database file itself will depend on several factors. The output from .dump is often smaller than the SQLite database file and will certainly compress much smaller using gzip or some similar utility.

However, restoring from an SQL file can take some time, particularly if there are a lot of indexes to be rebuilt after the data has been loaded. As a backup should be as quick and easy to restore as possible, copying the SQLite database file may be the correct choice.

Using .dump is a better choice for a long-term archive of old data. Although there is, in theory, no reason that the same version of SQLite will not be available several years from now, you might actually want to load your archived data into a newer version of SQLite or even some other database system. The SQL output file produced by .dump will be fairly portable.


Exploring the SQLite Virtual Database Engine

In this section we'll take a look under the hood of SQLite to see how the command process works and to examine the internals of the Virtual Database Engine (VDBE). Though it is not essential to know what goes on behind the scenes, a little insight into how the SQLite engine processes your queries can help you understand SQLite better, and if you want to gain a deeper knowledge of how SQLite is implemented, you will learn the roles of the different source code files.

SQLite Architecture

Let's begin by looking at the steps SQLite goes through in order to process a query. Figure 10.1 shows a diagram of the flow of information between submitting a query for processing and the output being returned.

Figure 10.1. Architecture of the SQLite Database Engine

Interface

The interface is the C library through which all SQLite processing is programmed. Even if you are using a different language API, at a lower level it is implemented using the C library. The core interface is defined in the main.c file of the SQLite source distribution. A few other files contain ancillary functions; for instance, printf.c contains sqlite_mprintf() and its derivatives.

After a command is received through the interface, it is passed into the SQL Command Processor, which consists of three separate steps.

Tokenizer

The tokenizer handles the process of breaking up a character string containing SQL statements into tokens that can be understood by the parser. Each individual component of the SQL language is a tokeneach combination of characters that forms a keyword or identifier and each symbol or sequence of symbols that forms an operator.

Parser

SQLite uses the Lemon LALR(1) parser generator, which is probably not already on your system, so the SQLite source code is distributed with the entire Lemon source code in a single file as tool/lemon.c. Though it does much the same job as BISON or YACC, Lemon was chosen for its ability to generate a parser that is thread-safe and less prone to memory leaks.

The source file parse.y contains the grammar definition for SQLite's implementation of SQL. Lemon reads this file and generates a corresponding C-language file for the actual parser.

There is a document giving an introduction to the Lemon parser at http://www.hwaci.com/sw/lemon/lemon.html.

Code Generator

The parser assembles the tokenized SQL components into statements, checking that its syntax is valid before calling the Code Generator to produce virtual machine code that will do the actual work of executing the SQL statement. We'll look at the way the virtual machine works shortly.

Many SQL statements have their own source code file to handle the conversion to virtual machine code. For instance, select.c, insert.c, update.c, and delete.c deal with SELECT, INSERT, UPDATE, and DELETE statements respectively. The WHERE clause is processed by where.c, and code to evaluate expressions is generated by expr.c.

Virtual Machine

The virtual machine implements an abstract computing engine for the specific purpose of manipulating database files. The virtual machine itself can be found in the source code files vdbe.c, and various utilities used to compile SQL statements to VDBE opcodes are contained in vdbeaux.c.

Backend

SQLite uses an abstraction layer between the virtual machine and the low-level storage and retrieval routines, implemented as a B-tree, page cache, and OS interface.

B-Tree

The database is stored to disk using a B-tree layera balanced tree structure that provides fast data access by minimizing the number of disk reads required to access a particular piece of information. The implementation and interface to SQLite's B-tree subsystem can be found in btree.c and btree.h.

Each table and index in the database is written using a separate B-tree though, as we have already seen, all the B-trees are stored in a single file on disk.

Pager

The B-tree storage mechanism requests data from the disk in 1KB chunks. The pager layer implements a page caching system that handles the reading, writing, and caching of these chunks, including the rollback and commit operations needed to implement atomic transactions.

The pager implementation and its interface can be found in pager.c and pager.h respectively.

OS Interface

For portability between different platforms, SQLite has implemented an abstraction layer that interfaces with the underlying operating system.

The file os.c contains routines that deal with low-level filesystem access on Windows and Unix. The definitions are contained in preprocessor ifdef instructions to ensure that the relevant interface is built into SQLite at compile time.

Red/Black Tree

SQLite stores in-memory databases using a red/black balanced binary tree structure, defined in btree_rb.c. No pager or OS interface is required for this part of the backend as the data is stored to RAM only.

Virtual Machine Opcodes

The VDBE's machine language contains more than 100 opcodes, defined in vdbe.c, that are used to handle every possible database operation. We will look at how a few common operations are translated into opcodes and use the vdbe_trace pragma to walk through the command execution process.

Each instruction in the virtual machine is made up of an opcode and between zero and three operands. The first operand, named P1, must be an integer. P2 is also an integer but must be non-negative. P2 will always contain the jump destination for an operation that may cause a jump. P3 can be either a NULL-terminated string, or simply NULL.

To see SQL commands converted to VDBE opcodes as they are executed from the sqlite program, begin by turning on the vdbe_trace pragma. To use this feature, SQLite must have been compiled without the NDEBUG compile time option.

sqlite> PRAGMA vdbe_trace=ON;
   0 Halt            0    0

Immediately we see some VDBE trace output, although not a lot has happened for it to trace just yet. The zero at the start of the line is the instruction numberas we'll see shortly, lines from the trace are numbered to indicate the sequence of operationfollowed by the Halt opcode with operands P1 and P2 both zero.

The Halt opcode instructs the VDBE to exit immediately, tidying up any incomplete operations. We will always see the Halt instruction at the end of every command trace.

P1 will contain the return code, which is usually SQLITE_OK, which is defined with a value of zero. The full list of return codes can be found in Appendix E, "C/C++ Interface Reference."

When P1 is non-zero, the return code indicates an error has occurred and the value of P2 tells the virtual machine whether to roll back the current transaction. The values OE_Fail, OE_Rollback, and OE_Abortdefined in sqliteInt.hwill cause the transaction to be saved, rolled back, or aborted.

A different set of operands to Halt would be seen if we violate a database constraint, for instance. The following operation was the final trace step of a command that attempted to insert a duplicate value into a PRIMARY KEY column:

14 Halt           19    2 column id is not unique

Here P1 has a value of 19, which represents a return code of SQLITE_CONSTRAINT and P2 has a value of OE_Abort (2). Additionally, P3 contains a textual error message indicating the SQL error that caused this constraint to be violated.

Let's look at a very simple SELECT statement that evaluates an expression without using any database table. The following trace output shows the VDBE opcodes along with the values on the stack as the operation takes place:

sqlite> SELECT 2+3 AS result;
   0 ColumnName      0    1 result
   1 ColumnName      1    0 NUMERIC
   2 Integer         2    0 2
Stack: si:2
   3 Integer         3    0 3
Stack: si:3 si:2
   4 Add             0    0
Stack: i:5
   5 Callback        1    0
5
   6 Halt            0    0

The first two opcodes on lines 0 and 1 are both ColumnName and contain the name and data type of the result column respectively. P1 is the number of the column, beginning at zero, for which the name in P3 is being assigned. P2 has a value of 1 to indicate the last column in the result. Remember that, with the show_datatypes pragma on, the data types are passed to the callback as later column index numbers in the result set.

The Integer opcode causes the integer value of P1 to be pushed onto the stack. P3 contains a string representation of the same integer. After line 2 the stack trace shows si:2, where si indicates that the value was placed onto the stack as both a string and an integer.

After line 3 the stack contains the values from both sides of the addition operator, and then the Add operation is performed. Add does not take any operands; it pops the top two elements from the stack, adds them together, and pushes the result back onto the stack. After this operation we can see that indeed the stack contains the value 5; however, this time it is shown as i:5 as the Add opcode has created only an integer result.

With the hard work done, the Callback opcode is called. Callback pops P1 values off the stack and invokes the callback function with an array of those values passed as the argv parameter.

The remainder of the trace shows the output of the callback function and a normal Halt operation.

Adding two integers isn't a complex operation, but it still has to be converted to VDBE opcodes in order to be evaluated by SQLite.

Let's look at an example of a database operation next. Listing 10.2 contains the virtual machine program code generated by the EXPLAIN command for a SELECT query that fetches two columns from the contacts table using an equality condition in the WHERE clause.

Listing 10.2. Virtual Machine Program Generated by EXPLAIN
sqlite> .explain
sqlite> EXPLAIN
   ...> SELECT first_name, last_name
   ...> FROM contacts
   ...> WHERE first_name = 'Chris';
addr  opcode        p1          p2          p3
----  ------------  ----------  ----------  -----------------------------------
0     ColumnName    0           0           first_name
1     ColumnName    1           1           last_name
2     ColumnName    2           0           CHAR
3     ColumnName    3           0           CHAR
4     Integer       0           0
5     OpenRead      0           4           contacts
6     VerifyCookie  0           3603
7     Rewind        0           15
8     Column        0           1
9     String        0           0           Chris
10    StrNe         1           14
11    Column        0           1
12    Column        0           2
13    Callback      2           0
14    Next          0           8
15    Close         0           0
16    Halt          0           0

Although EXPLAIN shows the complete program code that would be executed for a given SQL statement, the statement is not executed. However, the vdbe_trace pragma produces its response while the query is being executed. The trace shows the commands as they are executed, and any jumps that alter the flow of the program result in corresponding jumps in the trace output.

Listing 10.2 shows the VDBE trace for the same query featured in Listing 10.1. The operation numbers are the same in both listings so that you can compare the lines in the trace to the complete virtual machine code.

sqlite> PRAGMA vdbe_trace=on;
sqlite> SELECT first_name, last_name
   ...> FROM contacts
   ...> WHERE first_name = 'Chris';
   0 ColumnName      0    0 first_name
   1 ColumnName      1    1 last_name
   2 ColumnName      2    0 CHAR
   3 ColumnName      3    0 CHAR
   4 Integer         0    0
Stack: i:0
   5 OpenRead        0    4 contacts
   6 VerifyCookie    0 3455
   7 Rewind          0   15
   8 Column          0    1
Stack: s[Chris]
   9 String          0    0 Chris
Stack: t[Chris] s[Chris]
  10 StrNe           1   14
  11 Column          0    1
Stack: s[Chris]
  12 Column          0    2
Stack: s[Newman] s[Chris]
  13 Callback        2    0
Chris|Newman
  14 Next            0    8
   8 Column          0    1
Stack: s[Paddy]
   9 String          0    0 Chris
Stack: t[Chris] s[Paddy]
  10 StrNe           1   14
  14 Next            0    8
   8 Column          0    1
Stack: s[Tom]
   9 String          0    0 Chris
Stack: t[Chris] s[Tom]
  10 StrNe           1   14
  14 Next            0    8
   8 Column          0    1
Stack: s[Bill]
   9 String          0    0 Chris
Stack: t[Chris] s[Bill]
  10 StrNe           1   14
  14 Next            0    8
   8 Column          0    1
Stack: s[Jo]
   9 String          0    0 Chris
Stack: t[Chris] s[Jo]
  10 StrNe           1   14
  14 Next            0    8
  15 Close           0    0
  16 Halt            0    0

In operation 5, the OpenRead opcode opens a read-only cursor for a database table with the root page at the value of P2 in the database identified by P1. In this example the contacts table is opened from root page 4 of the main database. The temporary database has value 1 and any attached databases are given subsequent sequential numbers.

The VerifyCookie instruction checks that the schema version number of database P1 is equal to the value of P2. The schema version number is an arbitrary integer that changes whenever the schema is updated and is used so that SQLite knows when the schema has to be reread.

Next a Rewind takes place so that the following Column instruction will begin from the first record in the table. The P2 operand to Rewind is the jump location should the operation failin this case execution will jump to line 15 to perform an immediate Close and Halt.

The Column opcode in instruction 8 causes the column indicated by P2 to be pushed onto the stack. In this case P2 is 1, indicating the first_name column (with the first column, id, having index 0). We can see that the stack contains the string Chris, the first_name of the first row in the database.

Instruction 9 pushes the argument in the WHERE clause onto the stack ready to be compared. The comparison is performed with the StrNe opcode, which pops two elements from the stack and jumps to P2 if they are not equal.

In this example StrNe is used to implement an equality test by jumping past the conditional statement that returns execution to the top of the loop if the two strings are not equal. For this first record, the strings are equal, so operation continues to instruction 13 and pops the two fields in the result off the stack and passes them to the callback function.

The Next opcode at instruction 14 is where the jump from StrNe ends up if the strings are not equal. The cursor in P1 is advanced to the next data row and, as long as there are more records to fetch, execution jumps to the instruction number in P2.

Program execution continues for the remaining rows in the contacts table, with first_name values of Paddy, Tom, Bill, and Jo not being equal to the specified string, so no further Callback operations are performed.

Finally the Next instruction has no more rows to fetch and the virtual machine program calls a Close on the open table cursor and does a Halt with return code SQLITE_OK in instruction 16.

The complete list of opcodes for the SQLite VDBE is extensive and too large to cover in this chapter. A comprehensive list is maintained as part of the online documentation at http://sqlite.org/opcode.html if you find you want to analyze a particular SQL command in this level of detail.


Access to the Database File

Because SQLite uses a local filesystem to store its databases, the SQL language does not implement the GRANT and REVOKE commands that are commonly found in client/server database systems. They simply do not make any sense in this context.

Therefore the only access issues that need to be addressed are those associated with the database file itself and are done at the operating-system level.

File Permissions

For SQLite to be able to open a database, the user under which a process is running must have permission to access that file on the operating system.

Quite simply, to open a database for reading the process must be able to open the file in read-only mode, and to open it for writing the process must have both read and write access to the file.

On a Unix system, if the process runs as the owner of the database file, the file mode can be 0400 or 0600 for read and write access respectively. Mode 0444 would enable all users to access the database file as read-only, whereas mode 0644 would additionally grant write access to the owner of the file, and mode 0666 would allow writing by all users respectively. More information about file permissions can be found from man chmod.

Locking and Timeouts

Remember that although there is no limit to the number of concurrent reads allowed by SQLite, a single write process will lock the database and prevent any further read or write operations until it is complete.

It is therefore important to handle the SQLITE_BUSY return code if more than one process might access your database at a time. The more instances of SQLite there are running in your application, the greater chance that one of them will encounter a timeout when trying to access the database file.

The simplest approach is to continue attempting the query in a loop until SQLITE_OK is returned or some other exit forces you to stop trying, with a small pause on each iteration of the loop. Listing 10.3 shows how you might take care of this.

Listing 10.3. Handling the SQLITE_BUSY Status
[View full width]
#include <stdio.h>
#include <sqlite.h>

int main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("db1", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    return(1);
  }

  do {
    ret = sqlite_exec(db, "INSERT INTO mytable (col1, col2) VALUES (1, 2)", NULL, NULL,
   &errmsg);

    switch(ret) {

       case SQLITE_OK:    break;

       case SQLITE_BUSY:  fprintf(stderr, "Database locked, sleeping...\n");
                          sleep(1);
                          break;

       default:           fprintf(stderr, "Error: %s\n", errmsg);
                          sqlite_freemem(errmsg);
                          return(ret);

    }

  } while (ret != SQLITE_OK);

  sqlite_close(db);
  return(0);
}

The do ... while loop ensures that the query is executed at least once, and if the return code in ret is SQLITE_OK, the loop will not repeat a second time. However, if the SQLITE_BUSY error is encountered, the program outputs a message and sleeps for one second before the loop restarts and the query is attempted again.

Note

The sleep() function on Unix takes as its argument a number of seconds to wait. However, on Windows it requires a number of milliseconds. Listing 10.3 assumes a Unix platform so will only sleep for one thousandth of a second under Windows unless the value passed to sleep() is adjusted to 1000. However, it is much more useful if your program can do some other task than just hanging around while waiting to obtain a database lock.

Upon encountering any other return code, the program will halt execution displaying the contents of errmsg.

Similar functionality can be implemented using sqlite_busy_handler() to set a callback function that is invoked whenever a lock cannot be obtained.

Listing 10.4 shows a busy handler callback that will cause the program to attempt to obtain a lock five times, sleeping for one second between each try and giving up after the fifth attempt.

Listing 10.4. Using a Busy Handler Callback Function
#include <stdio.h>
#include <sqlite.h>

static int myhandler(void *NotUsed, const char *dbname, int numtries) {
  fprintf(stderr, "Database locked on try %d, sleeping...\n", numtries);
  if (numtries >= 5)
    return(0);
  else {
    sleep(1);
    return(1);
  }
}

int main()
{

  char *errmsg;
  int ret;

  sqlite *db = sqlite_open("db1", 0777, &errmsg);

  if (db == 0)
  {
    fprintf(stderr, "Could not open database: %s\n", errmsg);
    sqlite_freemem(errmsg);
    return(1);
  }

  sqlite_busy_handler(db, myhandler, NULL);

  ret = sqlite_exec(db, "INSERT INTO mytable (col1, col2) VALUES (1, 2)",
                    NULL, NULL, &errmsg);

  if (ret != SQLITE_OK) {
    fprintf(stderr, "Error: %s\n", errmsg);
    sqlite_freemem(errmsg);
    return(ret);
  }

  sqlite_close(db);
  return(0);
}

The busy callback function is registered with the line

sqlite_busy_handler(db, myhandler, NULL);

The function pointer myhandler references the callback function declared at the top of the listing. The callback takes three arguments. The first is the pointer passed as the third argument to sqlite_busy_handler(), which allows arbitrary data to be passed to the callback. We have not used this facility in this example. Secondly, the name of the database table or index that SQLite was attempting to access is passed in. The third argument contains the number of attempts that have been made to obtain a lock, and Listing 10.4 restricts the number of attempts that may be made by looking at this value.

if (numtries >= 5)
  return(0);
else {
  sleep(1);
  return(1);
}

If numtries exceeds 5, the sleep() instruction will not occur and a value of 1 is returned. When the busy callback returns zero, it instructs the query to continue attempting execution. A non-zero return value causes the query to be interrupted and the SQLITE_BUSY error to be raised.

The following output shows what happens when the program in Listing 10.4 is run on a locked database:

$ ./listing10.4
Database locked on try 1, sleeping...
Database locked on try 2, sleeping...
Database locked on try 3, sleeping...
Database locked on try 4, sleeping...
Database locked on try 5, sleeping...
Error: database is locked

Multithreaded Database Access

SQLite includes support for multithreaded database connections so that you can access your database from two or more threads simultaneously.

The ThrEADSAFE preprocessor macro is used to determine whether SQLite is built in thread-safe mode. Enable this feature by adding DTHREADSAFE=1 to an appropriate place in the Makefile. Precompiled binary distributions are thread-safe by default for Windows systems, but not for Unix systems.

The most important thing to remember is that each thread must make its own call to sqlite_open() to obtain an sqlite* type pointer. The same sqlite* pointer should not be accessed from more than one thread at a time as unpredictable results may be seen. The child process from the result of a fork() command should also open its own copy of the database after being spawned.

Be aware that because SQLite reads the database schema only once upon opening the database, if one thread changes the schema the other thread will not be able to see the amended schema until you close and reopen the database connectionjust the same as would happen in a multi-process environment.


Part IV: Appendixes

A Downloading and Installing SQLite

B Reference for the sqlite Tool

C SQL Syntax Reference

D PHP Interface Reference

E C Interface Reference

F Perl Interface Reference

G Tcl Interface Reference

H Python Interface Reference

I The Future of SQLite


Appendix A. Downloading and Installing SQLite

This appendix will show you where to go online to download the latest version of SQLite, how to pick the correct distribution, andif you decide on a source code installationhow to compile it.


Obtaining SQLite

The download page for SQLite is http://www.sqlite.org/download.html and is split into three sections:

  • Precompiled Binaries for Linux

  • Precompiled Binaries for Windows

  • Source Code

Unless you have a particular requirement to build SQLite from source, a binary distribution will work fine for Linux and Windows systems.

Version numbers shown in this appendix are current at the time of writing; however, you should check for the latest stable version.

RPM Installation for Linux

The easiest way to install SQLite on compatible Linux systems is to use the RPM packages. The rpm command is found on the RedHat and Fedora Core Linux distributions, though other distributions may also support it.

There are two RPM packages available for SQLite, found in the precompiled binaries section of the download page.

sqlite-2.8.15-1.i386.rpm contains the shared library required to run dynamically linked SQLite applications, and the sqlite program.

sqlite-devel-2.8.15-1.i386.rpm contains the static library, header files, and documentation in the form of man pages.

You should obtain both packages and install using the following steps as the root user:

# rpm -ivh sqlite-2.8.15-1.i386.rpm
Preparing...                ########################################### [100%]
   1:sqlite                 ########################################### [100%]
# rpm -ivh sqlite-devel-2.8.15-1.i386.rpm
Preparing...                ########################################### [100%]
   1:sqlite-devel           ########################################### [100%]

The sqlite package must be installed before sqlite-devel to satisfy the package dependencies.

Binary Installation for Linux

The non-RPM binary distribution of SQLite is split into three files.

sqlite-2.8.15.bin.gz is a statically linked version of the sqlite program. This can be used alone to access and create SQLite databases from the command line.

sqlite-2.8.15.so.gz is the shared library that is needed to compile programs using the C/C++ interface.

tclsqlite-2.8.15.so.gz is a library file containing the TCL bindings for SQLite as well as the C/C++ interface.

Installation of each package is done using the same basic stepsuncompress the file, move it to a suitable location on your system, and set the appropriate file permission.

The sqlite program should reside in a location that is in your path, such as /usr/bin or /usr/local/bin and should be executable by all users. Run the following commands as the root user:

# gunzip sqlite-2.8.15.bin.gz
# mv sqlite-2.8.15.bin /usr/local/bin/sqlite
# chmod 755 /usr/local/bin/sqlite

The libsqlite.so library should reside in a location where it will be found by the C compiler, such as /usr/lib or /usr/local/lib, and should also be world-readable with the execute bit set.

# gunzip sqlite-2.8.15.so.gz
# mv sqlite-2.8.15.so /usr/local/lib/libsqlite.so
# chmod 755 /usr/local/lib/libsqlite.so

On most systems, the file /etc/ld.so.conf contains a list of locations in which the linker will attempt to find a library file. In some cases it may be necessary to run the following command to ensure that libsqlite.so can be found:

# /sbin/ldconfig

The TCL-enabled library should be copied to a location that tclsh can import it from, such as a subdirectory of /usr/lib/tcl or /usr/share/tcl.

Binary Installation for Windows

The binary distribution for Windows consists of three Zip files.

sqlite-2_8_15.zip contains the sqlite.exe monitor program and library sqlite.dll.

sqlitedll-2_8_15.zip contains only the same sqlite.dll that is shipped with sqlite.exe. You do not need this package if you are also installing sqlite-2_8_15.zip.

tclsqlite-2_8_15.zip contains tclsqlite.dll, the library file with TCL bindings included.

Installation of each of these is a case of extracting the contents of the Zip file using WinZip or a similar program and moving the individual components to an appropriate location on your system.

You should either put sqlite.exe in a folder that is in your command path or add its location to the path variable. The following command adds C:\sqlite to the command path:

C:\>path %PATH%;C:\sqlite

The library files should be placed in the usual place for DLLs on your system, for example C:\WINDOWS\SYSTEM32.

Installing from Source Code

To install SQLite on a platform for which there is no binary distribution, your only option is to build from source. Two different source code archives are provided in .zip and .tar.gz format.

The following steps deal with extracting and compiling SQLite from the .tar.gz format on a Unix system. Obtain the file sqlite-2.8.15.tar.gz and extract with the following command:

# gunzip -c sqlite-2.8.15.tar.gz | tar xf

A directory called sqlite will be created. Go into this directory and run the configure command.

# cd sqlite
# ./configure
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
[...]
configure: creating ./config.status
config.status: creating Makefile
config.status: creating sqlite.pc

The output from configure is many screens of text while the program works out your system settings. When it has finished, you can commence compilation by issuing the make command, and if everything appears to be in order, install SQLite with make install.

# make
# make install

SQLite will then be installed to the appropriate places on your system.


Appendix B. Command Reference for the sqlite Tool

This appendix lists the commands that can be used within the sqlite monitor program.


Dot Commands

The dot commands for sqlite can be used to change the output format, fetch information about the database, and modify some settings. In this appendix, each dot command is shown first followed by an explanation.

Obtaining a List of Dot Commands

.help

Displays the full list of dot commands for the installed sqlite with brief descriptions.

Changing the Output Format

.mode list

The default output mode. Displays one line per record, with each column separated by a specific character or string. The default separator is the pipe character (|).

.separator string

Changes the separator for list mode output to string.

.mode lines

Causes sqlite to output each column in the result of a query to be displayed on a line by itself, with the value prefixed by the name of the column. Subsequent records are separated by a blank line.

.mode columns

Displays one line per record with data aligned in fixed-width columns.

.width width1 width2 ... widthN

Specifies the width in characters for each column in turn, where width1 is the leftmost column returned by the query.

A width setting of 0 (the default) will automatically size the column to whichever is largest of: the length of the column heading, the length of the value in the first row of data, or 10 characters.

.headers on|off

Determines whether column headings are displayed in column output mode.

.mode insert table-name

Causes a list of full INSERT statements to be generated for the records returned by the query. The table-name argument determines the name of the table for the INSERT statements.

.mode html

The output is generated as an XHTML table with one set of <TR> tags and each column as a <TD> element. If .headers on has been specified, <TH> tags are used for the column headings.

.nullvalue string

Specifies a value to display wherever a NULL value appears in the result of the query.

.echo on|off

When echo is set to on, the SQL command being executed is included as the first line of the output.

.explain on|off

Specifies a fixed-width column output mode suitable for the result of the EXPLAIN SQL command. Using .explain off reverts the output settings to their former values.

.show

Displays the current settings for echo, explain, headers, mode, nullvalue, output, separator, and width.

Reading SQL Commands from a File

.read filename

Executes one or more SQL commands from filename in turn. A syntax error in the file will cause an error to be displayed but will not prevent execution of the remaining commands.

Sending Output to a File

.output filename

Specifies that the output from queries should be sent to filename. If filename already exists, it is overwritten with the new output.

.output stdout

Specifies that output should be sent to standard outputusually the screen, but sqlite may also be used in a shell script with output redirection.

Getting Information about the Database Schema

.databases

Lists the names of currently attached databases and the paths to the database files.

.tables pattern

Lists all tables in the primary attached database that contain pattern. If no pattern is given, all tables in the database are listed. Note that tables in databases attached with the ATTACH DATABASE command are not listed.

.schema table-name

Shows the CREATE TABLE statement that was used to create table-name. If no table-name is given, the entire database schema is displayed.

.dump table-name

Generates a list of SQL commands that contain the CREATE TABLE and INSERT statements required to re-create the table specified. If no table-name is given, the entire database is dumped.

.indices table-name

Lists the names of each index in turn on the given table. The table-name argument is required.

Changing the sqlite Prompts

.prompt main continue

Allows you to customize the prompts displayed when using sqlite. The two arguments replace the main prompt (usually sqlite>) and the continuation prompt (usually ...>) respectively. If only one argument is given, it is assumed to be the main prompt.

Setting the Database Timeout

.timeout ms

Specifies the amount of time in milliseconds that sqlite will wait for a filesystem lock to clear when attempting to open a database file. If the time is exceeded, an error message is returned. The default value is zero, instructing sqlite to always return an error immediately.

Exiting the sqlite Tool

.quit

.exit

Either of these commands closes the running sqlite program. The program will also exit if an end-of-file character is received.


Appendix C. SQL Syntax Reference

This appendix summarizes the SQL language understood by SQLite. It provides the syntax for all supported SQL statements.


Naming Conventions

A standard identifier name must begin with a letter or an underscore character, and may contain any number of alphanumeric characters or underscores. No other characters can be used. There is no enforced upper limit on the length of an identifier name. Names can be as long as you like, but don't make them so long that you dread having to type them in full each time.

Square brackets or double quotes can be used to indicate a non-standard identifier name to the SQLite parser. Identifier names enclosed in this way can include characters other than the underscore, including spaces and even other square brackets, and this also allows you to use SQL keywords as identifiers.

Note

It is not generally a good idea to use non-standard identifier names in your database; although, square brackets can still be used around standard identifier names without consequence if you are familiar with this syntax from SQL Server or Microsoft Access.

Reserved Keywords

Tables C.1 through C.3 list the reserved keywords in SQLite. Table C.1 shows the fallback keywords, which can be used as identifiers without being delimited.

Table C.1. Fallback Keywords in SQLite

ABORT

AFTER

ASC

ATTACH

BEFORE

BEGIN

DEFERRED

CASCADE

CLUSTER

CONFLICT

COPY

CROSS

DATABASE

DELIMITERS

DESC

DETACH

EACH

END

EXPLAIN

FAIL

FOR

FULL

IGNORE

IMMEDIATE

INITIALLY

INNER

INSTEAD

KEY

LEFT

MATCH

NATURAL

OF

OFFSET

OUTER

PRAGMA

RAISE

REPLACE

RESTRICT

RIGHT

ROW

STATEMENT

TEMP

TEMPORARY

TRIGGER

VACUUM

VIEW

Table C.3. System Object Names in SQLite

_ROWID_

MAIN

OID

ROWID

SQLITE_MASTER

SQLITE_TEMP_MASTER

Table C.2 shows the normal keywords, which can only be used as identifier names if they are contained in square brackets or double quotes.

Table C.2. Normal Keywords in SQLite

ALL

AND

AS

BETWEEN

BY

CASE

CHECK

COLLATE

COMMIT

CONSTRAINT

CREATE

DEFAULT

DEFERRABLE

DELETE

DISTINCT

DROP

ELSE

EXCEPT

FOREIGN

FROM

GLOB

GROUP

HAVING

IN

INDEX

INSERT

INTERSECT

INTO

IS

ISNULL

JOIN

LIKE

LIMIT

NOT

NOTNULL

NULL

ON

OR

ORDER

PRIMARY

REFERENCES

ROLLBACK

SELECT

SET

TABLE

THEN

TRANSACTION

UNION

UNIQUE

UPDATE

USING

VALUES

WHEN

WHERE

Table C.3 shows the names of some system objects in SQLite that can only be used as an identifier name for a different type of object.


SQL Command Syntax

This section details the SQL command syntax understood by SQLite. For clarity SQL keywords are shown in uppercase; however, SQLite is not case sensitive. Keywords and identifiers can be typed in uppercase, lowercase, or mixed case, and different capitalizations of the same string can be used interchangeably.

Creating and Dropping Database Objects

The CREATE object and DROP object statements are used to create and drop database objects.

CREATE TABLE

To create a new database table, use CREATE TABLE.

CREATE [TEMP | TEMPORARY] TABLE table-name (
    column-def [, column-def]*
    [, constraint]*
)

A column in the CREATE TABLE statement is defined as follows:

name [type] [[CONSTRAINT name] column-constraint]*

To drop a table, use DROP TABLE.

DROP TABLE [database-name.] table-name

Column Constraints

The optional column-constraint is composed of one or more of these keywords: NOT NULL, DEFAULT, UNIQUE, PRIMARY KEY, CHECK, and COLLATE.

NOT NULL [ conflict-clause ]

NOT NULL enforces that the column must always contain a value. An error will be raised on any attempt to insert a NULL value into the column.

DEFAULT value

DEFAULT defines a value that the column should take if no value is given when a row is inserted.

UNIQUE [ conflict-clause ]

UNIQUE creates a UNIQUE INDEX on the column, ensuring that the same value cannot be entered into this column more than once. There can be more than one UNIQUE INDEX on a table if required.

PRIMARY KEY [sort-order] [ conflict-clause ]

PRIMARY KEY creates a UNIQUE INDEX on the column designated as primary key for the table. Additionally if the column type is INTEGER, this column is used internally as the actual key of the table and the value is assigned automatically by SQLite if it is not specified when a row is inserted. Only one PRIMARY KEY can be specified on each table.

CHECK ( expr ) [ conflict-clause ]

At the present time, the CHECK clause is ignored; however, SQLite allows one to be specified in the syntax for possible future use.

COLLATE collation-name

COLLATE specifies the text-collating function to be used when comparing values in this column and can be either TEXT or NUMERIC.

A UNIQUE, PRIMARY KEY, or CHECK constraint clause can also be specified after the column-def section using the following syntax:

UNIQUE ( column-list ) [ conflict-clause ]

PRIMARY KEY ( column-list ) [ conflict-clause ]

CHECK ( expr ) [ conflict-clause ]

CREATE INDEX

To create an indexa sorting key on a database tableuse CREATE INDEX.

CREATE [UNIQUE] INDEX index-name
ON [database-name.] table-name ( column-name [, column-name]* )
[ ON CONFLICT conflict-algorithm ]

There is no limit to the number of indexes that can be added to a single table, nor on the number of columns in any index.

To drop an index, use DROP INDEX.

DROP INDEX [database-name.] index-name

CREATE VIEW

To create a viewa pseudo-table based on a query on one or more tablesuse CREATE VIEW.

CREATE [TEMP | TEMPORARY] VIEW [database-name.] view-name AS select-statement

You can SELECT from a view in the same way as a regular table, but you cannot perform a COPY, DELETE, INSERT, or UPDATE operation on a view.

To drop a view, use DROP VIEW.

DROP VIEW view-name

CREATE TRIGGER

To create a triggera procedure that executes automatically on a type of database eventon a table use CREATE TRIGGER.

CREATE [TEMP | TEMPORARY] TRIGGER trigger-name [ BEFORE | AFTER ]
database-event ON [database-name.] table-name
trigger-action

To create a trigger on a view, use this syntax instead:

CREATE [TEMP | TEMPORARY] TRIGGER trigger-name INSTEAD OF
database-event ON [database-name.] view-name
trigger-action

The database-event can be DELETE, INSERT, UPDATE, or UPDATE OF column-list, any of which can reference the before and after values of a column using OLD.column-name and NEW.column-name respectively.

The syntax of trigger-action is as follows:

[ FOR EACH ROW | FOR EACH STATEMENT ] [ WHEN expression ]
BEGIN
    trigger-step ; [ trigger-step ; ]*
END

A TRigger-step can be any DELETE, INSERT, SELECT, or UPDATE statement or the special RAISE function to cause an exception to be raised in the SQL statement that caused the trigger to fire. The syntax for RAISE is as follows:

RAISE ( ABORT, error-message ) |
RAISE ( FAIL, error-message ) |
RAISE ( ROLLBACK, error-message ) |
RAISE ( IGNORE )

To drop a trigger use DROP TRIGGER.

DROP TRIGGER [database-name.] trigger-name

The SELECT Statement

To query records from a database use the SELECT statement, which has the following syntax:

SELECT [ALL | DISTINCT] result [FROM table-list]
[WHERE expr]
[GROUP BY expr-list]
[HAVING expr]
[compound-op select]*
[ORDER BY sort-expr-list]
[LIMIT integer [( OFFSET | , ) integer]]

The result takes the form of a list of one or more comma-separated columns, optionally prefixed with the table name or table alias to avoid ambiguity. A column alias can be specified with the AS keyword following a column identifier. The * character can be used in place of a column list to represent every column from the selected tables.

The table-list is either a comma-separated list of table or view names with optional aliases given following the keyword AS, or another SELECT statement supplied in parentheses. Tables can also be joined to the table-list using the following syntax.

[NATURAL] [LEFT | RIGHT | FULL] [OUTER | INNER | CROSS] JOIN
table-name
[ON expr] [USING ( id-list )]

The compound-op can be used to connect SELECT queries using set operations performed on the queries on either side of the operator. When three or more queries are compounded, they are evaluated in pairs working from left to right.

The operator can be one of the following keywords: UNION, UNION ALL, INTERSECT, or EXCEPT.

UNION

UNION takes the union of both queries, causing the result of both queries to be returned in one operation. Rows that appear in the result of both queries are returned only once, so the overall result will contain no duplicate rows.

UNION ALL

UNION ALL is the same as UNION except that duplicate rows may appear in the result.

INTERSECT

INTERSECT takes the intersection of the two datasets, causing only rows that appear in both querieseither side of the operatorto be returned once.

EXCEPT

EXCEPT takes the difference between the datasets returned by the queries. All rows from the result to the left query will be returned except those that also appear in the query to the right of the operator.

The INSERT Statement

To insert records into a table with a list of values given as part of the statement, use the following syntax for the INSERT statement:

INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
VALUES(value-list)

To insert the dataset returned by a SELECT query into a database table, use this syntax for INSERT.

INSERT [OR conflict-algorithm]
INTO [database-name .] table-name [(column-list)]
select-statement

The REPLACE statement is actually an alias for INSERT OR REPLACE provided for convenience. It can be considered to have its own syntax, as follows:

REPLACE INTO [database-name.] table-name [( column-list )] VALUES ( value-list )

REPLACE INTO [database-name.] table-name [( column-list )] select-statement

Refer to the section "Resolving Conflicts" later in this appendix for more information on this behavior.

The UPDATE Statement

To alter records in a database table, use the UPDATE statement with the following syntax:

UPDATE [ OR conflict-algorithm ] [database-name.] table-name
SET assignment [, assignment]*
[WHERE expr]

Each assignment simply takes the form column-name = expr. The expression expr may reference other column values, and all expressions are evaluated before any assignments are made.

When used without a WHERE clause, the update will be performed on every record in the database.

The DELETE Statement

To delete records from a database, use the DELETE statement.

DELETE FROM [database-name .] table-name [WHERE expr]

When used without a WHERE clause, every row in the table will be deleted.

The COPY Statement

To load a large amount of data into a table, use the COPY command.

COPY [ OR conflict-algorithm ] [database-name.] table-name FROM filename
[ USING DELIMITERS delim ]

The filename can be a file in the current directory or can contain a relative or absolute path. Using stdin for the filename will read data from standard input.

Resolving Conflicts

The ON CONFLICT and OR clauses are both used to define the conflict resolution algorithm to be used when a column constraint has been violated.

An ON-CONFLICT clause can be given in the CREATE TABLE and CREATE INDEX statements to specify the default behavior for that table. An OR clause forms part of a COPY, INSERT, or UPDATE statement and overrides any ON-CONFLICT clause on the table.

The ON-CONFLICT or OR keyword is simply followed by one of the following keywords to specify the conflict resolution algorithm to be used: ROLLBACK, ABORT, FAIL, IGNORE, or REPLACE.

ROLLBACK

Upon a violation, end the current transaction with a ROLLBACK and abort the current SQL command with a return code of SQLITE_CONSTRAINT. If there is no explicit transaction, the action is the same as ABORT.

ABORT

Upon a violation, the current SQL command is aborted with return code SQLITE_CONSTRAINT and any changes already made as part of that command are reversed. However, a ROLLBACK is not issued, so any changes from previous commands within the same transaction are committed. This is the default behavior.

FAIL

The SQL command that caused the violation will abort with return code SQLITE_CONSTRAINT; however, any changes made up to that point will be committed. For example, in an UPDATE operation, records that match the WHERE criteria that SQLite encounters before the row that causes the violation will be updated, but subsequent matching rows will not be affected.

IGNORE

When a violation occurs, that row is simply ignored and the SQL statement continues executing normally. No error is returned and the operation will be carried out successfully on every row that does not cause a violation.

REPLACE

Upon violation of a UNIQUE constraint, the record that already exists that prevents the update or insert operation from taking place is removedwithout any delete triggers being firedand the SQL statement continues as normal. Therefore an INSERT or UPDATE statement always takes place and no error is returned. If a NOT NULL constraint is violated and there is no DEFAULT value on the column, the ABORT algorithm is used instead.

Transactions

To begin a transactionan atomic block of statements that alter the databaseuse BEGIN TRANSACTION.

BEGIN [TRANSACTION [name]]

The keyword trANSACTION is optional, and the name argument, if given, is ignored. SQLite does not support nested transactions.

To end the transaction, use one of the following commands:

COMMIT [TRANSACTION [name]]

ends the transaction with any changes that have been made saved to the database.

ROLLBACK [TRANSACTION [name]]

ends the transaction, discarding any changes made within that transaction.

Attaching to Other Databases

To attach another database file to the current SQLite session, use ATTACH DATABASE.

ATTACH [DATABASE] database-filename AS database-name

The DATABASE keyword is optional. SQLite will search for database-filename in the current working directory if no path is given. After a database has been successfully attached, its tables can be referenced in SQL as database-name.table-name.

To detach a database file so that it can no longer be accessed by SQLite, use DETACH DATABASE.

DETACH [DATABASE] database-name

Performance Tuning

Use the VACUUM command to clean up your database.

VACUUM [index-or-table-name]

VACUUM copies the named index or table or, if no name is given, the entire database to a temporary area of disk and reloads the original database from that copy. By doing so, free pages in the database file are removed and the data in the file is made contiguous.

The EXPLAIN command causes SQLite to report back the virtual machine instructions that would be used to execute the command.

EXPLAIN sql-statement

From the sqlite program, the .explain command can be used to quickly set a suitable output format for the EXPLAIN command. Note that the command in sql-statement is not actually executed.

Use the PRAGMA command to modify the operation of the SQLite Library and retrieve information about the connected database.

PRAGMA name [= value]
PRAGMA function(arg)

These are the currently supported pragmas, which are discussed in depth in Chapter 10, "General Database Administration."

PRAGMA database_list
PRAGMA index_list(table-name)
PRAGMA index_info(index-name)
PRAGMA table_info(table-name)
PRAGMA foreign_key_list(table-name)

PRAGMA cache_size = Number-of-pages;
PRAGMA default_cache_size = Number-of-pages;

PRAGMA synchronous = FULL | NORMAL | OFF
PRAGMA default_synchronous = FULL | NORMAL | OFF
PRAGMA temp_store = DEFAULT | MEMORY | FILE
PRAGMA default_temp_store = DEFAULT | MEMORY | FILE

PRAGMA integrity_check

PRAGMA parser_trace = ON | OFF
PRAGMA vdbe_trace = ON | OFF

Comments

Commentstext included for reference purposes only, and to be ignored by the parserin SQLite can be written in the single-line SQL style, prefixed by two hyphens, or the multiple-line C style, inside /* ... */ characters.

-- single line SQL-style comment

/*
multiple line
C-style comment
*/


ANSI SQL Commands and Features Not Supported

The SQL language implemented by SQLite is fairly comprehensive; however, a few commands and features of the ANSI-92 specification are not available.

ALTER TABLE

SQLite does not allow the schema of a table to be changed, so the ALTER TABLE command is not supported.

To add, remove, or modify columns in a table, you must drop the table and re-create it with the revised schema. The usual way to do this is with a temporary table to hold the data from the old table while it is being re-created.

COUNT(DISTINCT column-name)

The DISTINCT keyword cannot appear inside a COUNT function. Instead you must use a nested subquery.

SELECT COUNT(DISTINCT mycol) FROM mytable;

becomes

SELECT COUNT(*) FROM (
  SELECT DISTINCT mycol FROM mytable
);

GRANT and REVOKE

Because access to SQLite databases takes place at the filesystem level, the only permissions that can be applied are those available to the underlying operating system. Therefore the GRANT and REVOKE commands are meaningless for SQLite.

INSERT, UPDATE, and DELETE on Views

SQLite does not allow write actions to be performed directly on a view, even if there is only one base table in the view. However, a trigger can be created on a view using the INSTEAD OF syntax, in which you can perform the appropriate INSERT, UPDATE, or DELETE on the underlying table(s).

RIGHT OUTER JOIN

LEFT OUTER JOIN is implemented, but not RIGHT OUTER JOIN. Therefore the table order in your queries must allow outer joins to be performed in this direction.

CHECK and FOREIGN KEY Constraints

Although the SQL syntax allows CHECK and FOREIGN KEY clauses to be included, they are ignored. They may be implemented in a future version.

Trigger Limitations

SQLite does not support the FOR EACH STATEMENT type of trigger, or INSTEAD OF TRiggers on tables. INSTEAD OF TRiggers can only be used on views.

Nested Transactions

Only one transaction can be active at a time. The name argument to BEGIN TRANSACTION is ignored.

Variable Subqueries

SQLite evaluates subqueries only once and therefore they cannot refer to variables in the main queryalso known as correlated subqueries.


Appendix D. PHP Interface Reference

This appendix lists the PHP functions that can be used to communicate with a SQLite database.

Further information and examples of usage submitted by users can often be found in the annotated PHP manual at http://www.php.net/manual/en/ref.sqlite.php.


Predefined Constants

Functions that return an array of results can take an optional result_type argument to determine what type of array is created. These are the valid constants:

  • SQLITE_ASSOC causes the array to use the string type column name as the array index.

  • SQLITE_NUM causes the array to use a numerical index starting from zero for each column in the result.

  • SQLITE_BOTH causes the array to use both string and numerical keys.

If no constant is specified, SQLITE_BOTH is assumed.


Runtime Configuration

In php.ini the sqlite.assoc_case value affects the case of column names used for key values in associative arrays. It can take the following values:

  • 0 Mixed case

  • 1 Uppercase

  • 2 Lowercase

The default behavior is to use mixed-case keys reflecting the natural case of the column headings. Using a setting of 1 or 2 will cause the case of the keys to be converted to uppercase or lowercase respectively.

Using this option incurs a slight performance penalty, but if case-folding of array keys is required, it is much faster to do so at this level than to implement it in your code.


Function Reference

In this section, SQLite functions and their results are described and organized by function category. The function is given first, followed by its description.

Opening and Closing a Database

resource sqlite_open ( string filename [, int mode [, string &error_message]])

Opens the database specified by filenameeither in the current directory or referenced via a relative or absolute pathand returns a database connection resource.

resource sqlite_popen ( string filename [, int mode [, string &error_message]])

Opens a persistent connection to the specified database file. To open an in-memory database, use :memory: as the filename.

Note

When the sqlite_popen() function is run in a web environment, file permissions must allow the web server user to gain read and write access to both the database file itself and its directory so that journal files can be written.

void sqlite_close ( resource dbhandle)

Closes a database connection.

void sqlite_busy_timeout ( resource dbhandle, int milliseconds)

Specifies the duration in milliseconds for which SQLite should wait for a lock to clear on the database file before failing with an SQLITE_BUSY return code. The default value is 60 seconds (60,000 milliseconds).

Executing a Query

resource sqlite_query ( resource dbhandle, string query)
resource sqlite_query ( string query, resource dbhandle)

Causes SQLite to execute the given query and returns a seekable result set resource.

Note

The arguments to sqlite_query() and other such functions can be specified in either order for consistency with both other SQLite interfaces and other PHP extensions. The preferred ordering is to specify the database resource firstthe order used by other SQLite extensions.

resource sqlite_unbuffered_query ( resource dbhandle, string query)
resource sqlite_unbuffered_query ( string query, resource dbhandle)

Works similarly to sqlite_query() but returns a result resource that can only be accessed sequentially. Unless random access is required, this function should be used as it provides much better performance.

string sqlite_escape_string ( string item)

Returns a version of item that is safe for storage by SQLite, by doubling up single quotes and encoding characters that are not binary-safe.

Error Reporting

int sqlite_last_error ( resource dbhandle)

Returns the error code of the last operation performed on database resource dbhandle.

string sqlite_error_string ( int error_code)

Converts the return value from sqlite_last_error() into a human-readable error message string.

Finding Information About a Query

int sqlite_last_insert_rowid ( resource dbhandle)

Returns the most recently assigned autoincrementing value of an INTEGER PRIMARY KEY field in dbhandle.

int sqlite_changes ( resource dbhandle)

Returns the number of rows affected by the most recent UPDATE or DELETE operation on dbhandle.

int sqlite_num_rows ( resource result)

Returns the total number of rows returned in the result resource result.

int sqlite_num_fields ( resource result)

Returns the total number of fields in the query that produced result resource result.

string sqlite_field_name ( resource result, int field_index)

Returns the name of the field at position field_index in result. The leftmost field in the query has field index zero.

Processing a Result Set

mixed sqlite_column ( resource result, mixed index_or_name
                      [, bool decode_binary])

Returns the value of the column from result resource result at the current result handle pointer specified by either field index index or column name name.

bool sqlite_next ( resource result)

Advances the result handle pointer for result to the next row.

bool sqlite_has_more ( resource result)

Returns trUE if result has more rows still to be fetched. FALSE indicates that the result handle pointer is at the end of the result.

array sqlite_fetch_array ( resource result
                           [, int result_type [, bool decode_binary]])

Returns an array containing the next row of data from result and advances the result handle pointer. The types of key values for the array are determined by the optional result_type constant, discussed at the beginning of the chapter.

array sqlite_current ( resource result
                       [, int result_type [, bool decode_binary]])

Returns an array containing the current row of data from result. The result handle pointer is not changed.

string sqlite_fetch_single ( resource result
                             [, int result_type [, bool decode_binary]])

Returns an array containing only the first row of data from result. This is the most optimal way to process a result if only the first record returned is required.

Random Data Access Functions

A buffered, seekable result resource is required for these functions. They cannot be used if the result was returned by sqlite_unbuffered_query().

bool sqlite_seek ( resource result, int rownum)

Moves the result handle pointer for result to the row specified by rownum. Returns trUE on success or FALSE if rownum does not exist.

bool sqlite_rewind ( resource result)

Moves the result handle pointer to the very first row of result. Equivalent to using sqlite_seek() to row zero.

Convenience Functions

array sqlite_array_query ( resource dbhandle, string query
                          [, int result_type [, bool decode_binary]])
array sqlite_array_query ( string query, resource dbhandle
                          [, int result_type [, bool decode_binary]])

Returns a single, two-dimensional array made up of associative arrays of values for each record in the dataset returned by query.

In $array[row][col], row is the row number beginning at zero and col is the field index number or name depending on the optional result_type setting.

Finding Information About SQLite

string sqlite_libversion ( void )

Returns a string containing the version of the library linked with PHP.

string sqlite_libencoding ( void )

Returns a string containing the encoding version of the linked library.

Custom Functions (UDF)

bool sqlite_create_function ( resource dbhandle, string function_name,
                              mixed callback [, int num_args])

Registers a user-defined SQL function called function_name that executes the PHP code in the function with the name given in the callback argument. The function is available only within the dbhandle resource.

bool sqlite_create_aggregate ( resource dbhandle, string function_name,
                               mixed step_func, mixed finalize_func
                               [, int num_args])

Registers a user-defined aggregating SQL function called function_name that executes step_func once for each row in the query, and finalize_func once after all rows have been returned.

string sqlite_udf_encode_binary ( string data)
string sqlite_udf_decode_binary ( string data)

Encodes and decodes, respectively, binary data passed as parameters to a user-defined function. These functions impact performance and should only be used where a user-defined function is required to handle binary data.


Appendix E. C Interface Reference

This appendix lists the C library functions that can be used to communicate with a SQLite database.


The Core API

The core interface to the SQLite library is considered to be just three functions that allow you to open and close a database and to execute a query using a user-defined callback function. In this section we'll also look at the error codes returned from the core API.

Opening and Closing a Database

You can open and close a database as follows:

sqlite *sqlite_open(
  const char *dbname,
  int mode,
  char **errmsg
);

void sqlite_close(sqlite *db);

The return value of sqlite_open() and the argument to sqlite_close() is an opaque sqlite data structure.

typedef struct sqlite sqlite;

Executing a Query

You can execute a query as follows:

int sqlite_exec(
  sqlite *db,
  char *sql,
  int (*xCallback)(void *, int, char **, char **),
  void pArg,
  char **errmsg
);

The callback function has the following prototype:

int callback(void *pArg, int argc, char **argv, char **columnNames) {
  return 0;
}

Error Codes

This section describes SQLite error codes; each error code is followed by a description of its meaning.

#define SQLITE_OK           0   /* Successful result */

Returned if everything worked and there were no errors.

#define SQLITE_ERROR        1   /* SQL error or missing database */

Indicates an error in the SQL statement being executed.

#define SQLITE_INTERNAL     2   /* An internal logic error in SQLite */

Indicates that an internal consistency check within the SQLite library failed. This code will only ever be returned as a result of a bug, and such occurrences should be reported to the SQLite mailing list.

#define SQLITE_PERM         3   /* Access permission denied */

Indicates that the database file cannot be opened due to insufficient file permissions.

#define SQLITE_ABORT        4   /* Callback routine requested an abort */

This value is returned if the callback function returns a non-zero value.

#define SQLITE_BUSY         5   /* The database file is locked */

Indicates that another program or thread has the database locked. Only one thread can open a database file for writing at a time, though multiple reads can take place simultaneously.

#define SQLITE_LOCKED       6   /* A table in the database is locked */

Indicates that the database is locked by a recursive call to sqlite_exec(). Calls to sqlite_exec() from within a callback function are permitted as long as they do not attempt to write to the same table.

#define SQLITE_NOMEM        7   /* A malloc() failed */

This value is returned if a call to malloc() fails due to not enough system memory being available.

#define SQLITE_READONLY     8   /* Attempt to write a readonly database */

Indicates that an attempt was made to write to a database file that was opened in read-only mode.

#define SQLITE_INTERRUPT    9   /* Operation terminated by sqlite_interrupt() */

This value is returned if a database operation is interrupted by the sqlite_interrupt() function being called.

#define SQLITE_IOERR       10   /* Some kind of disk I/O error occurred */

This value is returned if the operating system fails to perform a disk I/O operation, for instance if there is no space left on the device.

#define SQLITE_CORRUPT     11   /* The database disk image is malformed */

This value would only be returned as a consequence of an unknown error in SQLite or a hardware or operating-system malfunction. It may indicate that the database file has become corrupted, or that a disk I/O error has forced SQLite to leave the database file in a corrupted state.

#define SQLITE_NOTFOUND    12   /* (Internal Only) Table or record not found */

For internal use only. This value will never be returned by sqlite_exec().

#define SQLITE_FULL        13   /* Insertion failed because database is full */

This value is returned if an insert operation fails because the database is too big to take any more information, for instance when the size of the database file would exceed the operating system's file size limit.

#define SQLITE_CANTOPEN    14   /* Unable to open the database file */

This value indicates that that database file could not be opened for some other reason that does not have its own return code.

#define SQLITE_PROTOCOL    15   /* Database lock protocol error */

Indicates that the file-locking protocol on SQLite's rollback journal files has been violated.

#define SQLITE_EMPTY       16   /* (Internal Only) Database table is empty */

For internal use only. This value will never be returned by sqlite_exec().

#define SQLITE_SCHEMA      17   /* The database schema changed */

Indicates that another process has altered the database schema since it was first read into memory when the database was initially opened. SQLite will re-read the schema when this happens but will not be able to complete the SQL statement that was being processed. Submitting the statement a second time will usually allow the command to be executed.

#define SQLITE_TOOBIG      18   /* Too much data for one row of a table */

Indicates that the maximum row size for a table would be exceeded. The limit is defined at compile time by the MAX_BYTES_PER_ROW value in sqliteInt.h.

#define SQLITE_CONSTRAINT  19   /* Abort due to contraint violation */

Indicates that the SQL statement would have violated a database constraint.

#define SQLITE_MISMATCH    20   /* Data type mismatch */

This value is returned when an attempt is made to insert non-integer data into an INTEGER PRIMARY KEY column.

#define SQLITE_MISUSE      21   /* Library used incorrectly */

Indicates that the SQLite library has been misused in some way, for instance calling sqlite_exec() after the database has been closed, or calling sqlite_step() after a SQLITE_DONE or SQLITE_ERROR return code.

#define SQLITE_NOLFS       22   /* Uses OS features not supported on host */

This value is returned if an attempt is made to access a database file larger than 2GB in size on a legacy operating system that does not include large file support.

#define SQLITE_AUTH        23   /* Authorization denied */

Indicates that the authorizer callback has disallowed the SQL you are attempting to execute.


The Non-Callback API

The non-callback API provides an alternative way to retrieve data from a SQLite database by compiling an SQL statement into a virtual machine of type sqlite_vm:

typedef struct sqlite_vm sqlite_vm;

Creating a Virtual Machine

You can create a SQLite virtual machine as follows:

int sqlite_compile(
  sqlite *db,              /* The open database */
  const char *zSql,        /* SQL statement to be compiled */
  const char **pzTail,     /* OUT: uncompiled tail of zSql */
  sqlite_vm **ppVm,        /* OUT: the virtual machine to execute zSql */
  char **pzErrmsg          /* OUT: Error message. */
);

The return code from sqlite_compile() is SQLITE_OK if the operation is successful; otherwise, one of the error codes listed in the preceding example is returned.

Step-by-Step Execution of an SQL Statement

Each invocation of sqlite_step() for a virtual machine, except the last one, returns a single row of the result:

int sqlite_step(
  sqlite_vm *pVm,          /* The virtual machine to execute */
  int *pN,                 /* OUT: Number of columns in result */
  const char ***pazValue,  /* OUT: Column data */
  const char ***pazColName /* OUT: Column names and datatypes */
);

int sqlite_finalize(
  sqlite_vm *pVm,          /* The virtual machine to be finalized */
  char **pzErrMsg          /* OUT: Error message */
);

Return Codes

The return code from sqlite_step() can be SQLITE_BUSY, SQLITE_ERROR, SQLITE_MISUSE, or either of the following.

#define SQLITE_ROW         100  /* sqlite_step() has another row ready */

Indicates that another row of result data is available.

#define SQLITE_DONE        101  /* sqlite_step() has finished executing */

Indicates that the SQL statement has been completely executed and sqlite_finalize() should now be called.

The return code from sqlite_finalize() indicates the overall success of the SQL command and will be the same as if the query had been executed using sqlite_exec().


The Extended API

The extended API provides a range of non-core functions to assist with development of software that uses an embedded SQLite database.

Finding Information About the SQLite Library

const char sqlite_version[];

Contains the current library version number.

const char sqlite_encoding[];

Contains the current library encoding version.

Finding Information About Changes to the Database

Several functions can return information about changes that have been made to the database.

int sqlite_last_insert_rowid(sqlite *db);

Returns the most recently assigned autoincrementing value of an INTEGER PRIMARY KEY field.

int sqlite_changes(sqlite *db);

Returns the number of rows affected by an UPDATE or DELETE statement.

Checking SQL Statements

int sqlite_complete(const char *sql);

Returns trUE if a complete SQL statement is provided, that is, the statement ends with a semicolon. Returns FALSE if more characters are required.

Interrupting an SQL Statement

void sqlite_interrupt(sqlite *db);

Causes the current database operation to exist at the first opportunity, returning SQLITE_INTERRUPT to the calling function.

Convenience Functions

The following function fetches the entire result of a database query with a single function call:

int sqlite_get_table(
  sqlite *db,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg
);

The result will be an array of string pointers containing one element for each column of each row in the result. The first ncolumn elements contain the column names returned. Use nrow and ncolumn to determine which elements of the array correspond to which values.

The return code from sqlite_get_table() is the same as if sqlite_exec() had executed the query.

void sqlite_free_table(char **azResult);

Frees memory allocated by sqlite_get_table() when it is no longer required.

The _printf() Wrapper Functions

char *sqlite_mprintf(const char *zFormat, ...);

Works like sprintf() but also allows the format strings %q and %Q to manipulate strings for database storage. %q escapes any single quotes by doubling the quote character; %Q additionally encloses the result string within single quotes.

int sqlite_exec_printf(
  sqlite *db,
  char *sqlFormat,
  int (*)(void *, int, char **, char **),
  void *pArg,
  char **errmsg,
  ...
);

Combines sqlite_exec() with sqlite_mprintf(). The format string references items from the sixth argument onwards.

int sqlite_get_table_printf(
  sqlite *db,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg,
  ...
);

Combines sqlite_get_table() with sqlite_mprintf(). The format string references items from the seventh argument onwards.

Memory Management

void sqlite_freemem(void *p);

Where the library allocates memory using malloc() and returns a pointer to that data, such as errmsg or the result of sqlite_mprintf(), it is the responsibility of the calling program to free that memory.

Dealing with Locked Database Files

void sqlite_busy_timeout(
  sqlite *db,
  int ms
);

Specifies an amount of time in milliseconds for which SQLite will wait for a file lock to clear before returning SQLITE_BUSY.

void sqlite_busy_handler(
  sqlite *db,
  int (*xBusy)(void *, const char *, int),
  void *pArg
);

Specifies an alternative busy handler function xBusy. A non-zero response from the handler will cause SQLite to retry; a zero response causes SQLITE_BUSY to be returned.

Performing Background Jobs During Large Queries

void sqlite_progress_handler(
  sqlite *db,
  int nOps,
  int (*xProgress)(void *),
  void *pArg
);

Registers a callback routine xProgress to be invoked every nOps virtual machine operations during long-running query execution.


Adding New SQL Functions

SQLite allows you to add new functions to the SQL language that can subsequently be used in your queries.

Registering Functions

int sqlite_create_function(
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xFunc)(sqlite_func*,int,const char**),
  void *pUserData
);

Creates a regular function in SQL from the function pointed to by xFunc.

int sqlite_create_aggregate(
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xStep)(sqlite_func*,int,const char**),
  void (*xFinalize)(sqlite_func*),
  void *pUserData
);

Creates an aggregating function with function xStep executed once for each row returned by the query, and xFinalize invoked once after all rows have been returned.

The xFunc and xStep arguments are pointers to functions with the following prototype.

void xFunc(
  sqlite_func *context,
  int argc,
  const char **argv
);

The finalize function requires only the context argument.

void xFinalize(sqlite_func *context);

The context argument is an opaque data type sqlite_func.

typedef struct sqlite_func sqlite_func;

Setting Return Values

char *sqlite_set_result_string(
  sqlite_func *p,
  const char *zResult,
  int n
);

void sqlite_set_result_int(
  sqlite_func *p,
  int iResult
);

void sqlite_set_result_double(
  sqlite_func *p,
  double rResult
);

Use the appropriate function for the data type that is to be returned to SQL.

void sqlite_set_result_error(
  sqlite_func *p,
  const char *zMsg,
  int n);

Returns an error code to SQLite.

The integer n parameter to sqlite_set_result_string() and sqlite_set_result_error() is the number of characters to be returned. A negative value will return up to and including the first \0 character.

Referencing Arbitrary Data

void *sqlite_user_data(sqlite_func *p);

Returns the pUserData pointer given in the corresponding sqlite_create_function() or sqlite_create_aggregate() call.

void *sqlite_aggregate_context(
  sqlite_func *p,
  int nBytes
);

Allocates memory that is unique to a particular instance of the SQL function being called. Memory allocated is automatically cleaned up when the finalize function is invoked.


Appendix F. Perl Interface Reference

This appendix is a reference for the Perl DBI module, which can be used to communicate with a SQLite database. Methods and attributes specific to the DBD::SQLite module are also listed.

In this appendix, the commands or methods are shown first, followed by the explanation.


The Perl DBI

The Perl interface to SQLite is the Perl Database Interface (DBI) module using the SQLite Database Driver (DBD) module.

use DBI;

Loads the DBI module into your script. There is no need to explicitly load a DBD driver as DBI takes care of this automatically.

In this appendix we use $dbh to refer to a database handle object and $sth for a statement handle.

Opening and Closing a Database

$dbh = DBI->connect($data_source, $username, $auth, \%attr);

Opens a database connection using the arguments given and creates a database handle object $dbh.

The data_source argument for a SQLite connection takes the following form, where dbfile can refer to a file in the current working directory or may contain a relative or absolute path.

DBI:SQLite2:dbname=dbfile

The username and auth arguments are not required for a SQLite database and can be omitted or passed as empty strings.

The optional attr argument can be used to set certain database handle attributes, described later in this appendix.

$rc = $dbh->disconnect();

Closes an open database connection, returning a Boolean $rc value.

Executing SQL Statements

The following methods can be called on a statement handle:

$sth = $dbh->prepare($sql);
$sth = $dbh->prepare($sql, \%attr);

Prepares a statement, $sql, for execution by the database and returns a statement handle object, $sth.

The optional attr argument can be used to set certain statement handle attributes, described later in this appendix.

$sth->execute();
$sth->execute(@bind_values);

Executes a prepared SQL statement. Return value $rv will be the number of rows affected, or -1 if not known. If zero rows are affected, the return value is 0E0, which is treated as a zero value but regarded as true. A zero return code indicates statement failure.

$dbh->do($sql);
$dbh->do($sql, @bind_values);

A convenience function that prepares and executes a query in one step. The return value is the same as for execute().

Using Bind Values

The optional bind_values list in execute() and do() can be used to specify a comma-separated list of values for placeholders in the SQL statement. Upon execution, question mark characters in the SQL are replaced with the bind values in order. The underlying function called to bind a value is bind_param().

$sth->bind_param($p_num, $bind_value)
$sth->bind_param($p_num, $bind_value, $bind_type)

Parameter number $p_num is replaced with $bind_value and an optional data type hint can be given as $bind_type. Bind parameters are numbered starting at 1.

Making Data Safe

Data inserted using bind parameters is automatically safe; however, to ensure that data from static parameters will be inserted into the database correctly, some delimiting may be necessary.

$dbh->quote($str);

Returns $str enclosed in single quotes, with any awkward characters correctly delimited.

Getting Information About a Query

The rows() method can be called on a statement handle:

$dbh->rows();

Returns the number of rows affected by the most recent UPDATE or DELETE statement.

The last_insert_rowid private method from DBD::SQLite returns the last assigned value of an autoincrementing INTEGER PRIMARY KEY field. It is invoked as follows:

$dbh->func("last_insert_rowid");

Transactions

The following methods can be called on a database handle:

$dbh->begin_work();

Begins a transaction. Equivalent to issuing the SQL command BEGIN TRANSACTION.

$dbh->commit();

Ends a transaction, saving any changes that have been made to the database during the current transaction. Equivalent to issuing the SQL command COMMIT TRANSACTION.

$dbh->rollback();

Ends a transaction without saving the changes made during the current transaction. Equivalent to issuing the SQL command ROLLBACK TRANSACTION.

Fetching Rows from a Query

The following methods can be called on a statement handle:

$sth->fetchrow_arrayref();
$sth->fetchrow_array();
$sth->fetchrow_hashref();
$sth->fetchrow_hashref($name);

Fetches the next row of data and returns a data structure holding the field values. The data structure is an array reference, list, or hash reference respectively. NULL values are returned as undef.

If there are no more rows to fetch or an error occurs, the return value will be undef for fetchrow_arrayref() and fetchrow_hashref(), and an empty list for fetchrow_array().

The optional $name specifies the statement handle attribute that will be used as the key name in the hash returned by fetchrow_hashref(). The default is NAME, with NAME_uc and NAME_lc used to force the case of the key name.

$sth->fetchall_arrayref();
$sth->fetchall_arrayref($slice);
$sth->fetchall_arrayref($slice, $max_rows);

Returns an array containing the entire dataset from the prepared statement handle.

The optional $slice argument can limit which columns become part of the array using a list of column numbers starting from zero. Negative numbers can be used to count from the end of the column list. If no slice is required the argument can be undef.

The optional $max_rows argument can be used to limit the number of rows copied into the array.

$sth->fetchall_hashref($key_field);

Returns a hash that contains one entry per row for the entire dataset. The $key_field argument specifies the column name that is to be used as the key field in the hash.

$sth->finish();

Frees resources associated with a statement resource after it is finished with them.

Error Reporting

The following methods can be called on a database handle to retrieve information about the most recent error.

$dbh->err();

Returns the error code number. For a list of error code values, refer to Appendix E, "C/C++ Interface Reference."

$dbh->errstr();

Returns a string containing a description of the error.

Automatic error checking is activated using the PrintError and RaiseError database attributes, which pass the error message to warn() and die() respectively whenever an error is encountered.

$dbh->{PrintError} = 1;

Enables automatic error reporting via warn(). Program execution will continue after an error is reported.

$dbh->{RaiseError} = 1;

Enables automatic error reporting via die(). Program execution will terminate when the error occurs.

$dbh->trace($level);
$dbh->trace($level, $filename);

Sets the trace level for DBI methods performed on that database handle and an optional output filename. Level can be 1 to 15, with 1 and 2 being most useful. Level zero turns tracing off.

Tracing can also be enabled via the DBI_TRACE environment variable.

DBI_TRACE=1 perl myscript.pl

Runs myscript.pl with trace output at level 1 sent to stdout.

DBI_TRACE=2=trace.log perl myscript.pl

Runs myscript.pl with trace output at level 2 sent to trace.log.

Creating User-Defined Functions

The following calls can be made to private methods implemented by DBD::SQLite:

$dbh->func( $name, $argc, $func_ref, "create_function" )

Registers a new function, $name, in the SQL language that takes $argc arguments and calls the Perl function referenced by $func_ref. If $argc is -1, the SQL function will accept any number of arguments.

$dbh->func( $name, $argc, $pkg, 'create_aggregate' )

Registers a new aggregating function $name in the SQL language. $pkg is a package that implements the aggregator interface, comprising three methods:

  • new() is called once and returns the object on which the other methods should be run.

  • step() is called once for each row that is to be aggregated.

  • finalize() is called after all the rows have been passed to step() and returns the aggregated result.


Appendix G. Tcl Interface Reference

This appendix lists the Tcl commands that are used to communicate with a SQLite database.


The Tcl Library

The Tcl library for SQLite is called libtclsqlite.so on Unix platforms and sqlite.dll on Windows systems. The library file should reside in a location that your scripts will be able to load from. Typically this would be a subdirectory of /usr/share/tcl on Unix or C:\tcl\lib on Windows.

In this appendix, commands from the Tcl library are shown first, followed by the explanation.

package require sqlite

Imports the sqlite package into a Tcl script.

Opening and Closing a Database

sqlite dbcmd database-name

Opens a database called database-nameeither from the current directory or referenced via a relative or absolute pathcreating it if it does not already exist. On success, a new command called dbcmd is registered in Tcl, upon which the methods described in the rest of this appendix can be applied.

dbcmd close

Closes the database connection and destroys dbcmd.

Executing a Query

dbcmd eval query

Causes SQLite to execute the given query and returns the entire data set fetched as a single list.

dbcmd eval query array { code-block }

As SQLite executes the query, for each row fetched an element in array is created for every column returned by the query and the commands in code-block are executed.

Additionally the data types of the returned columns are stored as elements named typeof:column-name in array, and the list of columns returned can be found in array(*).

dbcmd eval query {} { code-block }

If the empty string is used in place of array, code-block is still executed once for each row in the dataset; however, the fetched columns are stored to scalar variables with the same name as their respective columns.

Convenience Functions

dbcmd onecolumn query

Causes SQLite to execute the given query as eval; however, it returns only the first column from the first row of the dataset.

Finding Information About a Query

dbcmd last_insert_rowid

Returns the most recently assigned auto-incrementing value of an INTEGER PRIMARY KEY field following an INSERT operation on dbcmd.

dbcmd changes

Returns the number of rows affected by the most recent UPDATE or DELETE operation on dbcmd or the number of rows inserted by an INSERT statement.

Checking SQL Statements

dbcmd complete query

Returns true if a complete SQL statement is provided, that is, the statement ends with a semicolon in the appropriate place. Returns false if more characters are required to complete the statement.

Dealing with Locked Database Files

dbcmd timeout ms

Specifies the duration in milliseconds for which SQLite should wait for a lock to clear on the database file when performing a write operation. The default value is zero, meaning it will not wait or retry if the file is locked.

dbcmd busy callback

Specifies a callback function to be executed in the event that a database lock cannot be obtained for a write operation. The callback should return zero if you want SQLite to continue trying to get a lock, or non-zero to interrupt the SQL statement.

Error Reporting

dbcmd errorcode

Returns the numeric error code that resulted from the most recent SQLite operation. The full list of error code values can be found in Appendix E, "C/C++ Interface Reference."

Finding Information About SQLite

The sqlite_version function will return the version of the tclsqlite library in use.

sqlite> select sqlite_version(*);
2.8.13

Custom Functions (UDF)

dbcmd function tcl-func sql-func

Registers a user-defined SQL function called sql-func that executes the Tcl code in the function with the name tcl-func. The function is available only within that instance of dbcmd.


Appendix H. Python Interface Reference

This appendix is a reference for the PySQLite extension, which provides an interface to SQLite that is compliant with the Python Database Specification 2.0.

Before attempting any database connectivity with PySQLite, you must import the sqlite module, using

import sqlite

In this appendix, the commands are shown first, followed by the explanation.


Opening and Closing a Database

cx = sqlite.connect(database, mode=0755, converters={}, autocommit=0,
                    encoding=None, timeout=None, command_logfile=None)

Opens a SQLite connection to database and returns a connection object to cx.

The mode argument is currently ignored but may be used in the future to specify the file mode with which the database file is opened.

The converters argument can be used to specify mappings from SQL data types using Python conversion functions.

Set autocommit to 1 if you want PySQLite to commit each SQL command immediately when executed, rather than the default behavior of batching groups of INSERT, UPDATE, and DELETE statements into a single transaction.

The encoding argument allows you to specify the encoding to be used on Unicode strings. Its value can be noneif only ASCII characters are to be acceptedor utf-8.

The default timeout value of None means that if the database file is locked, SQLite will return an error immediately. Setting a value in seconds instructs the database engine to keep trying to obtain a lock for the given amount of time.

Use command_logfile with a file object argument to specify a file to which all SQL statements executed through PySQLite will be written.

cx.close()

Closes a database connection and rolls back any uncommitted transactions. The connection object is unusable after .close() has been called.

Executing SQL Statements

All SQL statements must be executed through a cursor object.

cu = cx.cursor()

Creates a new cursor object for connection cx.

cu.execute(sql)
cu.execute(sql, args)

Passes sql to SQLite for execution. Formatted strings are permitted in sql using the standard Python format codes. Unsafe characters do not require delimiting if passed as format arguments.

cx.commit()

Issues a COMMIT TRANSACTION command to store any unsaved changes to the database. A .commit() operation is always required to save changes when autocommit is off.

cx.rollback()

Issues a ROLLBACK TRANSACTION command to reject any unsaved changes.

numrows = cu.rowcount

Following an operation that changes the database, cu.rowcount returns the number of affected rows. Following a SELECT statement, cu.rowcount returns the number of records in the dataset.

row = cu.fetchone()

Fetches the next data record and advances the cursor to the next row. Returns None if there are no more rows to select.

rows = cu.fetchmany(num)

Fetches the next num data records and advances the cursor row number by num.

rows = cu.fetchall()

Fetches all remaining data records from a cursor.

rownum = cu.rownumber

Returns the current cursor row position.

Setting Data Type Mappings

The default mappings from SQLite data types to Python are as shown in Chapter 9, "The Python Interface," in Table 9.2.

cu.execute("-- types col1, col2, ...")

Specifies the expected data types for col1, col2, and so forth on the next query executed on cursor cu.


Creating User-Defined Functions

cx.create_function(name, argc, func)

Registers a user-defined function called name in SQL using the Python function func. The number of arguments to the function is given in argc.

cx.create_aggregate(name, argc, aggregate)

Registers a user-defined aggregating function called name in SQL using the Python class aggregate, which must contain the member functions reset(), step(), and finalize().


Error Handling

PySQLite extends the StandardError class with its own Error class, with which errors related to SQLite operations can be handled.

Two subclasses of Error are implementedInterfaceError and DatabaseError. These allow you to detect whether the error originates with the SQLite engine or the PySQLite interface.

The DatabaseError class is further subdivided as follows:

  • DataError

  • OperationalError

  • IntegrityError

  • InternalError

  • ProgrammingError

  • NotSupportedError


Appendix I. The Future of SQLite

This appendix looks at how SQLite might be improved, enhanced, or extended in the future.


SQLite Version 3.0

At the time of writing, SQLite version 3.0 was close to release, and it may very well be available as a stable release by the time you are reading this book.

The decision was made to cover the latest 2.8 version throughout the book primarily because of the vast user base this version of SQLite already has.

Although version 3.0 adds some exciting functionality, it also has a completely different API, so existing users will be slow to migrate, if they switch to the new version at all. New users should consider carefully whether they want to use the new version or stick with a tried and tested library for their application.

New features will no longer be added to SQLite 2.8, but it will continue to be supported and have maintenance fixes issued for the foreseeable future, so do not be put off from using it simply because a new version has been released.

Let's take a look at some of the changes in SQLite version 3.0.

Naming Changes

Because it is important that SQLite 2.8 can continue to be used while the new version is introduced, version 3.0 uses a new naming convention and allows both versions of the SQLite library to be linked to the same program if required.

The library itself has been renamed to libsqlite3.so on Unix and sqlite3.dll on Windows, and the include file is now called sqlite3.h. The sqlite command-line tool has also been renamed to sqlite3.

Within the API itself, function names have also been changed to be prefixed with sqlite3, but the names themselves remain familiar; for example, sqlite3_open() and sqlite3_changes() will behave as expected.

Although most of the sqlite3 prefixed function names have the same prototype as their corresponding functions in SQLite 2, one significant difference is the sqlite3_open() function, which works as follows:

int sqlite3_open(const char *filename, sqlite3 **ppDb)

In SQLite 3, the return value from the database connection function is SQLITE_OK on success or the corresponding error code on failure. The database handle is returned in *ppDb.

The full list of version 3 API calls can be found at http://www.sqlite.org/capi3ref.html.

File Format Changes

The database file format has been overhauled. Version 2.8 databases cannot be read by the 3.0 library, and the reverse is true.

The new file format uses a B+Tree data structure for table storage to replace the B-Trees, which allows better scalability within a filesystem-based database. It is also more highly optimized through omitting unused fields from the disk image and better encoding of floating-point numbers, which has been shown to produce a 2535% reduction in the overall file size.

Migrating from the old database file format to SQLite 3.0 is very simple if you have both the sqlite and sqlite3 command-line tools installed. The following command would effectively upgrade olddb to the new file format, saving the new version as newdb.

$ sqlite olddb .dump | sqlite3 newdb

Data Typing

The typeless nature of SQLite columns has been altered in order that BLOBs (Binary Large Objects) can be supported.

Though any value can still be stored in a column of any declared data type, SQLite version 3.0 assigns each value stored a particular data type class. The library receives all values as strings; however, if a string appears to contain only a number, it may be considered an INTEGER or NUMERIC, depending on whether or not the number has a fractional part.

The data type classes in SQLite 3.0 are

  • NULL

  • TEXT

  • REAL

  • INTEGER

  • BLOB

The type of a value is associated with that value itself and not the column in which it is stored. This is sometimes called manifest typing and is not found in other SQL engines, where the declared column data type determines the type of all the values stored in it.

SQLite supports the concept of type affinity on columns, where the declared data type is the recommended type for values that are inserted into that column. Significantly, the declared types are only recommendations and not requirements, and it is still possible to insert any kind of data into any column. However, the declared column type will sometimes influence the manifest type of a value.

A column with TEXT affinity will store all data using only the NULL, TEXT, and BLOB classes. A numeric value will be converted to TEXT before storage. If the column type definition contains CHAR, CLOB, or TEXT, the column will be given TEXT affinity.

If the column type definition contains BLOB or no data type is given in the schema, the column will have affinity to NONE. This property ensures that no attempt at conversion will be made on inserted values.

An INTEGER or NUMERIC column can use any of the storage classes, but attempts to convert the value to an integer or real number before resorting to a TEXT or BLOB type. A decimal number inserted into a column with INTEGER affinity will cause the fractional part to be discarded and the number to be stored as an integer data type.

If the column type definition contains INT, its affinity will be INTEGER; otherwise, it will be NUMERIC.

Tables created with the CREATE TABLE ... AS SELECT syntax will have no data type definitions. All columns will have affinity NONE.

A few special rules apply when comparing values of different data types. Although INTEGER and NUMERIC types are compared numerically as you would expect, an INTEGER or NUMERIC type is always considered to be less than any TEXT or BLOB regardless of the actual values involved. Any TEXT type is also always less than any BLOB. When a TEXT or BLOB value is compared to a value of the same type, memcpy() is used to evaluate which is the greater value. A NULL is always considered less than a value of any other type.

SQLite 3.0 also features a strict affinity mode, whereby if a data item inserted into a column requires the conversion just described, the database engine will raise an error and perform a rollbackthe behavior that is expected of most other database engines. There is also no affinity mode, which causes SQLite 3 to handle data types in the same manner as SQLite 2.

User-Defined Collating Sequences

A collating sequence is a definition of the way in which text strings are ordered. SQLite 3.0 allows you to implement user-defined collating sequences as functions, which can then be used to determine the sort order within SQLite.

A collating sequence function takes two input arguments and compares them in some custom way. A negative, zero, or positive return value indicates, respectively, that the first value was less than, equal to, or greater than the second.

The COLLATE clause in SQL can appear in the CREATE TABLE statement to define the default collating sequence for a column or in the ORDER BY clause of a SELECT statement.

Improved Concurrency

The new version of SQLite supports better concurrent use through a more complex locking and journaling scheme that allows reader and writer processes to coexist in the same database. The new mechanism also adds the ability to perform atomic transactions across different attached database files.

Rather than being simply locked when a write operation takes place, the database file can now be in any of the following locking states:

  • UNLOCKED

  • SHARED

  • RESERVED

  • PENDING

  • EXCLUSIVE

The UNLOCKED state means that there are no locks on the database and therefore no read or write operations may be performed.

Many simultaneous reader processes can all hold SHARED locks on the same database; however, no write operation can take place when there is a SHARED lock.

When a process is planning to perform a write, it locks the file as RESERVED. There can be only one RESERVED lock; however, it can coexist with SHARED locks and new SHARED locks can even be acquired while the database is RESERVED. The implication of this is that a writer process can coexist with a readersomething that SQLite 2 was not capable of.

The PENDING lock means that a process needs to write to the database as soon as possible. It will already have requested a RESERVED lock and will be waiting for any SHARED locks to clear before it can perform its write. When a PENDING lock is active, no further SHARED locks can be acquired by other processes. This implements a priority system that ensures new reader processes do not lock the database when a writer is waiting, whereas in SQLite 2 it was possible that a constant stream of reader processes could prevent any writes from taking place.

The only way a write operation can take place is through an EXCLUSIVE lock on the database file. Equivalent to the writer lock in SQLite 2, no other locks can coexist with an EXCLUSIVE. Through the use of RESERVED and PENDING locks, SQLite keeps the amount of time for which an EXCLUSIVE lock is held to a minimum.


Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]


SYMBOL

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

' (apostrophe)
(backslash)
(quotes)
.databases command (sqlite) 2nd
.dump command (sqlite) 2nd 3rd 4th
.echo on|off command (sqlite)
.exit command (sqlite)
.explain on|off command (sqlite)
.header command (sqlite) 2nd
.headers command (sqlite)
.help command (sqlite) 2nd
.indices command (sqlite)
.mode columns command (sqlite)
.mode command (sqlite)
.mode html command (sqlite)
.mode insert command (sqlite)
.mode lines command (sqlite)
.mode list command (sqlite)
.NET
.nullvalue command (sqlite)
.output command (sqlite)
.prompt command (sqlite)
.quit command (sqlite)
.read command (sqlite) 2nd 3rd 4th
.schema command (sqlite) 2nd
.separator command (sqlite) 2nd
.show command (sqlite) 2nd
.tables command (sqlite)
.timeout command (sqlite)
.width command (sqlite) 2nd
|| (concatenation operator)


A

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

ABORT algorithm
ad-hoc file storage
Add opcode
addslashes() function
administration
backing up databases 2nd
.dump command 2nd 3rd
database analysis 2nd 3rd
database information, returning 2nd 3rd 4th
database parameters
changing for current session 2nd 3rd
changing permanently 2nd
query parameters
editing 2nd 3rd 4th
aggregate functions 2nd 3rd 4th 5th
creating
cx.create_aggregate() command 2nd 3rd 4th 5th
aggregating functions
creating 2nd 3rd 4th 5th 6th 7th 8th
creating with Perl DBI 2nd 3rd 4th 5th 6th
algorithms (conflict resolution)
ABORT
FAIL
IGNORE
REPLACE 2nd
ROLLBACK
aliases
column aliases 2nd
altcaps() function
ALTER TABLE statement
altering tables 2nd
analyzing
databases 2nd 3rd
AND operator
ANSI SQL92 unsupported features 2nd
apostrophe (')
arbtrary data
referencing
architecture
VDBE (Virtual Database Engine)
B-tree
back end
code generators
interface
OS interfaces
pagers
parsers
red/black tree
tokenizers
virtual machines
arithmetic operators 2nd 3rd
arrays
returning
returning result sets as 2nd 3rd
ATTACH DATABASE statement 2nd 3rd 4th 5th
attaching to databases 2nd 3rd 4th 5th 6th 7th 8th
autocommit argument (sqlite.connect() command)
available_drivers() function 2nd
avg() function
avgFinalize() function
avverages
calculating 2nd 3rd


B

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

B-tree
back end (VDBE)
background jobs
backing up databases 2nd
.dump command 2nd 3rd
backslash (
BEGIN TRANSACTION statement 2nd 3rd
begin_work() function 2nd
beginning
transactions
begin_work() function 2nd
benchmark.sh script 2nd
benchmarking 2nd 3rd
BETWEEN operator
binary data 2nd
encoding/decoding 2nd 3rd
UDFs (user-defined functions) 2nd
binary installation for Linux 2nd
binary installation for Windows 2nd
bind values
Perl DBI 2nd 3rd 4th 5th 6th 7th 8th 9th
bind_param() function 2nd 3rd
BLOB data type
BLOBs 2nd
browsing records
busy method


C

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

C/C++ 2nd 3rd
core API 2nd 3rd
databases
closing 2nd 3rd 4th 5th 6th 7th
finding information about database changes
inserting data into 2nd 3rd 4th
locked files 2nd
opening 2nd 3rd 4th 5th 6th 7th
updating 2nd
error codes 2nd 3rd 4th
functions
aggregating functions 2nd 3rd
avgFinalize()
callback() 2nd
creating 2nd 3rd 4th 5th
sqlite_aggregate_context()
sqlite_busy_handler()
sqlite_busy_handler[)
sqlite_busy_timeout[)
sqlite_changes() 2nd
sqlite_changes[)
sqlite_close() 2nd 3rd 4th
sqlite_compile() 2nd 3rd
sqlite_compile() function 2nd
sqlite_complete() 2nd
sqlite_complete[)
sqlite_create_aggregate()
sqlite_create_aggregate[)
sqlite_create_function() 2nd 3rd
sqlite_create_function[)
sqlite_encoding[) 2nd
sqlite_exec() 2nd 3rd 4th 5th 6th 7th
sqlite_exec_printf()
sqlite_finalize() 2nd 3rd 4th
sqlite_freemem()
sqlite_freemem[)
sqlite_get_table() 2nd 3rd
sqlite_get_table[)
sqlite_get_table_mprintf[)
sqlite_interrupt[)
sqlite_last_insert_rowid()
sqlite_last_insert_rowid[)
sqlite_mprintf()
sqlite_mprintf[)
sqlite_open() 2nd 3rd 4th
sqlite_progress_handler[)
sqlite_set_result_double()
sqlite_set_result_double[)
sqlite_set_result_error[)
sqlite_set_result_int[)
sqlite_set_result_string[)
sqlite_step() 2nd 3rd 4th 5th
sqlite_step() function
sqlite_user_data[)
sqlite_version[) 2nd
sqliteAtoF()
sumFinalize()
sumStep()
jobs
performing in background
memory management
non-callaback AP 2nd
preparing to use 2nd
queries
executing 2nd
result sets
returning 2nd 3rd
return codes 2nd
SQL statements
checking
executing with callback functions 2nd 3rd 4th 5th 6th 7th
executing without callback functions 2nd 3rd 4th 5th 6th
interrupting
step-by-step execution
SQLite library
finding version of 2nd 3rd
SQLite virtual machines, creating 2nd
C/C++ interface
C/C++losing
databases
sqlite_close() function
C/C++ore C API 2nd
cache_size directive (PRAGMA command)
calcualting
median averages 2nd 3rd
callback functions 2nd 3rd
Callback opcode
callback() function 2nd
calling
functions
UDFs (user-defined functions) 2nd
Cartesian joins 2nd
case sensitivity 2nd
changes method 2nd
CHECK clause
CREATE TABLE statement
CHECK statement
classes
DatabaseError 2nd
DataError
Error 2nd
IntegrityError
InterfaceError
InternalError
NotSupportedError
OperationalError
PEAR 2nd 3rd
ProgrammingError
StandardError
clauses
CHECK
CREATE TABLE statement
CONSTRAINT
CREATE TABLE statement 2nd
GROUP BY 2nd 3rd 4th
HAVING 2nd
LIMIT 2nd
ON CONFLICT
CREATE TABLE statement 2nd
ORDER BY 2nd 3rd
USING DELIMITERS
WHEN
CREATE TRIGGER statement
WHERE
DELETE statement
UPDATE statement
client/server applications 2nd
close method
closing
databases
cx.close() command 2nd
dbcmd close command
disconnect() function 2nd
sqlite_close() function 2nd 3rd 4th
Tcl interface 2nd
sqlite
code generators
code listings
benchmark.sh script 2nd
changes method
CREATE TABLE statement
executing with C/C++ 2nd
database timeouts
handling with Tcl interface
databases
adding data to 2nd
connecting with C/C++
opening
updating 2nd
functions
creating 2nd
index examples
joint1t2.sql
joint2t1.sql
t1word.sql
tl.sql
Perl DBI
aggregating functions, creating 2nd
available_drivers() function 2nd
bind values 2nd 3rd
database updates
databases, adding records to 2nd
databases, adding tables to
fetchrow_array() function
fetchrow_arrayref () function
quote() function
tracing
user-defined functions, creating 2nd
PRAGMA show_datatypes command
PySQLite
.execute() command
aggregate functions
database records, fetching 2nd
database records, inserting
database updates, checking
error handling
SQL statements, executing 2nd
user-defined functions, creating 2nd
user-defined functions, registering
queries
executing without callback functions 2nd
records
inserting with Tcl
result sets
accessing random rows in 2nd
displaying in HTML tables 2nd
displaying with callback functions 2nd
returning in single array 2nd
returning one at a time 2nd
result sets, displaying in HTML tables 2nd
rows
inserting with Tcl
SELECT queries
assigning output to variables
processing results of 2nd
special characters, escaping 2nd
SQL statements
CREATE TABLE
validating with Tcl interface 2nd
SQL statements, executing with Tcl
SQLite extension, checking for
SQLITE_BUSY Status 2nd
sqlite_busy_handler() function 2nd
tables
creating with ANSI data types
creating with made-up data types
creating without data types
UDFs (user-defined functions)
aggregating functions 2nd
calling with php() 2nd
creating 2nd
user-defined functions
registering with Tcl interface 2nd
vegetables.csv
virtual machine program 2nd
COLLATE statement
collating sequences
SQLite version 3.0 2nd
Column opcode
columns
aliases 2nd
binary data 2nd
BLOBs 2nd
constraints 2nd
dates 2nd
DEFAULT
defining 2nd
INTEGER PRIMARY KEY columns 2nd
multiple columns
indexing 2nd
NOT NULL 2nd
PRIMARY KEY
returning information about 2nd
UNIQUE
command_logfile argument (sqlite.connect() command)
commands [See names of specific commands] [See names of specific commands] [See statements]
comments
COMMIT TRANSACTION statement 2nd 3rd
commit() function 2nd
commit() method
committing
transactions
commit() function 2nd
comparing strings 2nd 3rd
compiling
PySQLite
complete method 2nd
concatenation operator
concurrency
high concurrency 2nd
SQLite version 3.0 2nd
configuration
PHP 2nd
Linux/Unix configuration 2nd
SQLite extension, returning information about 2nd
SQLite support, checking for 2nd
Windows configuration 2nd
configure command
confirming
database changes 2nd 3rd
conflict resolution 2nd 3rd 4th
connect() function 2nd 3rd 4th 5th 6th 7th
connecting to databases 2nd
connection parameters
PySQLite 2nd
constants
SQLITE_ASSOC
SQLITE_BOTH
SQLITE_NUM
CONSTRAINT clause
CREATE TABLE statement 2nd
constraints (column) 2nd
convenience functions 2nd 3rd
converters argument (sqlite.connect() command)
converting
time zones
Coordinated Universal Time (UTC)
COPY statement 2nd 3rd
copying
tables
core C API
coreutils package
count() function 2nd 3rd
count_changes directive (PRAGMA command) 2nd
CREATE INDEX statement 2nd 3rd
CREATE TABLE statement
executing 2nd
executing with C/C++ 2nd
sample database 2nd 3rd 4th 5th
CREATE TRIGGER statement 2nd 3rd 4th
CREATE UNIQUE INDEX statement 2nd 3rd 4th
CREATE VIEW statement 2nd
create_aggregate() function 2nd 3rd 4th 5th
create_function() function 2nd 3rd 4th
cu.execute() command 2nd 3rd 4th 5th
cu.fetchall() command 2nd
cu.fetchmany() command 2nd 3rd 4th
cu.fetchone() command 2nd 3rd
cu.rowcount() command 2nd 3rd
cu.rownumber() command
custom functions
registering
dbcmd function command
customization 2nd 3rd [See also user-defined functions]
cx = sqlite.connect() command 2nd 3rd 4th 5th
cx.close() command 2nd
cx.commit() command 2nd
cx.create_aggregate() command 2nd 3rd 4th 5th 6th
cx.create_function() command 2nd 3rd
cx.cursor() command 2nd
cx.rollback() command


D

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

data safety
Perl DBI
Data Source Name (DSN)
data types 2nd 3rd
manifest typing
mapping
PySQLite 2nd 3rd 4th 5th
SQLite version 3.0 2nd 3rd
Database Browser 2nd 3rd 4th
database-driven websites
database_list directive (PRAGMA command)
DatabaseError class 2nd
databases 2nd 3rd 4th 5th [See also queries] [See also queries]
adding data to 2nd 3rd 4th 5th 6th
SELECT query results 2nd
single rows 2nd
analyzing 2nd 3rd
attaching to 2nd 3rd 4th 5th 6th 7th 8th
backing up 2nd
.dump command 2nd 3rd
checking changes to
Tcl interface 2nd
checking updates to
PySQLite interface 2nd 3rd
closing
cx.close() command 2nd
dbcmd close command
disconnect() function 2nd
sqlite_close() function 2nd 3rd 4th 5th
Tcl interface 2nd
columns
aliases 2nd
constraints 2nd
DEFAULT
defining 2nd
NOT NULL 2nd
PRIMARY KEY
UNIQUE
confirming changes to 2nd 3rd
conflict resolution 2nd
connecting to 2nd
creating 2nd
dates/times 2nd
date() function
date/time conversion specifiers
datetime() function
displaying 2nd
jullianday() function
modifiers 2nd 3rd
strftime() function 2nd
time string formats 2nd
time zones 2nd
time() function
deleting records from
Perl DBI 2nd 3rd
detaching
detaching from
fetching records from
Perl DBI 2nd 3rd 4th 5th
PySQLite interface 2nd 3rd 4th 5th
file permissions 2nd
finding information about database changes
indexes 2nd
benchmarking 2nd 3rd
benefits of 2nd 3rd 4th
creating 2nd 3rd 4th
dropping 2nd
examples 2nd 3rd 4th 5th 6th
multiple columns 2nd
sort order
table lists 2nd
unique indexes 2nd 3rd 4th
when to create 2nd
when to use 2nd 3rd
inserting data into
C/C++ interface 2nd 3rd 4th
inserting records into
PySQLite interface 2nd 3rd 4th 5th 6th
joins 2nd 3rd 4th 5th
Cartesian joins 2nd
left joins 2nd 3rd
locked databases
handling with Tcl interface 2nd
locked files 2nd 3rd 4th
locking 2nd 3rd 4th 5th
SQLite version 3.0
multithreaded database access 2nd
nested subqueries 2nd
opening
connect() function 2nd 3rd 4th 5th 6th 7th
cx = sqlite.connect() command 2nd 3rd 4th 5th
PEAR database class 2nd 3rd
sqlite dbcmd command
sqlite_open() function 2nd 3rd 4th 5th 6th 7th 8th 9th
Tcl interface 2nd
parameters
changing for current session 2nd 3rd
changing permanently 2nd
persistent connections
query parameters
editing 2nd 3rd 4th
querying with SELECT statement
aggregate functions 2nd 3rd 4th 5th
AND operator
arithmetic operators 2nd 3rd
BETWEEN operator
callback functions 2nd 3rd
column aliases 2nd
FROM clause 2nd
GROUP BY clause 2nd 3rd 4th
HAVING clause 2nd
IN operator
IS NOT NULL operator
IS NULL operator
joins 2nd 3rd 4th 5th
LEFT JOIN operator 2nd 3rd
LIKE operator 2nd
LIMIT clause 2nd
NOT LIKE operator
NULL values 2nd
OR operator
ORDER BY clause 2nd 3rd
string comparisons 2nd 3rd
string operators 2nd
syntax 2nd 3rd 4th
WHERE clause 2nd 3rd 4th 5th 6th
records
adding with Tcl interface 2nd 3rd 4th 5th
deleting 2nd 3rd 4th
fetching with Tcl interface 2nd 3rd
updating 2nd 3rd 4th
returning information about 2nd 3rd 4th
rows
inserting with Tcl interface 2nd
schemas
returning information about 2nd
SQLite Database Browser 2nd 3rd 4th
table schemas, viewing 2nd
tables
adding
altering 2nd
binary data 2nd
BLOBs 2nd
browsing
conflict resolution 2nd
copying
creating 2nd
creating with Perl DBI 2nd
data types 2nd 3rd
dates 2nd
dropping 2nd 3rd
full table scan
INTEGER PRIMARY KEY columns 2nd
searching
sqlite_master 2nd 3rd
temporary tables 2nd
time-tracking sample database
adding data to 2nd 3rd 4th 5th
binary data 2nd
BLOBs 2nd
data types 2nd 3rd
dates 2nd
INTEGER PRIMARY KEY columns 2nd
tables 2nd 3rd
timeouts 2nd 3rd 4th 5th
setting
Tcl interface 2nd
transactions 2nd 3rd 4th 5th 6th 7th 8th 9th
nesting
PySQLite interface 2nd
triggers
creating 2nd 3rd 4th
defined
dropping 2nd 3rd
example 2nd
interrupting 2nd
limitations
triggers on views 2nd
updating 2nd
C/C++ interface 2nd
Perl DBI 2nd 3rd
VDBE (Virtual Database Engine)
B-tree
back end
code generators
interface
opcodes 2nd 3rd 4th 5th 6th 7th
OS interfaces
pagers
parsers
red/black tree
tokenizer
virtual machines
views
creating 2nd 3rd
dropping 2nd
example 2nd 3rd
DataError class
datasets [See result sets]
date() function
dates 2nd
dates/times 2nd
date() function
date/time conversion specifiers
datetime() function
displaying 2nd
jullianday() function
modifiers 2nd 3rd
strftime() function 2nd
time string formats 2nd
time zones 2nd
time() function
datetime() function
dbcmd busy callback command
dbcmd changes command 2nd
dbcmd close command
dbcmd complete command
dbcmd errorcode command
dbcmd eval query command 2nd 3rd 4th
dbcmd function command
dbcmd onecolumn command
dbcmd timeout command
decoding
binary data 2nd 3rd
DEFAULT columns
default directive (PRAGMA command) 2nd
DEFAULT statement
defining
table columns 2nd
DELETE statement 2nd 3rd 4th
deleting
records 2nd 3rd 4th
Perl DBI 2nd 3rd
DETACH DATABASE statement 2nd
detaching databases
detaching from databases
die() function 2nd
directives
sqlite.assoc_case directive
directives (PRAGMA command)
cache_size
count_changes 2nd
database_list
default 2nd
full_column_names
index_info 2nd
index_list 2nd
integrity_check
parser_trace
show_datatypes
synchronous 2nd
table_info
temp_store
vdbe_trace 2nd
disconnect() function 2nd
displaying
dates/times 2nd
result sets in HTML tables 2nd 3rd 4th 5th
DISTINCT statement
dl() function
do() function
documentation
PEAR class
PHP 2nd
downloading
PySQLite
SQLite 2nd
Tcl interface
drivers
Perl DBI 2nd
DROP INDEX statement
DROP TABLE statement
DROP TRIGGER statement 2nd
DROP VIEW statement 2nd
dropping
indexes 2nd
tables 2nd
triggers 2nd 3rd
views 2nd
DSN (Data Source Name)


E

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

editing
database parameters
for current session 2nd 3rd
permanently 2nd
query parameters 2nd 3rd 4th
embedded devices
encoding
binary data 2nd 3rd
encoding argument (sqlite.connect() command)
err() function
Error class 2nd
error handling
PySQLite 2nd 3rd 4th 5th 6th
error reporting
dbcmd errorcode command
Perl DBI 2nd 3rd 4th
errors
C error codes 2nd 3rd 4th
PHP error reporting 2nd 3rd
errstr() function
escaping special characters 2nd
eval method 2nd 3rd
events
triggers
creating 2nd
dropping
EXCEPT statement
exceptions
PrintError
RaiseError 2nd 3rd
EXCLUSIVE locking state
execute() function 2nd 3rd 4th 5th 6th 7th
executing
queries
dbcmd eval query command 2nd
sqlite_escape_string() function 2nd 3rd
sqlite_exec() function 2nd
sqlite_query() function 2nd 3rd
sqlite_unbuffered_query() function
queries with callback functions 2nd 3rd
sqlite_exec() function 2nd 3rd 4th
queries without callback functions 2nd 3rd 4th 5th 6th
SQL statements
step-by-step execution
SQL statements from files 2nd
SQL statements with Perl DBI 2nd 3rd 4th
SQL statements with PySQLite 2nd 3rd 4th 5th 6th 7th
SQL statements with SQLite Database Browser
SQL statements with Tcl interface 2nd
EXPLAIN command
EXPLAIN statement 2nd 3rd 4th 5th
extension_loaded() function


F

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

faggregate functions
creating
cx.create_aggregate() command
FAIL algorithm
fetchall_arrayref() function 2nd 3rd
fetchall_hashref() function 2nd
fetchrow_array() function 2nd 3rd 4th
fetchrow_arrayref() function 2nd
fetchrow_hash() function
fetchrow_hashref() function
fields
returning information about
files
executing SQL statements from 2nd
file permissions 2nd
file storage
loading data from 2nd
locked database files 2nd
reading SQL commands from 2nd 3rd
sending output to
SQLite version 3.0 file formats 2nd
finalize() function 2nd 3rd
finish() function 2nd
FOREIGN KEY statement
formatting
dates/times 2nd
time string formats 2nd
FROM clause 2nd
full table scans
full_column_names directive (PRAGMA command)
func() function
function method 2nd 3rd
functions 2nd [See also specific function names] [See also names of specific functions]
aggregate functions 2nd 3rd 4th 5th
aggregating functions
creating 2nd 3rd
creating with Perl DBI 2nd 3rd 4th 5th 6th
arbtrary data
referencing
callback functions 2nd 3rd
calling
creating 2nd 3rd 4th 5th
registering 2nd
return values, setting 2nd
UDFs (user-defined functions)
aggregating functions 2nd 3rd 4th 5th
binary data 2nd
calling 2nd
creating 2nd 3rd 4th
registering 2nd 3rd
user-defined functions
creating 2nd 3rd 4th 5th 6th 7th
creating with Perl DBI 2nd 3rd 4th 5th 6th
registering
registering with Tcl interface 2nd 3rd
future of SQLite [See SQLite version 3.0]


G

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

getAll() function
getRow() function
GMT (Greenwich Mean Time)
GRANT statement
GROUP BY clause 2nd 3rd 4th


H

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

Halt opcode
HAVING clause 2nd
help 2nd
high concurrency 2nd
high-volume websites 2nd
Hipp, D. Richard
Hipp, Wyrick & Company, Inc
HTML tables
displaying query results in 2nd 3rd 4th 5th


I

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

identifiers [See naming conventions]
IGNORE algorithm
IN operator
index info directive (PRAGMA command) 2nd
index list_directive (PRAGMA command) 2nd
indexes 2nd
benchmarking 2nd 3rd
benefits of 2nd 3rd 4th
creating 2nd 3rd 4th
dropping 2nd
examples 2nd 3rd 4th 5th 6th
multiple columns 2nd
sort order
table lists 2nd
unique indexes 2nd 3rd 4th
when to create 2nd
when to use 2nd 3rd
INSERT INTO statement 2nd
sample database 2nd 3rd 4th 5th
INSERT statement 2nd 3rd
VALUES keyword 2nd
installation
Perl DBI (Database Interface) 2nd 3rd
PySQLite 2nd
SQLite
binary installation for Linux 2nd
binary installation for Windows 2nd
installing from source code 2nd
RPM installation for Linux 2nd
installed_versions() function
INTEGER data type
Integer opcode
INTEGER PRIMARY KEY columns 2nd
integrity_check directive (PRAGMA command)
IntegrityError class
InterfaceError class
internal data manipulation
InternalError class
interrupting
SQL statements
triggers 2nd
INTERSECT statement
IS NOT NULL operator
IS NULL operator
isError() function


J

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

Java
jobs
background jobs
joins 2nd 3rd 4th 5th
Cartesian joins 2nd
left joins 2nd 3rd
jullianday() function


K

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

keywords [See specific keywords] [See statements]


L

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

last_insert_rowid command
last_insert_rowid() function 2nd
LEFT JOIN operator 2nd 3rd
left joins 2nd 3rd
length of names
length() function
libraries
finding version of 2nd
libsqlite.so
libsqlite3.so
sqlite.so.gz
sqlitedll.zip
tclsqlite.so.gz
tclsqlite.zip
library
finding version of 2nd
libsqlite.so library
libsqlite3.so library
libtclsqlite.so 2nd
LIKE operator 2nd
LIMIT clause 2nd
limitations of SQLite 2nd
limiting data 2nd
Linux
binary SQLite installation 2nd
PHP configuration 2nd
RPM SQLite installation 2nd
loading
data from files 2nd
Perl DBI (Database Interface) 2nd
locked database files 2nd 3rd 4th
locked databases
handling with Tcl interface 2nd
locking
databases
SQLite version 3.0
locking databases 2nd 3rd 4th 5th
lower() function


M

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

macros
THREADSAFE
mailing lists
make command
make install command
manifest typing
mapping
data types
PySQLite 2nd 3rd 4th 5th
max() function
median averages, calculating 2nd 3rd
median() function 2nd 3rd 4th 5th 6th 7th 8th 9th
memory
management
min() function
mode argument (sqlite.connect() command)
modifiers
dates/time modifiers 2nd 3rd
modules
Perl DBI (Database Interface)
multiple columns
indexing 2nd
multithreaded database access 2nd


N

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

naming conventions
case sensitivity 2nd
name length
reserved keywords 2nd
SQL 2nd
SQLite version 3.0 2nd
valid characters 2nd
nested subqueries 2nd
nesting
transactions
networks 2nd
new() function 2nd 3rd
Next opcode
non-callback API (C) 2nd
normal keywords (SQL) 2nd
NOT LIKE operator
NOT NULL columns 2nd
NOT NULL statement
NotSupportedError class
NULL values 2nd


O

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

objects
BLOBs 2nd
ON CONFLICT clause
CREATE TABLE statement 2nd
ON-CONFLICT clause 2nd
onecolumn method
opcodes (VDBE) 2nd 3rd 4th 5th 6th 7th
opening
databases
connect() function 2nd 3rd 4th 5th 6th 7th
cx = sqlite.connect() command 2nd 3rd 4th 5th
PEAR database class 2nd 3rd
persistent connections
sqlite dbcmd command
sqlite_open() function 2nd 3rd 4th 5th 6th 7th 8th 9th
Tcl interface 2nd
OpenRead opcode
OperationalError class
operators
AND
arithmetic operators 2nd 3rd
BETWEEN
concatengation
IN
IS NOT NULL
IS NULL
joins 2nd 3rd 4th 5th
LEFT JOIN 2nd 3rd
LIKE 2nd 3rd
OR
relational operators
string operators 2nd
OR clause 2nd
OR operator
ORDER BY clause 2nd 3rd
ordering data 2nd 3rd
OS interfaces


P

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

P1 opcode
packages
coreutils
sqlite-2.8.15-1.i386.rpm
sqlite-2.8.15.bin.gz
sqlite-2.8.15.so.gz
sqlite-2.8.15.tar.gz
sqlite-2_8_15.zip
sqlite-devel
sqlite-devel-2.8.15-1.i386.rpm
sqlitedll-2_8_15.zip
tclsqlite-2.8.15.so.gz
tclsqlite-2_8_15.zip
tclsqlite.so.gz
tclsqlite.zip
pagers
parameters
database parameters
changing for current session 2nd 3rd
changing permanently 2nd
query parameters
editing 2nd 3rd 4th
parser_trace directive (PRAGMA command)
parsers
PEAR class 2nd 3rd
pear tool 2nd
PENDING locking state
performance optimization
EXPLAIN statement 2nd 3rd 4th 5th
indexes 2nd
benchmarking 2nd 3rd
benefits of 2nd 3rd 4th
creating 2nd 3rd 4th
dropping 2nd
examples 2nd 3rd 4th 5th 6th 7th
multiple columns 2nd
table lists 2nd
unique indexes 2nd 3rd 4th
when to create 2nd
when to use 2nd 3rd
PRAGMA statement 2nd
transactions 2nd
VACUUM statement 2nd
Perl DBI
aggregating creating 2nd 3rd 4th 5th 6th
user-defined functions, creating 2nd 3rd 4th 5th 6th
Perl DBI (Database Interface) 2nd 3rd
available drivers 2nd
bind values 2nd 3rd 4th 5th 6th 7th 8th 9th
data safety
databases
adding tables to 2nd
closing 2nd
deleting records from 2nd 3rd
fetching records from 2nd 3rd 4th 5th
opening 2nd 3rd 4th 5th 6th 7th
updating 2nd 3rd
error reporting 2nd 3rd 4th
functions
available_drivers() 2nd
begin_work() 2nd
bind_param() 2nd 3rd
commit() 2nd
connect() 2nd 3rd 4th 5th 6th 7th
create_aggregate() 2nd 3rd 4th 5th
create_function() 2nd 3rd 4th
die() 2nd
disconnect() 2nd
do()
err()
errstr()
execute() 2nd 3rd 4th 5th 6th 7th
fetchall_arrayref() 2nd 3rd
fetchall_hashref() 2nd
fetchrow_array() 2nd 3rd 4th
fetchrow_arrayref() 2nd
fetchrow_hash()
fetchrow_hashref()
finalize() 2nd 3rd
finish() 2nd
func()
installed_versions()
last_insert_rowid()
new() 2nd 3rd
prepare() 2nd 3rd 4th
quote() 2nd 3rd 4th
rollback() 2nd
rows()
step() 2nd 3rd
trace() 2nd 3rd 4th 5th
warn() 2nd
installing 2nd 3rd
loading 2nd
modules
queries
fetching rows from 2nd 3rd 4th 5th 6th
returning information about 2nd
SQL statements, executing 2nd 3rd 4th
tracing 2nd 3rd 4th
transactions 2nd 3rd 4th
version, finding
permissions
file permissions 2nd
persistent connections
PHP 2nd
configuring for Linux/Unix 2nd
configuring for Windows 2nd
constants 2nd
databases
closing 2nd
confirming changes to 2nd 3rd
opening 2nd 3rd 4th 5th
persistent connections
documentation 2nd
error reporting 2nd 3rd
functions
addslashes()
date()
datetime()
dl()
extension_loaded()
getAll()
getRow()
isError()
jullianday()
last_insert_rowid()
length()
lower()
median() 2nd 3rd 4th
php() 2nd
phpinfo()
query()
sqlite_array_query() 2nd 3rd 4th
sqlite_busy_timeout()
sqlite_changes() 2nd 3rd 4th
sqlite_close() 2nd
sqlite_column() 2nd
sqlite_create_aggregate() 2nd 3rd 4th 5th
sqlite_create_function() 2nd 3rd 4th 5th 6th
sqlite_current() 2nd
sqlite_decode_binary()
sqlite_encode_binary()
sqlite_error_string() 2nd 3rd
sqlite_escape_string() 2nd 3rd
sqlite_fetch_array() 2nd 3rd
sqlite_fetch_single() 2nd 3rd
sqlite_field_name() 2nd
sqlite_has_more() 2nd
sqlite_last_error() 2nd 3rd
sqlite_last_insert_rowid() 2nd
sqlite_libencoding() 2nd
sqlite_libversion() 2nd
sqlite_next() 2nd 3rd
sqlite_num_fields() 2nd
sqlite_num_rows() 2nd
sqlite_open() 2nd 3rd 4th
sqlite_popen() 2nd 3rd 4th
sqlite_query() 2nd 3rd 4th 5th
sqlite_rewind() 2nd
sqlite_seek() 2nd 3rd
sqlite_udf_decode_binary() 2nd
sqlite_udf_encode_binary() 2nd
sqlite_unbuffered_query() 2nd
strftime() 2nd
substr()
time()
ucwords()
upper()
word_reverse() 2nd
PEAR database class 2nd 3rd
queries
executing 2nd 3rd 4th 5th
returning information about 2nd 3rd 4th
result sets
accessing random rows 2nd
displaying in HTML tables 2nd 3rd 4th 5th
processing 2nd 3rd 4th 5th 6th 7th 8th
returning in single array 2nd 3rd
runtime configuration 2nd
safe mode
special characters, escaping 2nd
SQLite extension, returning information about 2nd
SQLite support, checking for 2nd
suPHP
PHP interface 2nd
php() function 2nd
php.ini
directives
sqlite.assoc_case
phpinfo() function
portability of SQLite 2nd
ppm.bat script
PRAGMA command
database analysis 2nd 3rd
database information, returning 2nd 3rd 4th
database parameters
changing for current session 2nd 3rd
changing permanently 2nd
directives
cache_size
count_changes 2nd
database_list
default 2nd
full_column_names
index_info 2nd
index_list 2nd
integrity_check
parser_trace
show_datatypes
synchronous 2nd
table_info
temp_store
vdbe_trace 2nd
query parameters
editing 2nd 3rd 4th
PRAGMA statement 2nd
prepare() function 2nd 3rd 4th
PRIMARY KEY columns
PRIMARY KEY statement
PrintError exception
printf() (_printf()) wrapper functions 2nd
ProgrammingError class
prompts
changing
PySQLite 2nd 3rd
aggregating functions
creating 2nd 3rd 4th 5th
commands
cu.execute() 2nd 3rd 4th 5th
cu.fetchall() 2nd
cu.fetchmany() 2nd 3rd 4th
cu.fetchone() 2nd 3rd
cu.rowcount() 2nd 3rd
cu.rownumber()
cx = sqlite.connect() 2nd 3rd 4th 5th
cx.close() 2nd
cx.commit() 2nd
cx.create_aggregate() 2nd 3rd 4th 5th 6th
cx.create_function() 2nd 3rd
cx.cursor() 2nd
cx.rollback()
compiling
connection parameters 2nd
data type mappings 2nd 3rd 4th 5th
databases
checking updates to 2nd 3rd
closing 2nd 3rd 4th 5th
fetching records from 2nd 3rd 4th 5th
inserting records into 2nd 3rd 4th 5th 6th
opening 2nd 3rd 4th 5th
downloading
error handling 2nd 3rd 4th 5th 6th
installing 2nd
preparing to use
SQL statements
executing 2nd 3rd 4th 5th
modifying query values 2nd
transactions 2nd
user-defined functions
creating 2nd 3rd 4th 5th 6th 7th
Python interface [See PySQLite] [See PySQLite]


Q

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

queries [See also result sets]
DELETE 2nd 3rd
executing
dbcmd eval query command 2nd
sqlite_escape_string() function 2nd 3rd
sqlite_exec() function 2nd
sqlite_query() function 2nd 3rd
sqlite_unbuffered_query() function
executing with callback functions 2nd 3rd
sqlite_exec() function 2nd 3rd 4th
executing without callback functions 2nd 3rd 4th 5th 6th
finding information about
dbcmd changes command 2nd
INSERT statement 2nd
joins 2nd 3rd 4th 5th
nested subqueries 2nd
optimization
EXPLAIN statement 2nd 3rd 4th
indexes 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
PRAGMA statement 2nd
transactions 2nd
VACUUM statement 2nd
result sets
accessing random rows 2nd
displaying in HTML tables 2nd 3rd 4th 5th
returning in single array 2nd 3rd
returning information about 2nd 3rd 4th
last_insert_rowid() method
rows() method
SELECT
aggregate functions 2nd 3rd 4th 5th
AND operator
arithmetic operators 2nd 3rd
BETWEEN operator
callback functions 2nd 3rd
column aliases 2nd
FROM clause 2nd
GROUP BY clause 2nd 3rd 4th
HAVING clause 2nd
IN operator
IS NOT NULL operator
IS NULL operator
joins 2nd 3rd 4th 5th
LEFT JOIN operator 2nd 3rd
LIKE operator 2nd
LIMIT clause 2nd
NOT LIKE operator
NULL values 2nd
OR operator
ORDER BY clause 2nd 3rd
string comparisons 2nd 3rd
string operators 2nd
syntax 2nd 3rd 4th
WHERE clause 2nd 3rd 4th 5th 6th
SELECT statement
output, assigning to variables
processing results of 2nd
UPDATE 2nd 3rd 4th
variable subqueries
query() function
quote() function 2nd 3rd 4th
quotes ( )


R

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

RAISE() function 2nd
RaiseError exception 2nd 3rd
rand() function
random access data functions 2nd
random() function
raw_input() function
reading
SQL commands from files 2nd 3rd
records
adding with Tcl interface 2nd 3rd 4th 5th
browsing
deleting 2nd 3rd 4th
Perl DBI 2nd 3rd
fetching
Perl DBI 2nd 3rd 4th 5th
fetching with Tcl interface 2nd 3rd
inserting
C/C++ interface 2nd 3rd 4th
PySQLite interface 2nd 3rd 4th 5th 6th
returning
PySQLite interface 2nd 3rd 4th 5th
updating 2nd 3rd 4th
C/C++ interface 2nd
red/black tree
referencing
arbitrary data
registering
custom functions
functions 2nd
UDFs (user-defined functions) 2nd 3rd
user-defined functions
cx.create_function() command 2nd 3rd 4th 5th 6th
Tcl interface 2nd 3rd
relational operators
REPLACE algorithm 2nd
reporting errors [See error reporting] [See error reporting]
C error codes 2nd 3rd 4th
reporting errors (PHP) 2nd 3rd
reserved keywords [See specific keywords]
reserved keywords (SQL)
RESERVED locking state
reset() function
resolving conflicts 2nd
result sets
displaying with callback functions 2nd
processing 2nd 3rd 4th 5th 6th 7th 8th
returning
sqlite_get_table() function 2nd 3rd
returning one at a time 2nd
return codes 2nd
return values
setting 2nd
reverse method
reverse_string method
REVOKE statement
Rewind opcode
RIGHT OUTER JOIN statement
ROLLBACK algorithm
ROLLBACK TRANSACTION statement 2nd 3rd
rollback() function 2nd
rolling back transactions
rollback() function 2nd
rows
accessing randomly 2nd
inserting
C/C++ interface 2nd 3rd 4th
inserting with Tcl interface 2nd
returning information about
rows() function
RPM installation for Linux 2nd
Ruby


S

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

safe mode
scalability of SQLite 2nd
schemas
returning information about 2nd
viewing 2nd
scripts
benchmark.sh 2nd
ppm.bat
searching
tables
SQLite Database Browser
security 2nd
database locking 2nd 3rd 4th 5th
file permissions 2nd
multithreaded database access 2nd
safe mode
timeouts 2nd 3rd 4th 5th
SELECT statement 2nd
aggregate functions 2nd 3rd 4th 5th
AND operator
arithmetic operators 2nd 3rd
BETWEEN operator
callback functions 2nd 3rd
column aliases 2nd
FROM clause 2nd
GROUP BY clause 2nd 3rd 4th
HAVING clause 2nd
IN operator
IS NOT NULL operator
IS NULL operator
joins 2nd 3rd 4th 5th
LEFT JOIN operator 2nd 3rd
LIKE operator 2nd
LIMIT clause 2nd
NOT LIKE operator
NULL values 2nd
OR operator
ORDER BY clause 2nd 3rd
output, assigning to variables
processing results of
Tcl interface 2nd
string comparisons 2nd 3rd
string operators 2nd
syntax 2nd 3rd 4th
WHERE clause 2nd 3rd 4th 5th 6th
sending
output to files
seq command
SHARED locking state
show_datatypes directive (PRAGMA command)
SmallTalk
sort order
indexes
source code, installing SQLite from 2nd
special characters, escaping 2nd
speed of SQLite 2nd
SQL
commands
reading from files 2nd 3rd
functions [See also specific function names]
aggregating functions 2nd 3rd
arbitrary data, referencing 2nd
creating 2nd 3rd 4th 5th
registering 2nd
return values, setting 2nd
UDFs (user-defined functions) 2nd 3rd 4th 5th
aggregating functions 2nd 3rd 4th 5th
binary data 2nd
calling 2nd
creating 2nd 3rd 4th
SQL statements
ALTER TABLE
ANSI-92 unsupported features 2nd
ATTACH DATABASE 2nd 3rd 4th 5th
BEGIN TRANSACTION 2nd 3rd
case sensitivity
CHECK 2nd
checking
dbcmd complete command
COLLATE
comments
COMMIT TRANSACTION 2nd 3rd
COPY 2nd 3rd
CREATE INDEX 2nd 3rd
CREATE TABLE 2nd 3rd
CHECK clause
conflict resolution
CONSTRAINT clause 2nd
executing 2nd
executing with C/C++ 2nd
ON CONFLICT clause 2nd
CREATE TABLE statement
sample database 2nd 3rd 4th 5th
CREATE TEMPORARY TABLE 2nd
CREATE TRIGGER 2nd 3rd 4th
CREATE UNIQUE INDEX 2nd 3rd 4th
CREATE VIEW 2nd
DEFAULT
DELETE 2nd 3rd 4th
DETACH DATABASE 2nd
DISTINCT
DROP INDEX
DROP TABLE 2nd 3rd
DROP TRIGGER 2nd
DROP VIEW 2nd
EXCEPT
executing from files 2nd
executing with callback functions 2nd 3rd
sqlite_exec() function 2nd 3rd 4th
executing with Perl DBI 2nd 3rd 4th
executing with PySQLite 2nd 3rd 4th 5th 6th 7th
executing with SQLite Database Browser
executing with Tcl interface 2nd
executing without callback functions 2nd 3rd 4th 5th 6th
EXPLAIN 2nd 3rd 4th 5th
FOREIGN KEY
GRANT
INSERT 2nd 3rd
VALUES keyword 2nd
INSERT INTO 2nd
INSERT INTO statement
sample database 2nd 3rd 4th 5th
interrupting
INTERSECT
naming conventions 2nd
normal keywords 2nd
NOT NULL
ON-CONFLICT 2nd
OR 2nd
PRAGMA 2nd
PRIMARY KEY
reserved keywords
REVOKE
RIGHT OUTER JOIN
ROLLBACK TRANSACTION 2nd 3rd
SELECT 2nd
aggregate functions 2nd 3rd 4th 5th
AND operator
arithmetic operators 2nd 3rd
BETWEEN operator
callback functions 2nd 3rd
column aliases 2nd
FROM clause 2nd
GROUP BY clause 2nd 3rd 4th
HAVING clause 2nd
IN operator
IS NOT NULL operator
IS NULL operator
joins 2nd 3rd 4th 5th
LEFT JOIN operator 2nd 3rd
LIKE operator 2nd
LIMIT clause 2nd
NOT LIKE operator
NULL values 2nd
OR operator
ORDER BY clause 2nd 3rd
output, assigning to variables
processing results of 2nd
string comparisons 2nd 3rd
string operators 2nd
syntax 2nd 3rd 4th
WHERE clause 2nd 3rd 4th 5th 6th
SQL92 unsupported statements 2nd
step-by-step execution
system object names
UNION
UNION ALL
UNIQUE
UPDATE 2nd 3rd 4th 5th 6th
VACUUM 2nd
validating with Tcl interface 2nd 3rd
SQL92 unsupported features 2nd
SQLite
downloading 2nd
installation
binary installation for Linux 2nd
binary installation for Windows 2nd
installing from source code 2nd
RPM installation for Linux 2nd
version, finding
sqlite_version command
SQLite Database Browser 2nd 3rd 4th
sqlite dbcmd command
SQLite mailing list
sqlite tool 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
closing
commands
.databases 2nd
.dump 2nd 3rd 4th
.echo on|off
.exit
.explain on|off
.header 2nd
.headers
.help 2nd
.indices
.mode
.mode columns
.mode html
.mode insert
.mode lines
.mode list
.nullvalue
.output
.prompt
.quit
.read 2nd 3rd 4th
.schema 2nd
.separator 2nd
.show 2nd
.tables
.timeout
.width 2nd
obtaining list of
database schemas
returning information about 2nd
database timeouts
setting
output
sending to files
output format, changing 2nd 3rd 4th 5th
prompts
changing
SQL commands
reading from files 2nd 3rd
sqlite_master table 2nd 3rd
SQLite version 3.0 2nd
concurrency 2nd
data typing 2nd 3rd
file format changes 2nd
naming conventions 2nd
user-defined collating sequences 2nd
SQLite, overview of 2nd
customization 2nd
help and support 2nd
limitations 2nd
portability 2nd
scalability 2nd
security 2nd
speed 2nd
when to avoid 2nd
high concurrency 2nd
high-volume websites 2nd
network or client/server applications 2nd
when to use
ad-hoc file storage
database-driven websites
embedded devices
internal data manipulation
sqlite-2.8.15-1.i386.rpm package
sqlite-2.8.15.bin.gz package
sqlite-2.8.15.so.gz package
sqlite-2.8.15.tar.gz package
sqlite-2_8_15.zip file
sqlite-devel package
sqlite-devel-2.8.15-1.i386.rpm package
sqlite.assoc_case directive (php.ini)
sqlite.so.gz library
sqlite3_open() function
SQLITE_ABORT constant
SQLITE_ABORT error code
sqlite_aggregate_context() function
sqlite_array_query() function 2nd 3rd 4th
SQLITE_ASSOC constant
SQLITE_AUTH constant
SQLITE_AUTH error code
SQLITE_BOTH constant
SQLITE_BUSY constant
SQLITE_BUSY error code
SQLITE_BUSY Status 2nd
sqlite_busy_handler() function 2nd 3rd
sqlite_busy_handler[) function
sqlite_busy_timeout() function
sqlite_busy_timeout[) function
SQLITE_CANTOPEN constant
SQLITE_CANTOPEN error code
sqlite_changes() function 2nd 3rd 4th 5th 6th 7th
sqlite_changes[) function
sqlite_close() function 2nd 3rd 4th 5th 6th
sqlite_column() function 2nd
sqlite_compile() function 2nd 3rd 4th 5th
sqlite_complete() function 2nd
sqlite_complete[) function
SQLITE_CONSTRAINT constant
SQLITE_CONSTRAINT error code
SQLITE_CORRUPT constant
SQLITE_CORRUPT error code
sqlite_create_aggregate() function 2nd 3rd 4th 5th 6th
sqlite_create_aggregate[) function
sqlite_create_function() function 2nd 3rd 4th 5th 6th 7th 8th 9th
sqlite_create_function[) function
sqlite_current() function 2nd
sqlite_decode_binary() function
SQLITE_DONE constant
SQLITE_DONE return code
SQLITE_EMPTY constant
SQLITE_EMPTY error code
sqlite_encode_binary() function
sqlite_encoding[) function 2nd
SQLITE_ERROR constant
SQLITE_ERROR error code
sqlite_error_string() function 2nd 3rd
sqlite_escape_string () function 2nd 3rd
sqlite_exec() function 2nd 3rd 4th 5th 6th 7th
sqlite_exec_printf() function
sqlite_fetch_array() function 2nd 3rd
sqlite_fetch_single() function 2nd 3rd
sqlite_field_name() function 2nd
sqlite_finalize() function 2nd 3rd 4th
sqlite_freemem() function
sqlite_freemem[) function
SQLITE_FULL constant
SQLITE_FULL error code
sqlite_get_table() function 2nd 3rd
sqlite_get_table[) function
sqlite_get_table_mprintf[) function
sqlite_has_more() function 2nd
SQLITE_INERRUPT error code
SQLITE_INTERNAL constant
SQLITE_INTERNAL error code
SQLITE_INTERRUPT constant
sqlite_interrupt[) function
SQLITE_IOERR constant
SQLITE_IOERR error code
sqlite_last_error() function 2nd 3rd
sqlite_last_insert_rowid() function 2nd 3rd
sqlite_last_insert_rowid[) function
sqlite_libencoding() function 2nd
sqlite_libversion() function 2nd
SQLITE_LOCKED constant
SQLITE_LOCKED error code
sqlite_master table 2nd 3rd
SQLITE_MISMATCH constant
SQLITE_MISMATCH error code
SQLITE_MISUSE constant
SQLITE_MISUSE error code
sqlite_mprintf() function
sqlite_mprintf[) function
sqlite_next() function 2nd 3rd
SQLITE_NOLFS constant
SQLITE_NOLFS error code
SQLITE_NOMEM constant
SQLITE_NOMEM error code
SQLITE_NOTFOUND constant
SQLITE_NOTFOUND error code
SQLITE_NUM constant
sqlite_num_fields() function 2nd
sqlite_num_rows() function 2nd
SQLITE_OK constant
SQLITE_OK error code
sqlite_open() function 2nd 3rd 4th 5th 6th 7th 8th 9th
SQLITE_PERM constant
SQLITE_PERM error code
sqlite_popen() function 2nd 3rd 4th
sqlite_progress_handler[) function
SQLITE_PROTOCOL constant
SQLITE_PROTOCOL error code
sqlite_query() function 2nd 3rd 4th 5th
SQLITE_READONLY constant
SQLITE_READONLY error code
sqlite_rewind() function 2nd
SQLITE_ROW constant
SQLITE_ROW return code
SQLITE_SCHEMA constant
SQLITE_SCHEMA error code
sqlite_seek() function 2nd 3rd
sqlite_set_result_double() function
sqlite_set_result_double[) function
sqlite_set_result_error[) function
sqlite_set_result_int[) function
sqlite_set_result_string[) function
sqlite_step() function 2nd 3rd 4th 5th 6th
SQLITE_TOOBIG constant
SQLITE_TOOBIG error code
sqlite_udf_decode_binary() function 2nd
sqlite_udf_encode_binary() function 2nd
sqlite_unbuffered_query() function 2nd
sqlite_user_data[) function
sqlite_version command
sqlite_version[) function 2nd
sqliteAtoF() function
sqlitedll-2_8_15.zip file
sqlitedll.zip library
squite command
StandardError class
statements (SQL)
ALTER TABLE
ANSI-92 unsupported features 2nd
ATTACH DATABASE 2nd 3rd 4th 5th
BEGIN TRANSACTION 2nd 3rd
case sensitivity
CHECK 2nd
checking
COLLATE
comments
COMMIT TRANSACTION 2nd 3rd
COPY 2nd 3rd
CREATE INDEX 2nd 3rd
CREATE TABLE 2nd 3rd
CHECK clause
conflict resolution
CONSTRAINT clause 2nd
executing 2nd
executing with C/C++ 2nd
ON CONFLICT clause 2nd
sample database 2nd 3rd 4th 5th
CREATE TEMPORARY TABLE 2nd
CREATE TRIGGER 2nd 3rd 4th
CREATE UNIQUE INDEX 2nd 3rd 4th
CREATE VIEW 2nd
DEFAULT
DELETE 2nd 3rd 4th
DETACH DATABASE 2nd
DISTINCT
DROP INDEX
DROP TABLE 2nd 3rd
DROP TRIGGER 2nd
DROP VIEW 2nd
EXCEPT
executing from files 2nd
executing with callback functions 2nd 3rd
sqlite_exec() function 2nd 3rd 4th
executing with Perl DBI 2nd 3rd 4th
executing with PySQLite 2nd 3rd 4th 5th 6th 7th
executing with SQLite Database Browser
executing with Tcl interface 2nd
executing without callback functions 2nd 3rd 4th 5th 6th
EXPLAIN 2nd 3rd 4th 5th
FOREIGN KEY
GRANT
INSERT 2nd 3rd
VALUES keyword 2nd
INSERT INTO 2nd
sample database 2nd 3rd 4th 5th
interrupting
INTERSECT
naming conventions 2nd
normal keywords 2nd
NOT NULL
ON-CONFLICT 2nd
OR 2nd
PRAGMA 2nd
PRIMARY KEY
reserved keywords
REVOKE
RIGHT OUTER JOIN
ROLLBACK TRANSACTION 2nd 3rd
SELECT 2nd
aggregate functions 2nd 3rd 4th 5th
AND operator
arithmetic operators 2nd 3rd
BETWEEN operator
callback functions 2nd 3rd
column aliases 2nd
FROM clause 2nd
GROUP BY clause 2nd 3rd 4th
HAVING clause 2nd
IN operator
IS NOT NULL operator
IS NULL operator
joins 2nd 3rd 4th 5th
LEFT JOIN operator 2nd 3rd
LIKE operator 2nd
LIMIT clause 2nd
NOT LIKE operator
NULL values 2nd
OR operator
ORDER BY clause 2nd 3rd
output, assigning to variables
processing results of 2nd
string comparisons 2nd 3rd
string operators 2nd
syntax 2nd 3rd 4th
WHERE clause 2nd 3rd 4th 5th 6th
SQL92 unsupported statements 2nd
step-by-step executing
system object names
UNION
UNION ALL
UNIQUE
UPDATE 2nd 3rd 4th 5th 6th
VACUUM 2nd
validating with Tcl interface 2nd 3rd
step() function 2nd 3rd 4th
strftime() function 2nd
string map command
string operators 2nd
strings
comparing 2nd 3rd
operators 2nd
special characters, escaping 2nd
time string formats 2nd
StrNe opcode
su command
substr() function
sum() function
sumFinalize() function
sumStep() function
suPHP
support
synchronous directive (PRAGMA command) 2nd
system object names (SQL)


T

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

table_info directive (PRAGMA command)
tables
adding
adding data to 2nd 3rd 4th 5th 6th
SELECT query results 2nd
single rows 2nd
altering 2nd
binary data 2nd
BLOBs 2nd
browsing
columns
aliases 2nd
constraints 2nd
DEFAULT
defining 2nd
NOT NULL 2nd
PRIMARY KEY
UNIQUE
conflict resolution 2nd 3rd 4th
copying
creating 2nd 3rd
creating with Perl DBI 2nd
data types 2nd 3rd
dates 2nd
dropping 2nd 3rd
full table scan
INTEGER PRIMARY KEY columns 2nd
joins 2nd 3rd 4th 5th
Cartesian joins 2nd
left joins 2nd 3rd
records
adding with Tcl interface 2nd 3rd 4th 5th
fetching with Tcl interface 2nd 3rd
rows
inserting with Tcl interface 2nd
schemas
viewing 2nd
searching
SQLite Database Browser
sqlite_master 2nd 3rd
temporary tables 2nd
updating 2nd
Tcl
commands
dbcmd busy callback
dbcmd changes 2nd
dbcmd complete
dbcmd errorcode
dbcmd eval query 2nd
dbcmd function
dbcmd onecolumn
dbcmd timeout
sqlite close
sqlite dbcmd
sqlite_version
convenience functions
custom functions, registering
databases
opening/closing 2nd
error reporting
libtclsqlite.so library 2nd
locked database files, handling 2nd
queries
executing 2nd
finding information about 2nd
SQL statements
checking
SQLite version, finding
Tcl (Tool Command Language) interface 2nd 3rd
databases
adding records to 2nd 3rd 4th 5th
closing 2nd
fetching records from 2nd 3rd
opening 2nd
timeouts 2nd
downloading
methods
busy
changes 2nd
close
complete 2nd
eval 2nd 3rd
function 2nd 3rd
last_insert_rowid
onecolumn
reverse
reverse_string
string map
timeout 2nd
SQL statements, executing 2nd
SQL statements, validating 2nd 3rd
user-defined functions, registering 2nd 3rd
Tcl interface
tclsqlite-2.8.15.so.gz package
tclsqlite-2_8_15.zip file
tclsqlite.so.gz library
tclsqlite.so.gz package
tclsqlite.zip library
tclsqlite.zip package
temp_store directive (PRAGMA command)
temporary tables 2nd
text
strings
comparing 2nd 3rd
TEXT data type 2nd
THREADSAFE preprocessor macro
time zones 2nd
time() function
time-tracking sample database
adding data to 2nd 3rd 4th 5th
binary data 2nd
BLOBs 2nd
data types 2nd 3rd
dates 2nd
INTEGER PRIMARY KEY columns 2nd
tables 2nd 3rd
timeout argument (sqlite.connect() command)
timeout method 2nd
timeouts 2nd 3rd 4th 5th
setting
Tcl interface 2nd
times/dates 2nd
date() function
date/time conversion specifiers
datetime() function
displaying 2nd
jullianday() function
modifiers 2nd 3rd
strftime() function 2nd
time string formats 2nd
time zones 2nd
time() function
tokenizers
tools [See names of specific tools]
pear 2nd
trace() function 2nd 3rd 4th 5th
tracing
Perl DBI
Perl DBI (Database Interface) 2nd 3rd 4th
traffic
transactions 2nd 3rd 4th 5th 6th 7th 8th 9th
beginning
begin_work() function 2nd
committing
commit() function 2nd
nesting
PySQLite interface 2nd
rolling back
rollback() function 2nd
triggers
creating 2nd 3rd 4th
defined
dropping 2nd 3rd
example 2nd
interrupting 2nd
limitations
triggers on views 2nd
troubleshooting
SQLite error codes 2nd 3rd 4th
typeless data types 2nd 3rd
types 2nd 3rd


U

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

ucwords() function
UDFs (user-defined functions)
aggregating functions 2nd 3rd 4th 5th
binary data 2nd
calling 2nd
creating 2nd 3rd 4th
registering 2nd 3rd
dbcmd function command
UNION ALL statement
UNION statement
UNIQUE columns
unique indexes 2nd 3rd 4th
UNIQUE statement
Unix
PHP configuration 2nd
UNLOCKED state
UPDATE statement 2nd 3rd 4th 5th 6th
updates
verifying with PySQLite interface 2nd 3rd
updating
databases 2nd
C/C++ interface 2nd
Perl DBI 2nd 3rd
records 2nd 3rd 4th
upper() function
use DBI command 2nd
user-defined collating sequences
SQLite version 3.0 2nd
user-defined functions [See UDFs]
creating
cx.create_function() command 2nd 3rd 4th 5th 6th 7th
creating with Perl DBI 2nd 3rd 4th 5th 6th
registering
dbcmd function command
registering with Tcl interface 2nd 3rd
USING DELIMITERS clause
UTC (Coordinated Universal Time)


V

Team LiB

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

VACUUM statement 2nd
valid characters 2nd
validating
SQL statements
Tcl interface 2nd 3rd
VALUES keyword 2nd
variable subqueries
VDBE (Virtual Database Engine)
B-tree
back end
code generators
interface
opcodes 2nd 3rd 4th 5th 6th 7th
OS interfaces
pagers
parsers
red/black tree
tokenizers
virtual machines
vdbe_trace directive (PRAGMA command) 2nd
vegetables.csv
VerifyCookie opcode
version 3.0 (SQLite) 2nd
concurrency 2nd
data typing 2nd 3rd
file format changes 2nd
naming conventions 2nd
user-defined collating sequences 2nd
versions
Perl DBI
views
creating 2nd 3rd
dropping 2nd
example 2nd 3rd
triggers on views 2nd
Virtual Database Engine [See VDBE]
virtual machines
creating 2nd


W

Team LiB
Previous Section

Index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

warn() function 2nd
websites
database-driven websites
high-volume websites 2nd
WHEN clause
CREATE TRIGGER statement
WHERE clause 2nd 3rd 4th 5th 6th
DELETE statement
UPDATE statement
Windows
PHP configuration 2nd
SQLite installation 2nd
word_reverse() function 2nd

Team LiB
Previous Section