Data Warehousing > Database

Teradata Database Queue Tables

By:
David Craig

Teradata Database Queue Tables

Table of Contents

Executive Summary

Introduction

The Need for Queue Tables

What is a Queue Table?

Event Processing with Queue Tables

4-5

Competitive Advantage

Conclusion

References

EB-4310 > 1007 > PAGE 2 OF 6

Executive Summary

This white paper describes the Queue Tables feature

that was introduced in Teradata Database V2R6.
This feature enables new and improved ways of building
applications to support event processing use cases.

Teradata Database Queue Tables

Introduction

CREATE PROCEDURE anyevent ()

rules, analytic modeling, or XML shred-

BEGIN

ding. Instead of processing the incoming

A prerequisite to a successful implemen-

anyrows:

data immediately upon arrival, it needs to

tation of an active data warehouse in

LOOP

be buffered in a table so that the data

Teradata Database is the ability to kickoff

FOR checkevents AS checkcount

loading can be considered complete.

tactical event-driven business processes

CURSOR FOR

Before queue tables, the data were

and applications based on recent activity.

Queue tables enable new ways of building these event processing applications.

SELECT * FROM eventtab

DO
IF ACTIVITY_COUNT > 0 THEN

These provide significant performance

and usability improvements over capabilities in prior releases that were not
optimized for event processing.

LEAVE anyrows;

processed directly by a group of stored

procedures using a polling loop. The
stored procedures shared the work, i.e.,
each piece of data was given to one of the

END IF;

stored procedures for processing. As in

END FOR;

the Event Alert case, the polling loop is

END LOOP anyrows;

an inefficient mechanism to scan for

END;

incoming data.

Once an eventtab row is present, those

Message Processing

event data are returned to the external

When an airline needs to change a flight

application for further processing. The

schedule, different systems need to know

external application then proceeds back to

about it. The booking system handles

Event Alert

the anyevent polling loop to find more

rebooking transactions. Call centers work

When an event is detected by application

events. The polling loop and repeated

through a queue of customers who must

code running in the Teradata Database,

SELECT processing are an inefficient use

be called. Before queue tables, the input of

data are inserted into an event table (e.g.,

of CPU resources.

each of these systems would be modeled

The Need for Queue Tables

Here are some examples of architectural
approaches to event processing before
queue tables functionality was available:

by a trigger) for later processing. Before

queue tables, an external event processing
application pulled these events out of the
database using a polling loop in a stored
procedure. In the following example,
the anyevent stored procedure is an

as regular base tables. Messages in one

A retail inventory management system is an
example of an Event Alert application.
Point-of-sales data capture can automate the
process of recognizing low inventory level
events and replenishing retail store items.

table represent schedule changes. Stored

procedures reading that table could put
messages into another table for the
booking system and call centers. Call
center agents are assigned to customers

example of a polling loop, repeating a

Load with Asynchronous

from the call center table. Application

SELECT statement to determine if any

Processing

logic must ensure that messages are not

events are present.

A typical case is data being loaded into a

dropped or duplicated and are processed

table from an external source. Each piece

in order. Messages must be deleted after

of data may require a significant amount

theyre processed from the message queue.

of processing, e.g., complex logic, business

EB-4310 > 1007 > PAGE 3 OF 6

Teradata Database Queue Tables

What is a Queue Table?

> The FIFO pop operation is defined as

The Teradata Database maintains a

an SQL SELECT AND CONSUME

memory cache on every PE to track

Teradata Database is differentiated from

operation to retrieve a row from a

information about, and the FIFO order of,

competitive implementations by the

queue table and delete that selected

non-consumed rows for each queue table.

queue table functionality introduced in

row upon completion of the read. This

The cache is updated as rows are inserted,

the Teradata Database V2R6. It supports

is also referred to as consume mode.

updated, deleted, dropped, restored, or

the usual properties of a persistent

A consume mode request goes into a

consumed in the queue tables.

database table, and now also the charac-

delayed state when a SELECT AND

teristics that enable queue-oriented

CONSUME finds no rows in the queue

applications. Teradata Database queue

table. The request remains idle until an

tables are implemented as a Teradata

INSERT to that queue table awakens

extension to ANSI SQL at the database/

the request; that row is returned, and

table level. This provides greater flexibility,

then its deleted. Consumed rows are

power, and functionality, and leverages

returned in FIFO order.

the natural performance characteristic

of Teradata Database.

The default FIFO ordering of a queue

table may be altered before consumption

The SQL CREATE TABLE statement is

by altering the QITS. This is done with

extended to support queue tables. The

the SQL UPDATE, UPSERT form of

queue table definition is different from a

UPDATE, or MERGE statements. Rows

standard base table in that a queue table

may also be removed from a queue table

always contains a user-defined insertion

with the SQL DELETE.

timestamp (QITS) as the first column of

Event Processing with

Queue Tables
Now we look at event processing cases
explaining how queue tables are used to
simplify logic and improve performance
of event-driven applications.
Event Alerts are handled more efficiently
when using queue tables. A SELECT AND
CONSUME is used by the event processing
application to retrieve the event rows in
order. The oldest event row (based on the
QITS value) from the queue table is the
next row to be popped. If the event queue

the table. The QITS contains the time the

Since a Teradata Database queue table is

table is empty, the SELECT AND CON-

row was inserted into the queue table as

a persistent base table, the basic features

SUME request will wait for the table to be

the means for approximate FIFO ordering.

for transactions are provided. Queue table

populated. When data arrive in the queue

transactions are isolated, atomic, and

table, the SELECT AND CONSUME

Teradata Database queue tables support

recoverable. Consumed rows are rolled

statement resumes processing, returning

asynchronous first-in-first-out (FIFO)

back when a transaction containing a

the result row to the application that

push, pop, and peek queue operations:

SELECT AND CONSUME does not

processes the event. The event row that was

> The FIFO push operation is defined as

complete successfully.

retrieved is deleted from the queue table as

part of the SELECT AND CONSUME.

a SQL INSERT operation to store rows

into a queue table.
> The FIFO peek operation is defined as
an SQL SELECT operation to retrieve
rows from a queue table without
deleting them. This is also referred to
as browse mode.

EB-4310 > 1007 > PAGE 4 OF 6

Queue table operations may be triggered.

The triggered statements may access

The event detecting application populates

queue tables through SQL, an executed

the event queue table with a SQL INSERT

macro, or a stored procedure.

operation. A trigger may be used to push

rows into the queue table. The event

Teradata Database Queue Tables

processing application may repeat the

Controlling the number of asynchronous

messages from a queue table of customers

pop sequence with another SELECT

processing applications can smooth out

who need to be called. Call center agents

AND CONSUME statement to wait for

the resource consumption in response to

are assigned customers by consuming

more events.

varying insertion rates to the queue table.

from the call center queue table.

It is also possible to control the maximum

Queue tables solve the complex and
inefficient polling loop problem. A
repeated SELECT statement is no longer
needed. Numerous DML statements of
the anyevent stored procedure are
replaced with a single SQL SELECT AND

rate of resource consumption and the

maximum throughput rate. For example,
during a short time period with a high
inflow rate, the data are buffered in the
queue table and processed by the stored
procedures at their limited rate.

CONSUME statement.
SELECT AND CONSUME TOP 1 *
FROM eventtab;

Application logic is simplified since

messages, as queue table rows, are not
dropped, duplicated, or processed out of
order. Messages are automatically deleted
after they are consumed from each queue
table. The use of transactions when
accessing the queue tables allows isolation

Now lets see how message processing in

and atomic operations in the message

the airline booking system handles

processing sequence.

rebooking transactions when using queue

Competitive Advantage

Consider Load with Asynchronous

tables. Messages representing booking

Processing using queue tables. A load

changes are first pushed into

application buffers all incoming rows

a schedule change queue table using a

into a queue table with SQL INSERTs.

triggered INSERT. Stored procedures

Rows may arrive and be inserted at

waiting for a schedule change event to

different rates. An asynchronous process-

occur awaken from their SELECT AND

ing application pops the oldest data row

CONSUME on the schedule change queue

tables with the properties of an

using SELECT AND CONSUME and

table and INSERT messages into other

asynchronous FIFO queue.

calls a series of stored procedures to

queue tables for use for the booking

perform the analysis on the loaded data.

system or call centers. Data integrity is

SQL DDL. The CREATE TABLE syntax

The time it takes to analyze the queue

maintained when these operations are

is the only DDL extended to support

table data may be longer than the time it

processed in a single transaction. For

the queue tables feature.

takes to load one incoming row. Hence,

example, a row consumed from a schedule

rows may back up in the queue table

change queue table is rolled back when an

a Stored Procedure, or UDF SQL

before they are analyzed. Using a queue

INSERT to the booking or call center

interface. In Teradata Database, queue

table, SELECT AND CONSUME

queue tables fails.

tables are accessed with SQL DML.

solves the polling loop problem in the

asynchronous processing application.

The booking system pops messages from

a queue table of rebooking transactions
using SELECT AND CONSUME. Similarly, the call center application will pop

EB-4310 > 1007 > PAGE 5 OF 6

The queue table functionality in Teradata

Database differs from competitive
implementations in the following areas:
> Queue tables are persistent database

> Queue tables are administered with

> Competitive implementations provide

A new SELECT AND CONSUME statement provides the FIFO pop capability.

Teradata Database Queue Tables

Teradata.com

Conclusion

With queue tables, the Teradata Database

provides the capability to run real-time,

The new Teradata Database V2R6 Queue

event-driven active data warehouse

Tables feature simplifies the implementa-

activity while still having access to histori-

tion of asynchronous, event-driven

cal data providing the capability to make

applications. Performance is improved

more effective business decisions in real-

by saving CPU and network resources.

time processing environments.

Queue tables possess all the properties

David Craig is a senior member of the

of regular Teradata Database tables:

Teradata Corporations technical staff.

persistence, scalability, parallelism, ease of

He has been with Teradata since 1990.

use, reliability, availability, recovery, and

During this time, David has architected

security. In addition, queue tables possess

and developed such features as Queue

FIFO queue properties. These properties

Tables, Enhanced Cultural Data Type

allow queue tables to provide flexibility,

Formatting, and International Character

power, functionality, and leverage the

Set Support.

natural performance characteristic of the

Teradata Database.

This document, which includes the information contained herein, is the exclusive property of Teradata Corporation. Any person is hereby authorized to view, copy,
print, and distribute this document subject to the following conditions. This document may be used for non-commercial, informational purposes only and is provided
on an AS-IS basis. Any copy of this document or portion thereof must include this copyright notice and all other restrictive legends appearing in this document.
Note that any product, process or technology described in the document may be the subject of other intellectual property rights reserved by Teradata and are not
licensed hereunder. No license rights will be implied. Use, duplication or disclosure by the United States government is subject to the restrictions set forth in DFARS
252.227-7013 (c) (1) (ii) and FAR 52.227-19. Other brand and product names used herein are for identification purposes only and may be trademarks of their
respective companies.
Teradata continually improves products as new technologies and components become available. Teradata, therefore, reserves the right to change specifications
without prior notice. All features, functions, and operations described herein may not be marketed in all parts of the world. Consult your Teradata representative
or Teradata.com for more information.
Copyright 2004-2007 by Teradata Corporation

EB-4310 > 1007 > PAGE 6 OF 6

All Rights Reserved.

Produced in U.S.A.