Data Structures

You can use complex data structures such as dictionaries and lists in Couchbase. These data structures may be manipulated with basic operations without retrieving and storing the entire document.

Data structures in Couchbase are similar in concept to data structures in Python:

Map is like Python dict, and is a key-value structure, where a value is accessed by using a key string.
List is like a Python list and is a sequential data structure. Values can be placed in the beginning or end of a list, and can be accessed using numeric indexes.
Queue is a wrapper over a list which offers FIFO (first-in-first-out) semantics, allowing it to be used as a lightweight job queue.
Set is a wrapper over a list which provides the ability to handle unique values.

These data structures are stored as JSON documents in Couchbase, and can therefore be accessed using N1QL, Full Text Search, and normal key-value operations. Data structures can also be manipulated using the traditional sub-document and full-document key-value APIs.

Using the data structures API can help your application in two ways:

Simplicity: Data structures provide high level operations by which you can deal with documents as if they were container data structures. Adding an item to a dictionary is expressed as map_add, rather than retrieving the entire document, modifying it locally, and then saving it back to the server.
Efficiency: Data structure operations do not transfer the entire document across the network. Only the relevant data is exchanged between client and server, allowing for less network overhead and lower latency.

Creating a Data Structure

Data structures can be created implicitly using the create=True. For example, to add an item to a map, specify the document ID of the map itself (i.e. the ID which uniquely identifies the map in the server), the key within the map, and the value to store under the key:

bucket.map_add('map_id', 'name', 'Mark Nunberg', create=True)

Likewise, to create a list, specify the document ID and the value to add:

bucket.list_append('list_id', 'hello', create=True)

Note that the create=True will not overwrite an existing document with the same name if it already exists. It is therefore always safe to use, unless your application creates data structures independently.

Data structures can be explicitly created and reset using full-document methods, and you can initialize a data structure with its JSON equivalent. To create a new empty list, set, or queue, use bucket.upsert('list_id', []). To create an empty map, use bucket.upsert('map_id', {}).

Accessing Data Structures

Data structures can be accessed using their appropriate methods. Most data access methods will return an ValueResult-like object with the actual returned value under the value property.

bucket.list_get(0).value  # 'hello'
bucket.map_get('map_id', 'name').value  # 'mark nunberg'

Native Python exceptions such as IndexError will be thrown if a given map key or list index is not found within the document. If the document itself does not exist, a couchbase.exceptions.NotFoundError will be raised instead.

Here is a list of common operations:

Operation Description

map_exists

Determine if a key exists in a map.

bucket.map_exists('map_id', 'some_key') #=> True or False

map_remove

Remove a key from a map.

bucket.map_remove('map_id', 'some_key')

map_get

Get an item from a map.

bucket.map_get('map_id', 'some_key').value #=> value

If the key is not found, an IndexError is raised.

list_append

Add an item to the end of a list.

bucket.list_append('list_id', 'some_value')

list_prepend

Add an item to the beginning of a list.

bucket.list_prepend('list_id', 'some_value')

list_get

Get an item from a list by its index.

bucket.list_get('list_id', 'some_key').value #=> value

If the index is out of range, an IndexError will be thrown. Note that you can get the last array element by specifying -1 as the index.

set_add

Add an item to a set, if the item does not yet exist in the set.

bucket.set_add('list_id', 'some_value')

Note that a set is just a list. You can retrieve the entire set by simply using a full-document get operation:

set = bucket.get('list_id').value

Currently, you can only store primitive values in sets, such as strings, integers, and Booleans.

set_remove

Remove an item from a set, if it exists. An exception is not thrown if the item does not exist. You can determine if an item exists or not by the return value. If the item did not exist beforehand, None is returned.

bucket.set_remove('list_id', 'some_value')

queue_push

Add an item to the beginning of the queue.

bucket.queue_push('a_queue', 'job123', create=True)

Note that a queue is just a list. You can retrieve items from the middle of the queue by using list_get

queue_pop

Remove an item from the end queue and return it.

item = bucket.queue_pop('a_queue') #=> 'job123'

If the queue is empty, then a couchbase.exceptions.QueueEmpty error is thrown.

map_size, list_size, set_size, queue_size

These methods get the length of the data structure. For maps, this is the number of key-value pairs inside the map. For lists, queues, and sets, this is the number of elements in the structure.

len = bucket.list_size('a_list') #=> 42

Note that there are only two basic types: map and list. Types such as queue and set are merely derivatives of list.

Data Structures and Key-Value APIs

Data structures can be accessed using key-value APIs as well. In fact, the data structure API is actually a client-side wrapper around the key-value and sub-document API. Most of the data structure APIs wrap the sub-document API directly. Therefore, for example, the operation map_set can be implemented as follows:

import couchbase.subdocument as SD
def map_set(bucket, docid, key, value):
    bucket.mutate_in(docid, SD.upsert(key, value)

Some data structure operations are compound operations. This means that they do not correspond to a single server-side operation. For example, the operation set_remove does not exist on the server. In order to implement set_remove, it would require an implementation like so:

def set_remove(bucket, docid, value):
    rv = bucket.get(docid)
    doc = rv.value
    try:
        index = doc.index(value)
        bucket.mutate_in(docid, SD.remove('[{0}]'.format(index), cas=rv.cas)
    except ValueError:
        pass

In the example above, the entire document is fetched and the index of the value is retrieved. If the value exists, the index of the value is removed at the server side.

Because the data structure API is just a wrapper around the various key-value APIs, you are free to switch between them in your code.