The I3 Client API

The i3 Client API

In general, a native i3 client performs one or more of the following i3-level operations apart from other network operations it performs:

Inserts and periodically refreshes triggers
Removes triggers
Receives and processes data corresponding to triggers
Sends data to IDs

Since most of these operations are common across all i3-clients, we provide a i3-client API, the methods of which perform these common operations. This document explains how to use the API to write native i3 client applications. In some cases, we provide details about how this is implemented.

Overview

At a high level, the i3_client API opens and manages the socket required by the client for sending and receiving the packets (and hides the details from the client). It also performs insertion and refreshing of triggers. Clients associate callback functions with different events that are possible. For example, client associates a function, say action_packet_receive() with the event corresponding to receipt of a packet on an ID. A typical i3-client that uses the client API contains the following pieces of code (not necessarily a total order):

Client initialization
Trigger operations (insertion, removal, etc)
Callback initialization
Select loop
Client shutdown

We now explain each of them in detail. For each functionality, we provide API calls that vary in the amount of control it gives to the applications and the granularity of operation. All the code is written in C and can be found in i3_client/i3_client_api.h. Sample examples can be found in the examples/ directory. i3_receiver.c and i3_sender.c contain the code for sending and receiving packets.

Client initialization

Function call:

cl_init(char* filename)

Parameters:
- filename: the name of the xml file which contains the configuration information.

Purpose: Initialize i3 client data structures. This procedure reads in the list of i3 servers from the configuration file. Optionally, this code also spawns a thread that automatically chooses the closest i3 server. Following are the parameters in the configuration file relating to the client:

Trigger operations

Choosing a ID for a trigger: Performance

The ID of a trigger determines which i3 server the trigger resides on. For reducing latency, it is preferable that the trigger resides on a i3 server close to the client machine. This can be achieved by constructing the ID carefully (if latency is not an issue, a random ID can be chosen). i3 uses Chord as the underlying DHT, and in order to ensure that the trigger resides on a specific i3 server, the ID of the trigger should be chosen such that the ID numerically lies in between the desired i3 server and its predecessor. Thus, given the chordID of the desired i3 server, it is possible to construct such an ID that works with high probability (by setting ID numerically slightly less than chordID).

The client API offers two function calls to aid in this process (if the client has been configured to determine the closest i3 server automatically):

cl_get_rtt_id(ID *id, uint64_t *rtt)

Purpose: This function call returns the latency (RTT) in micro-seconds of the i3 server with ChordID id. It returns CL_RET_OK on success.

cl_get_top_k_ids(int *k, ID best_id[], uint64_t best_rtt[]);

Purpose: This function call returns two lists (best_id, best_rtt) containing the ChordIDs and measured RTT of the k closest I3 servers. Both lists are allocated by the caller. If the client has less than k servers in its list, then k is set to the number of servers. It returns CL_RET_OK on success.

Creating a trigger

cl_create_trigger_id(id, prefix_len, id_target, flags)

This function inserts a trigger with identifier id which points to another identifier id_target. prefix_len specifies the length that is used for longest-prefix matching (the minimum value allowed is MIN_PREFIX_LEN). flags are used to specify the properties of the trigger being created. The following flags can be used individually or ORed together.

CL_TRIGGER_CFLAG_R_CONSTRAINT - Trigger is right-constrained.
CL_TRIGGER_CFLAG_L_CONSTRAINT - Trigger is left-constrained.
CL_TRIGGER_CFLAG_PUBLIC - Trigger is public. If this is flag is not used, the trigger is private by default. Since a public trigger must be left-constrained, CL_TRIGGER_CFLAG_R_CONSTRAINT and CL_TRIGGER_CFLAG_PUBLIC may not be used together.

cl_create_trigger_stack(id, prefix_len, stack, stack_len, flags)

More generally, the above function creates a stack of triggers.

Inserting and removing triggers

After creating a trigger, it can be inserted in i3 using the following call.

cl_insert_trigger(ctr, flags)

The client code would add them to the list of triggers to maintain and refresh them until they are explicitly removed. To remove a trigger, the following call is used.

cl_remove_trigger(ctr)

Callback initialization

Operations associated with triggers are specified using callback functions. Clients must associate callback functions with all the events that need to be handled. Callbacks can be specified globally or on a per-trigger basis.

cl_register_callback(uint16_t cbk_type, void (*fun)(), void *data) cl_register_trigger_callback(i3_trigger *t, uint16_t cbk_type, void (*fun)(), void *data)

These functions register callback function fun with callback type cbk_type. data is passed back to the callback function when it is invoked. Supported callback types are:

CL_CBK_TRIGGER_INSERTED: This callback is invoked when the client receives the first ack as a result of a trigger insertion. The callback function has the template: fun(i3_trigger *t, void *data) where t represents the inserted trigger, and data represents the client's data
CL_CBK_TRIGGER_REFRESH_FAILED: This callback is invoked when a trigger refresh fails. Refresh failure occurs when none of the client's refresh messages is ACKed during a refresh period which has a duration of TRIGGER_REFRESH_PERIOD sec. The client refreshes a trigger by sending a refresh message MAX_NUM_RETRIES*ACK_TIMEOUT sec before the refreshing period expires. If the first refresh message is not acked, the client resends the refresh message approximately every ACK_TIMEOUT sec. The callback function has the following template: fun(i3_trigger *t, void *data). A typical response of the client to this callback is to reinsert the trigger using the cl_reinsert_trigger function.
CL_CBK_TRIGGER_NOT_FOUND: This callback is invoked when the client sends packets to an ID id, and there is no trigger in the network matching this "id". The callback function has the following template: fun(ID *id, void *data).
CL_CBK_RECEIVE_PACKET: This callback is invoked upon receiving an i3 packet. The callback function has the following template: fun(i3_trigger *t, i3_header *hdr, cl_buf *b, void *data), where "t" represents the trigger matching the the packet's ID, "hdr" represents the packet's header, "b" contains the packet's payload, and "data" represents the client's callback data.
CL_CBK_RECEIVE_PAYLOAD: This callback is invoked upon receiving a data packet. This callback is suppressed by the CL_CBK_RECEIVE_PACKET callback. The callback function has the following template: fun(i3_trigger *t, cl_buf *b, void *data)
CL_CBK_SERVER_DOWN: This callback is invoked when the client concludes than an i3 server is down. This happens when the client either receives no ACKs (in the form of i3 _OPT_CACHE_ADDR replies) to sending packets to that server during a refresh period of ID_REFRESH_PERIOD sec, or when the client receives no acks for three consecutive i3 _OPT_REQUEST_FOR_CACHE queries. The callback function has the following template: fun(struct in_addr ip_addr, uint16_t port, void *data), where "ip_addr" and "port" identify the reported server.

Select loop

All i3 clients that require to either (i) receiving packets, or (ii) insert triggers must use cl_select(). Since the i3-client is currently single-threaded, refreshing of triggers is performed inside the cl_select() function call.

int cl_select(int n, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *cl_to)

The cl_select performs a "select" on all the parameters specified. In addition it also adds the socket it uses for receiving i3 packets to the list of read-fildes. Moreover, it changes cl_to to account for trigger refresh. When a packet is received in cl_select, the callback function is called it is an i3 packet (and after the function returns, cl_select returns to the user), else cl_select returns with the appropriate return value of select. The list of triggers to be refreshed is maintained as a priority queue sorted by the time to wait before sending the next refresh.

Pseudocode for cl_select():

forever do { t_ref <- time left to refresh next trigger if (cl_to is NULL) /* Client has specified inf timeout */ timeout = t_ref else timeout = min(cl_to, t_ref) select(..., timeout) if (packet received on i3 socket) invoke appropriate callback function if (current_time >= time to refresh next trigger) call refresh_triggers /* Warning: This assumes that t_ref >> callback processing time * If this is not true, make trivial change to * account for the correct time elapsed */ cl_to <- cl_to - t_ref; if (packet was received OR cl_to has gone down to zero) adjust cl_to and add time left in "timeout" return /* Warning: Note that this function returns when a * packet was received on i3 socket also. This is * because client semantics might require action upon * receipt of any packet */ }

Client shutdown

To free all client-allocated resources, the following function is called.

cl_exit()