Simple Atomic Counter with DynamoDB and API Gateway

Subscribe to my newsletter and never miss my upcoming articles

A web application often needs a central counter. We have several simple use-cases like the member id for a registration application. Or the amount collected on a sales application - there are many that we all have seen. We have to make sure the concurrent updates do not clash, and the no data is lost in the process. This problem increases further when our application scales larger on the cloud - with millions of concurrent users.

It is best to avoid such a use case - in large scale applications. It is always advisable that we use a design that does not need an atomic counter - for example using UUID instead of an increasing number. But there are times where we just need the counter. When I found myself in such a situation, here is a solution that I used. It is based on API Gateway - Dynamo DB integration. It has a very good response time and minimal cost.

DynamoDB

We all know what is DynamoDB. For those who don't, it is the wonderful serverless NoSQL database provided by AWS. It scales seamlessly to huge volumes and throughput - maintaining extremely fast operations with latency in single digit milliseconds. It is the ideal database for anyone - startup, or enterprise. DynamoDB ensures eventual consistency for all updates. But there is a way to get atomic updates as well.

Going by the One App - One Table philosophy, create a new Item in the table. You can choose the "primaryKey" field as per your data design.

{
   primaryKey: "atomicCounter",
   counterValue: 0
}

So we have a counter that starts with 0. Now, we to configure it to increment atomically and return the number. DynamoDB API provides a method to do it. In order to invoke it, we have to configure the API Gateway

API Gateway

The API Gateway is a versatile service to configure and export API's to the external world. We can use it to invoke Lambda functions, or directly invoke specific AWS Services. We will use the latter to invoke DynamoDB directly.

So we go to the API Gateway Console, to create a new Rest API - or just a new Resource/Method in your existing application API. Create a new API, and resource and then add a GET method into it. We can use a POST as well. But, GET is more elegant since we only get the counter without providing anything.

Next, we configure the API integration to invoke the DynamoDB.

Request Integration

Request Integration.jpg

Next we provide an integration mapping for the request. We do it here - at the bottom of the Request Integration page.

Request Mapping.jpg

UpdateItem
{
    "TableName": "donation-data",
    "Key": {
        "context": {
            "S": "atomicCounter"
        },
        "id": {
            "S": "id"
        }
    },
    "UpdateExpression": "set counterValue = counterValue + :num",
    "ExpressionAttributeValues": {
        ":num": {"N": "1"}
    },
    "ReturnValues" : "UPDATED_OLD"
}

Response Integration

Similarly, we have to map the output on the response. Click on the Response Integration link and add this mapping in there

Response Integration

#set($value = $input.json('Attributes.counterValue.N'))
#set($l = $value.length())
#set($l = $l - 1)
$value.substring(1,$l)

Deploy

We are done with configuring the API. Now click on Deploy API, select a stage and it is ready to test.

Invoke the API from Postman. Everytime you invoke it, you will see the number increasing.

The update operation we used is atomic, so we will always have a unique number from the API.

Comments (3)

Owain McGuire's photo

I did hunt around a while back to find this as a SaaS application. Couldn't find it. With an authorizer, you could launch!

Under heavy workload on a specific counter, this is going to hotspot the item in DynamoDB. As it has to be atomic, there is no way to cache (check you haven't got any caching in API Gateway) and the usual Scatter-Gather pattern defeats the atomic nature too.

Fun though.

Vikas Solegaonkar's photo

That's a good idea. Will push this as a public API. API Gateway caching can be disabled. I have not tested this under Very High load - But it worked pretty well in 100 hits per second

Owain McGuire's photo

Vikas Solegaonkar there might be a little domain model here:

user has a clientId (PK) user can create multiple counters (SK) user can admin a counter, e.g. suspend, delete, and set value (reset to zero). each counter has a userkey without the admin privileges, getCounter (reads current value need to set TTL so there is no caching), incCounter (returns the new value with a timestamp) users could be internal services: orders, customers,

Add a billing mechanism for clients and their users and "this time next year we will be millionaires". youtube.com/watch?v=qp5hxHPlTq8