Exploring the Depths of Amazon Kinesis Data Streams - Part 2: Scaling

Another way of handling this issue is to manually split and merge shards yourself. If you first merge your shards into 1 shard or perform the UpdateShardCount API down to 1 shard, you can then perform the SplitShard API call. This API call allows you to provide your own minimum hash value for the shard in question. This operation can take one shard and produce two new shards whose hash ranges are [0,n-1] and [n,9] where n is your choice of minimum hash value. Looking at our example above, if we joined back to a single shard with the hash range of [0,9], we can then manually split with a minimum hash value of 3. This will create two shards with the following values: [0,2] and [3,9]. Even though new values will not be evenly distributed between these shards, we do know that our keys will be evenly distributed. If we have a new value of bam, this would have a 30% chance to be mapped to shard 1 and a 70% chance to be mapped to shard 2.

Some interesting things to consider when performing this operation is that Kinesis does not retroactively fix previous shards. If you took the time to review the DescribeStream API call, you will notice two values for ParentShardId and AdjacentParentShardId. These values are how Kinesis hides from you the fact that records for some period of time may exist across multiple shards. Referring back to our example, if we have our initial shards with IDs 0 and 1, when we perform the merge, we create a new shard with id 2 whose parent shard is 0 and adjacent parent shard id is 1. Once we perform the split, we create two more shards with id 3 and 4. Their parent shard ids are both 2 and there is no adjacent parent shard id.

If you now take time to read through some of the restrictions and considerations for scaling from the API documents for UpdateShardCount, they should hopefully begin to make some sense.

Although we don’t have the ability to alter the underlying hash function which AWS uses for kinesis, an alternative solution is to provide our own hash value as an explicit hash. When posting a record to Kinesis, you have the opportunity to override the generated hash with an explicit hash value. The primary concern with this approach is that you have to be able to consistently maintain that hash value indefinitely and make certain any changes are a well-documented event. Altering the manual hash value can result in records being published to a different shard and resulting in loss of order over a short period of time.

These are not the only options to further scale your solution. Kinesis has a fairly large feature set and other advanced features to help in further optimization. Join us in our next blog post in this series as we dive once more into some unique features such as Parallelization Factor and Enhanced Fan Out.