Secret Rotation for JWT tokens

When you are using JSON Web Token (JWT), or any other token technology that requires to sign or encrypt payload information, it is important to set an expiration date to the token, so if the token expires, you can either assume that this might be considered a security breach and you refuse any communication using this token, or you decide to enable the token by updating it with new expiry date.

But it is also important to use some kind of secret rotation algorithm, so the secret used to sign or encrypt a token is periodically updated, so if the secret is compromised the tokens leaked by this key is less. Also in this way you are decreasing the probability of a secret being broken.

There are several strategies for implementing this, but in this post, I am going to explain how I implemented secret rotation in one project I developed some years ago to sign JWT tokens with HMAC algorithm.

I am going to show how to create a JWT token in Java.

Notice that what you need to do here is creating an algorithm object setting HMAC algorithm and set a secret that is used to sign and verify instance.

So what we need is to rotate this algorithm instance every X minutes, so the probability of breaking the secret, and that the broken secret is still valid, becomes very low.

So how to rotate secrets? Well, with a really simple algorithm that everyone (even if you are not a crypto expert) can understand. Just using time.

So to generate the secret, you need a string, in the previous example was secret String, of course, this is not so secure, so the idea is to compose this secret string by a root (something we called the big bang part) + a shifted part time. In summary the secret is <bigbang>+<timeInMilliseconds>

Bing bang part has no mystery, it is just a static part, for example, my_super_secret.

The interesting part is the time part. Suppose you want to renew secret every second. You only need to do this:

I am just putting 0s to milliseconds part, so if I run this I get something like:

1515091335543
1515091335500
1515091335500

Notice that although between second and third print, it has passed 50 milliseconds, the time part is exactly the same. And it will be the same during the same second.

Of course, this is an extreme example where the secret is changed every second but the idea is that you remove the part of the time that you want to ignore, and fill it with 0s. For this reason, first, you are dividing the time and then multiply by the same number.

For example, suppose that you want to rotate the secret every 10 minutes you just need to divide and multiply for 600000.

There are two problems with this approach that can be fixed although one of them is not really a big issue.

The first one is that since you are truncating the time if you want to change the secret every minute and for example, the first calculation occurs in the middle of a minute, then for just this initial case, the rotation will occur after 30 seconds and not 1 minute. Not a big problem and in our project, we did nothing to fix it.

The second one is what's happening with tokens that were signed just before the secret rotation, they are still valid and you need to be able to verify them too, not with the new secret but with previous one.

To fix this, what we did was to create a valid window, where the previous valid secret was also maintained. So when the system receives a token, it is verified with the current secret, if it passes then we can do any other checks and work with it, if not then the token is verified by the previous secret. If it passes, the token is recreated and signed with the new secret, and if not then obviously this token is invalid and must be refused.

To create the algorithm object for JWT you only need to do something like:

What I really like about this solution is:

• It is clean, no need for extra elements on your system.
• No need for triggered threads that are run asynchronously to update the secret.
• It is really performant, you don't need to access an external system.
• Testing the service is really easy.
• The process of verifying is responsible for rotating the secret.
• It is really easy to scale, in fact, you don't need to do anything, you can add more and more instances of the same service and all of them will rotate the secret at the same time, and all of them will use the same secret, so the rotating process is really stateless, you can scale up or down your instances and all instances will continue to be able to verify tokens signed by other instances.

But of course there are some drawbacks:

• You still need to share a part of the secret (the big bang part) to each of the services in a secure way. Maybe using Kubernetes secrets, Vault from Hashicorp or if you are not using microservices, you can just copy a file into a concrete location and when the service is up and running, read the big bang part, and then just remove it.
• If your physical servers are in different time zones, then using this approach might be more problematic. Also, you need that the servers are more or less synchronized. Since you are storing the previous token and current token, it is not necessary that they are synced in the same second and some seconds delay is still possible without any issue.

So we have seen a really simple way of rotating secrets so you can keep your tokens safer. Of course, there are other ways of doing the same. In this post, I just have explained how I did it in a monolith application we developed three years ago, and it worked really well.

We keep learning,
Alex.

You just want attention, you don't want my heart, Maybe you just hate the thought of me with someone new, Yeah, you just want attention, I knew from the start, You're just making sure I'm never gettin' over you (Attention - Charlie Puth)

Music: https://www.youtube.com/watch?v=nfs8NYg7yQM

Follow me at https://twitter.com/alexsotob

Cloud Native Applications with JWT

A native cloud application is an application that is developed for a cloud computing environment.

There is no specific answer to the question "what is a cloud-native application" but different concepts that must be met. 

One of the most important in my opinion is the ability to scale up and down at a rapid rate. And this means that our applications cannot have any state on each of the servers since if one server goes down or is scaled down, then the state stored in that server will be lost.

This is very well summarized at https://www.youtube.com/watch?v=osz-MT3AxqA where it is explained with a shopping cart example. In monolith approach, you store the products of the shopping cart in a server session, if the server went down then all products of shopping cart were lost as well. In a cloud-native app, where server instances can be scaled up and down quickly, it is important to not have this stateful behavior on your services and design them to be stateless.

There are different approaches to achieve this goal of implementing a stateless architecture but they can be summarized into two categories:

• Use a distributed in-memory key/value data store like Infinispan.
• Use a token which acts as a session between client and server using for example JWT.

In this post, I am going to introduce you the later approach.

From https://jwt.io/introduction/ site:

JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object.  

This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret using HMAC or a public/private key pair using RSA.

JSON Web Tokens consist of three Base64Url strings separated by dots which are: Header.Payload.Signature

So the idea basic idea for implementing stateless architecture on backend using JWT is the next one:

1. When the user adds the first product, the backend service generates a new JWT token with the product added and sent it back to the frontend.
2. When the user adds a new product, it sends the product to add and also the JWT token that was sent before by backend.
3. Then the backend verifies that the token has not been modified (verifying the signature), then it gets the products from JWT payload added previously and add the new one to the list. Finally, it creates a new token with previous and new products and it sent it back to the frontend.
4. The same process is repeated all the time.

So as you can see, now it is not necessary to maintain any state or add any new database service on backend side, you just need to sent back and forward the JWT token with the products inside.

I have recorded a video of a simple shopping cart example where I show the stateless nature of the solution. It can be seen at:

Also if you want to check the project that I used for recording you can take a look at https://github.com/lordofthejars/shop-jwt.

Notice that this is just a simple post so you can get the basic idea. But you need to take into consideration next things to use it in production:

1. Use HTTPS instead of HTTP
2. JWT just signs the token, if you want extra protection apart from HTTPS, use JWE to encrypt the payload of JWT token as well.
3. Fingerprinting the token to avoid any man-in-the-middle attack and use these parameters as authentication parameters for the token.
4. JWT can be used for passing authentication and authorization things as well.

You can watch my talk at JavaZone where I introduce some of these techniques:

Make your REST services attack proof - Alex Soto Bueno from JavaZone on Vimeo.

The good part of JWT approach is that it simplifies a lot the deployment of the service, you don't need to deploy or configure any other distributed database to share the content across the cluster, which minimizes the problems related to the network for communicating to the distributed database or misconfiguring of any of the nodes.

The drawback is that the client needs to be aware to receive and sent back the token and deal with it. In backend side, you need to sign and verify every token all the time.

Note that this approach might work in some cases and might get into some troubles in others (for example if there are parallel connections to backend all of them modifying the token). This post just shows an example of how I implemented this stateless thing in a project with specific requirements, but in other cases it might be wrong to do it. A real shopping cart implementation would have some problems, but for the sake of simplicity and having a business model that everyone undertands, I decided to implement it in this way.

We keep learning,

Alex.

Turn it up, it's your favorite song (hey), Dance, dance, dance to the distortion, Turn it up (turn it up), keep it on repeat, Stumbling around like a wasted zombie (like a wasted zombie) (Chained to the Rhythm - Katy Perry)

Music: https://www.youtube.com/watch?v=Um7pMggPnug

Follow me at https://twitter.com/alexsotob

Writing end to end test for a microservices architecture

UPDATE: I was not really sure if I should call this post with e2e tests since I knew it can get some confusions, but I couldn0t think about any other name. After reading https://martinfowler.com/bliki/IntegrationTest.html I can be sure that what I am describing here is how to do Narrow Integration Test.

One of the main aspects of microservices architecture is that the application is formed as a collection of loosely coupled services each one deployable independently and communicated each other with some kind of light protocol.

It is because of microservices architecture is a distributed system that makes writing end to end tests really hard. Suppose next simple example provided by Red Hat as an example of microservices architecture (https://github.com/jbossdemocentral/coolstore-microservice):

Now suppose that you want to write an end to end test for Cart Service. You will quickly see that it is not easy at all, let me enumerate some of the reasons:

• Cart Service needs to know how to boot up Pricing Service, Catalog Service, and MongoDB (and if you want to involve the front-end as well then Coolstore GW and WebUI).
• Cart Service needs to prepare some data (fixtures) for both of external services.
• You communicate with services using a network. It might occur that some tests fail not because of a real failure but because of an infrastructure problem or because the other services have any bug. So the probability of these tests become flaky and start failing not because any changed introduced in current service is higher.
• In more complex cases running these tests might be expensive, in terms of cost (deploying to the cloud), time (booting up all the infrastructure and services) and maintenance time.
• Difficult to run them in developer machine, since you need all the pieces installed on the machine.

For this reason, the end to end tests is not the best approach for testing a microservice, but you still need a way to test from the beginning to the end of the service.

It is necessary to find a way to "simulate" these external dependencies without having to inject any mock object. What we need to do is cheat the service under test so it really thinks it is communicating with the real external services, when in reality it is not.

The method that allows us to do it is Service Virtualiztion.  Service virtualization is a method to emulate the behavior of component applications such as API based.

You can think about service virtualization as mocking approach you used to implement in OOP but instead of simulating at the object level, you simulate at the service level. It is mocking for the enterprise.

There are a lot of service virtualization tools out there, but in my experience, in the JVM ecosystem, one of the tools that work better is Hoverfly.

Let's see how an "end-to-end" test looks like for Cart Service.

This service is implemented using Spring Boot, so we are using Spring Boot Test framework. The important part here is that the URL where Catalog service is deployed is specified by using CATALOG_ENDPOINT property.  And for this test, it is set to catalog.

The next important point is the Hoverfly class rule section. In that rule next things are specified:

1. An Http proxy is started before the test and all outgoing traffic from JVM is redirected to that proxy.
2. It records that when a request to host catalog is done and the path is /api/products it must return a success result with given json document.

The test itself just uses TestRestTemplate (it is a rest client) and validates that you can add some elements to the cart.

Notice that you don't need to configure where the Http proxy is started or configure any port because Hoverfly automatically configures JVM network parameters so any network communication goes through Hoverfly proxy.

So notice that now you don't need to know how to boot up Catalog service nor how to configure it with correct data.

You are testing the whole service within its boundaries, from incoming messages to outgoing messages to other services, without mocking any internal element.

Probably you are wondering "What's happening in case of current  service has also a dependency on a database server?"

In this case, you do as usual since the service itself knows which database server is using and the kind of data it requires, you only need to boot up the database server, populate required data (fixtures) and execute tests. For this scenario I suggest you using Arquillian Cube Docker to bootup database service from a Docker container so you don't need to install it on each machine you need to run tests and Arquillian Persistence Extension for maintaining the database into a known state.

In next example of rating service, you can see briefly how to use them for persistence tests:

With this approach, you are ensuring that all inner components of the service work together as expected and avoiding the flakiness nature of end to end tests in microservices.

So end to end tests in any microservice is not exactly the same of an end to end test in a monolith application, you are still testing the whole service, but just keeping a controlled environment, where test only depends on components within the boundary of service.

How contract tests fit on this? Well actually everything showed here can be used in consumer and provider side of contract testing to avoid having to boot up any external service.  In this way, as many authors conclude, if you are using contract tests, these are becoming the new end to end tests.

You can see full projects where both tests are used at https://github.com/lordofthejars/coolstore-microservice/blob/a6fb45710f4830f77221ed3d47e808f7ef3f2790/cart-service/src/test/java/com/redhat/coolstore/CartServiceBoundaryTest.java and https://github.com/lordofthejars/coolstore-microservice/blob/22779d9e70df2db244e6be694c7e35676fb1674f/rating-service/src/test/java/com/redhat/coolstore/rating/verticle/service/ApeCubeRatingServiceTest.java

We keep learning,
Alex.

When every star falls from the sky, And every last heart in the world breaks, It's gonna be OK. (Ok - Robin Schulz)

Music: https://www.youtube.com/watch?v=P9-4xHVc7uk

Follow me at https://twitter.com/alexsotob