Simple REST Api Throttling in Spring Boot

A couple weeks I set out to implement some basic sanity REST Api Throttling for our Spring Boot application at work... turns out it was much more of a rabbit hole than I expected. I wish I could expand on why in great detail, but I my time to write is limited at the moment.

I recommend this "Stop Rate Limiting! Capacity Management Done Right" talk to get your head around the problem domain: https://www.youtube.com/watch?v=m64SWl9bfvk

TL;DR; of the video:

• Key takeaway: Limit concurrency, not request rate
• Throughput != capacity
• It's really hard to know the true capacity of a service
• Learned about Little's Law https://en.wikipedia.org/wiki/Little%27s_law

My initial implementation was just a Guava RateLimiter per subdomain (aka tenant). After watching the above video and reading various articles on the internet, I landed on the implementation below which uses a Semaphore and RateLimiter per subdomain:

Points on the Implementation Above

• This RateLimitFilter is at the very top of the stack (aka it's registration order is zero) so that no unnecessary throw away work is done in the case of a service denial
• It is registered in our WebApplicationConfig class via an @Bean method that returns a FilterRegistrationBean (see image below)
• The Semaphor acts as a "Leaky Bucket" in queue theory
• The Guava RateLimiter acts as the mechanism to keep requests per subdomain flowing at a consistent rate (which can cause the Semaphor to "fill up" and deny requests
• The filter sends a HTTP Error 429 if there are too many concurrent requests. Be sure you have an external integration test against a live environment that can actually produce 429 failures!
• Our production cluster is 3 VM's, so per tenant it effectively allows 30 concurrent requests at a stable rate of 12 requests per second per tenant.

This is in our WebApplicationConfig class:

Why I put the Api Throttling inside Spring Boot app vs somewhere "above the application"

• I wanted the "origin service" (aka our Spring Boot app) to have some "built in" self protection even if we eventually better protection "above the app"
• Since the implementation is per subdomain (aka tenant), I have all the control right in the code base for easy tweaking/deploying later
• We want to tie the # of Api Requests into a billing plan down the road. My assumption was this would be easier to do inside Spring Boot.

That's all I have for now. What do you think? I'd love to hear your feedback on this implementation and/or how you have solved this problem at your organization.

Out with the Old, In with the New

So I've had this old Drupal 6 website setup for rogerdiller.com since 2009. It was built & designed for a very different time in my career. Back then, I was trying to attract consulting offers via the contact form. Now days, I would not consider consulting. Period. I have more than enough to do at my day job.

My website provider has continued to raise prices on me for the last few years since I needed some kinda "extended PHP 5.2 support". I finally decided to today that I've "had enough". I stopped the website hosting contract and moved my domain to a simple "domain only" contract. I setup a simple https://about.me page for my basic info and redirect my rogerdiller.com domain to that now.

Gah, this is so much simpler (and boy, do I love simple). No more mind bending Drupal upgrade jazz to worry about and really, there is nothing to maintain or worry about now. Besides just making sure my rogerdiller.com is on auto renew, which it is.

I also revamped the theme on this blog. Now, to just develop a more regular blogging habit...

Statement on ExtJS

In late August the news came out that Sencha was acquired by a company called IDERA. When I first saw the email, I was shocked & confused at the news. Was this a good thing or bad thing? I spent the following weekend googling for info on the internet, reading tweets, etc to try to figure out what it meant. There were rumors the engineering team was being let go, but nothing was conclusive and it remained that way for a few weeks.

In the last week or so it has become abundantly clear. The Sencha engineering team has been let go. The evidence was all the Sencha engineers announcing their last day on social media.

I work at Flex Rental Solutions, and we bought into ExtJS and its "one platform, no dependencies" philosophy a couple years ago. We are rewriting our Adobe Flash app starting with mobile first with plans to move onto desktop next year. We've had great success with our mobile app using the toolkit.

However, with the news that the Sencha engineers are gone, we are seriously questioning whether we will stick with ExtJS. Our mobile product is about to ship, so we are not going to pivot for mobile in the foreseeable future BUT we are considering alternatives for desktop. We have been banking on the "modern" toolkit to get fully up to par with the "classic" toolkit (presumably in the 7.0 release). We want to be able to build phone, tablet, and desktop with a single toolkit and codebase, and do not want to use the "classic" toolkit (since it's basically the legacy toolkit). With the Sencha engineers gone, the "modern" toolkit parity seems highly unlikely to happen, at least on a timeline that works for us.

ExtJS is a great BUT highly unused framework these days. Nobody really talks about it in the tech community, at conferences etc. It's sad ExtJS is such a great framework that almost nobody has heard of. I fully believe that is due to bad management of the toolkit, and not the toolkit itself. Here are some of my recommendations to give ExtJS a shot at becoming relevant again:

ExtJS should be free

Charging for the framework at all is just completely misguided. Even worse, you have to start with a 5 developer license pack. That pretty much rules out all the individual developers and anybody just looking to get an app off the ground. There is nothing wrong with having a company behind it (that makes money), but it should make money off support of the framework and other value added tooling, services, etc.

ExtJS should be open sourced

With ExtJS engineers gone, the only future ExtJS has is to be open sourced. Otherwise, the toolkit is just going to die. ExtJS is an incredibly powerful and complex toolkit. IDERA is kidding themselves if they think they can just pick up things where the Sencha engineers left off. Open it up to the community, and ExtJS actually has a shot at competing with the likes of React, Angular, Vue.js etc.

ExtJS needs to embrace mainstream JavaScript

It's been frustrating to me that ExtJS doesn't embrace the JavaScript ecosystem in terms of tooling and access to 3rd party JS libraries. ExtJS should just be a dependency that is declared in a NPM "package.json" file with build support via mainstream tools (such as Babel & Webpack). I know ExtJS likes to promote "one platform, no dependencies", but c'mon, sometimes you just want to bring in a node module that helps you do String manipulation or a multitude of other non-UI JavaScript operations that is not included in ExtJS or JavaScript.

What We'll Do at Flex Rental Solutions

Basically if full "modern" toolkit parity doesn't come out by early 2018, we will be forced to ditch ExtJS for desktop. This means that when our annual subscription comes around in June 2018, we will NOT be renewing. We have everything we need to build mobile in 6.2 & 6.5, so there will be no point in us renewing. We are already putting together a Plan B. We can already see there is viable path for us with React and mainstream JavaScript.

Summary

So IDERA, are you going to make the right choices? There are tons of other companies banking on modern toolkit parity and for ExtJS to embrace mainstream JavaScript. You will see companies leaving in droves for React/Vue.js etc if you make the wrong choices. ExtJS is a fantastic piece of software. Please do what's best for ExtJS and all its dedicated developers.

Why ExtJS UI Testing is Hard

Consider this simple ExtJS modern toggle field declaration below.



The above ExtJS code results in the following UI rendering display in a browser.

Now, consider the HTML output from that simple ExtJS declaration:



Yeah, crazy. The reason for this HTML craziness is because ExtJS does dynamic rendering. You declare some simple JavaScript code, and it takes care of the rest, including making it look the same across browsers. In order to do that, the HTML it emits can be different from browser to browser. Essentially ExtJS is a web abstraction layer that takes care of rendering so you can focus on building your app.

Now, that is just a simple toggle field component. Enter an advanced data grid with lots of rows or a tree with lots of nodes and the HTML is going to explode way beyond what's shown above for the toggle field.

Trying To UI Test this Jazz with Selenium

Selenium is designed to interact with DOM elements. It has different mechanisms to locate elements. The most reliable way to select a web element is by something unique such as an element id. However, in the case of ExtJS, developers are not supposed to set the "id" property as a potential duplicate "id" could crash or at least mess up the app. We can set the "itemId" in the ExtJS code, however that doesn't come through to the DOM at all. As a side note, I would like to know why ExtJS doesn't emit the itemId in some way to the DOM? If Sencha would change ExtJS to emit the itemId somehow into the DOM, locating unique top level div elements would be so much easier!

However, even if we did hard code the "id" or create a way to resolve the ExtJS itemId to an actual id (which is possible via JavaScript by calling the ExtJS component query Api), and it came through to DOM, there is a new problem. Take the toggle field example above. The "id" would resolve to the top level div with an id of "ext-togglefield-3". That is just the top level wrapper div for the form field and doesn't give us the id of the actual slider component. It's a couple div's down and nested a bit, with an id of "ext-toggleslider-3". This is completely dynamic HTML and there isn't any way to hard code that "id" even if I tried. Essentially, it's "inside" and unreachable from my ExtJS JavaScript declaration above.

Knowing the top level div id does help some, because then you could use Selenium look it up by id and then use a CSS selector and look for an "div.x-toggleslider" class inside that top level element. But this process is going to be different for every ExtJS component and is subject to change from version to version. Here is some Selenium Java code that would click on the toggle field above:

//For purposes of illustration, assume the toggleFieldId below 
//was resolved in a reliable way and not hard coded!
//e.g. this toggleFieldId could be resolved via ExtJS itemId by
//invoking ExtJS JavaScript code
String toggleFieldId = "ext-togglefield-3";

//get top level field by the resolved id
WebElement toggleField = driver.findElement(By.id(toggleFieldId));

//now select actual clickable slider by CSS selector on
//the toggle field element
WebElement toggleSlider = toggleField.findElement(By.cssSelector("div.x-toggleslider"));

//Now click the toggle slider
action.moveToElement(toggleSlider).click().build().perform();

The code above will actually work. However, we have to dance down through a top level ExtJS component div and select the "real" clickable components inside it via CSS selectors to get to the real thing. This is going to vary greatly from component to component and could change from version to version.

Use a UI Test Tool Designed for ExtJS [like Sencha Test]?

It seems this dynamic rendering craziness is why tools are created to help UI test ExtJS. Sencha built their own tool called "Sencha Test" which basically takes away all the DOM selection grunt & guess work away by creating an Api to select and manipulate components. The Sencha Test equivalent of the above code would be this:

ST.field('#tablet-inventory-model-view-expendable')
.setValue(true);

The "#tablet-inventory-model-view-expendable" above is an itemId we set on the ExtJS side and then from Sencha Test you can just target that direct and reliable itemId and set the value to true! That's it! You don't have to worry about figuring out the correct top level id and then finding the "real" clickable component via a CSS selector like my Java code example earlier.

This is just a simple example and as you can imagine, with data grids, trees, etc, the complexity is going to explode. With growing complexity, Sencha Test is going to shine even more.

So What Do You Do?

My team is still trying to figure that out. We'd love to use tools that are agnostic of the underlying web framework, however it's just highly complicated and frustrating dealing with dynamic DOM output from ExtJS. We are giving Sencha Test a look over as it appears to make life much easier, but it's own animal and you'd need to learn it's Api and ways of doing things.

I'll try to write another blog post in the future once we figure out our approach. Stay tuned!

If anybody has any ideas they'd like to share on how to UI test ExtJS, please comment below! Cheers!

Video Chat == Trust

This post is written from the context of remote teams. If you are in the office with your co-workers it's not as applicable.

Everybody instinctively seems to know that when you get in the same room as your co-workers, you get more accomplished in terms of planning & strategy.

Why is that?

You have the same voices that you do on the phone. You can create a group conference call and discuss with your voices just like you do in a room.

But everybody knows it's not the same. In person communication always works better.

But why?

Because eye contact and body language. When you only have voice, you have a small fraction of the data that you do in person. This makes it much harder to get your points across because you're unable to tell if your listeners are understanding, connecting, objecting, etc. Not to mention you don't have nice collaborative things like whiteboards.

But yet people are still resistant to video chats. Video chats allow a much greater percentage of communication data to flow (vs just voice) thus allowing much more productive meetings. But it puts people out of their comfort zone so they would sooner not do it.

This frustrates me (but I do empathize with those who object... more on that in bit).

We are human, people. We connect via eye contact and body language. This creates trust & connection.

Trust. This is probably the most important word for remote teams.

Without trust, a team is dysfunctional.

Why in the world in these days of modern technology would we cut off the most vital means of communication data when we don't have to?

To me, not using video in this modern age of 2016 is equivalent to being in your office with your door shut and talking to a person outside in the hall with just voice. How silly would that be? But yet we refuse to do video because it makes us feel uncomfortable.

Bottom line, it's because of fear. We are scared to look someone in the eye via video because it puts us out of our comfort zone. I've been there. There was a time when I was literally freaked to even think about turning on my video. I think I may have broke out in a sweat at times just thinking about it.

But eventually I started trying it. And it was super uncomfortable.

But eventually it got better. After you do this for a while, you'll begin to wonder why in the world you were so resistant.

However don't take my word for it... just google "successful remote teams" and see how many articles mention video chat. It IS a critical part of remote team success.

Conquer your fears. Do video chat with your remote co-workers. Build trust. You'll thank me someday. :)

Leadership == Communication

A question I've often pondered is "what is leadership"? Is it a state, is it making decisions, is it a title, is it management?

Turns out I don't think it's really fundamentally any of those. One thing I have discovered for certain, you don't need a "leadership" title to be a leader.

If you are reading this, you are a leader in some form. You might be the newest person in your company but you can lead in some way.

When I break down leadership to it's most basic element, I come up with one word.

Communication.

And really, what is communication? It is information sharing. As a leader, to facilitate communication, you have to ask questions. And then listen... like a lot. You have to deeply absorb what people around you are saying. Once you've done that, you will eventually get to the point where you begin to come up with suggestions and solutions to solving the problems around you.

Leadership is having the audacity to be vulnerable. It means once I've communicated with those around me and I think I have a solution, I'm going to throw my idea out there regardless of what my co-workers think. It means I will allow my "good" idea to be chewed on by the team and turned into something even better (or thrown away).

This is super hard, because when you think you have a good idea, and it gets ripped to shreds, it hurts. But, it's how you get better. You pick yourself up again, rinse and repeat. Eventually something will stick and you will begin to have an impact.

In fact I believe a properly functioning team is when everybody is leading. It means everybody initiates communication and is trying to solve the problems around them.

It's just not possible to lead without communication. If someone "leads" without communication, at best it is merely management.

So if you are still wondering if you are a leader (because you don't have the title), let me ask you... do you have the ability to ask questions? If your answer is yes, than yes you can be a leader.

So get up, get out of your comfort zone and go start asking some questions.

Question, listen, absorb, innovate. That's the leadership cycle. Go do it!

Embracing JavaScript & the Web

So back in October 2014 I wrote a post about Managing Web Complexity. In that post I discussed web abstractions that allow you to write in a solid engineering language such as Java and take care of generating all the "web stuff". The main tools that I am familiar with that do this are GWT and Vaadin.

For years I have been on the side of the fence solidly FOR server side web abstractions. My main line of reasoning was something along the lines that "the web is too finicky & complex for large enterprise web applications, and JavaScript stinks". Over the last half year or so, I have changed course quite a bit. I have began a journey where I am learning the fundamental building blocks of the web. I've studied CSS and lately have been doing courses on JavaScript. Yes, you read that correctly, JavaScript. I've even dabbled with node.js (oh the thought!!). Believe me, node.js is not for the large enterprise app but it sure is a nice way to learn JavaScript. As an aside, I think node.js could work well in a microservices environment for certain kinds of microservice applications. If what you need is super fast responses with small data payloads, then it would be a good candidate. But, building a large monolith app with node.js? No way, that would not end well!

One of the big triggers for this change of course was learning about EXT.JS. While I had seen the name around before, I never took a good look at it. Turns out, it does quite a good job at managing web complexity cause all you do is write JavaScript, you don't have to deal with HTML & browser idiosyncrasies. EXT is really the jQuery of the enterprise. It's been around a long time, it's stable & mature, and it's not going away. These other JavaScript frameworks like Angular or React may just fade after a few years. Plus they just don't have the out-of-the-box UI toolkit that is needed for large enterprise web applications.

Another angle is separation of concerns. I think any UI needs to be completely separate from the server side & be completely back end implementation agnostic. In a world of many possible clients (Desktop web, mobile, integration clients, etc), this forces a good communication architecture with the back end.

With these server side frameworks, you are completely tied to the back end. In the case of Vaadin it's even worse because by default, any client side action has to go back to the server to do even the most simple UI interaction. That's bad.

By having server side agnostic clients, you could literally rewrite the back end if needed. Even though that would be a huge undertaking for any decent size application, it's totally doable, and probably easier than rewriting both the UI & back end at the same time. Easier because you have the front end "contract" and you just need to reimplement that "contract". It could also be a very necessary thing in this day and age with domains getting larger & the sheer size of data increasing rapidly. It's very possible that more & more relational database applications will need migrated to better storage systems.

So is JavaScript an awesome language? Not really, but once you get used to it, it's tolerable. As long as you are working with a framework such as EXT or jQuery, it's much better. It's easy to rat on JavaScript, but in reality it has won, so we should just stop trying to avoid it with these "compile to JavaScript frameworks". Cheers to the web!