Saturday, April 15, 2017

Why ExtJS UI Testing is Hard

Consider this simple ExtJS modern toggle field declaration below.



The above ExtJS code results in the following UI rendering display in a browser.



Now, consider the HTML output from that simple ExtJS declaration:



Yeah, crazy. The reason for this HTML craziness is because ExtJS does dynamic rendering. You declare some simple JavaScript code, and it takes care of the rest, including making it look the same across browsers. In order to do that, the HTML it emits can be different from browser to browser. Essentially ExtJS is a web abstraction layer that takes care of rendering so you can focus on building your app.

Now, that is just a simple toggle field component. Enter an advanced data grid with lots of rows or a tree with lots of nodes and the HTML is going to explode way beyond what's shown above for the toggle field.

Trying To UI Test this Jazz with Selenium

Selenium is designed to interact with DOM elements. It has different mechanisms to locate elements. The most reliable way to select a web element is by something unique such as an element id. However, in the case of ExtJS, developers are not supposed to set the "id" property as a potential duplicate "id" could crash or at least mess up the app. We can set the "itemId" in the ExtJS code, however that doesn't come through to the DOM at all. As a side note, I would like to know why ExtJS doesn't emit the itemId in some way to the DOM? If Sencha would change ExtJS to emit the itemId somehow into the DOM, locating unique top level div elements would be so much easier!

However, even if we did hard code the "id" or create a way to resolve the ExtJS itemId to an actual id (which is possible via JavaScript by calling the ExtJS component query Api), and it came through to DOM, there is a new problem. Take the toggle field example above. The "id" would resolve to the top level div with an id of "ext-togglefield-3". That is just the top level wrapper div for the form field and doesn't give us the id of the actual slider component. It's a couple div's down and nested a bit, with an id of "ext-toggleslider-3". This is completely dynamic HTML and there isn't any way to hard code that "id" even if I tried. Essentially, it's "inside" and unreachable from my ExtJS JavaScript declaration above.

Knowing the top level div id does help some, because then you could use Selenium look it up by id and then use a CSS selector and look for an "div.x-toggleslider" class inside that top level element. But this process is going to be different for every ExtJS component and is subject to change from version to version. Here is some Selenium Java code that would click on the toggle field above:

//For purposes of illustration, assume the toggleFieldId below 

//was resolved in a reliable way and not hard coded!
//e.g. this toggleFieldId could be resolved via ExtJS itemId by
//invoking ExtJS JavaScript code
String toggleFieldId = "ext-togglefield-3";

//get top level field by the resolved id
WebElement toggleField = driver.findElement(By.id(toggleFieldId));

//now select actual clickable slider by CSS selector on
//the toggle field element
WebElement toggleSlider = toggleField.findElement(By.cssSelector("div.x-toggleslider"));

//Now click the toggle slider
action.moveToElement(toggleSlider).click().build().perform();


The code above will actually work. However, we have to dance down through a top level ExtJS component div and select the "real" clickable components inside it via CSS selectors to get to the real thing. This is going to vary greatly from component to component and could change from version to version.

Use a UI Test Tool Designed for ExtJS [like Sencha Test]?

It seems this dynamic rendering craziness is why tools are created to help UI test ExtJS. Sencha built their own tool called "Sencha Test" which basically takes away all the DOM selection grunt & guess work away by creating an Api to select and manipulate components. The Sencha Test equivalent of the above code would be this:

ST.field('#tablet-inventory-model-view-expendable')
.setValue(true);

The "#tablet-inventory-model-view-expendable" above is an itemId we set on the ExtJS side and then from Sencha Test you can just target that direct and reliable itemId and set the value to true! That's it! You don't have to worry about figuring out the correct top level id and then finding the "real" clickable component via a CSS selector like my Java code example earlier.

This is just a simple example and as you can imagine, with data grids, trees, etc, the complexity is going to explode. With growing complexity, Sencha Test is going to shine even more.

So What Do You Do?

My team is still trying to figure that out. We'd love to use tools that are agnostic of the underlying web framework, however it's just highly complicated and frustrating dealing with dynamic DOM output from ExtJS. We are giving Sencha Test a look over as it appears to make life much easier, but it's own animal and you'd need to learn it's Api and ways of doing things.

I'll try to write another blog post in the future once we figure out our approach. Stay tuned!

If anybody has any ideas they'd like to share on how to UI test ExtJS, please comment below! Cheers!

Wednesday, February 3, 2016

Video Chat == Trust

This post is written from the context of remote teams. If you are in the office with your co-workers it's not as applicable.

Everybody instinctively seems to know that when you get in the same room as your co-workers, you get more accomplished in terms of planning & strategy.

Why is that?

You have the same voices that you do on the phone. You can create a group conference call and discuss with your voices just like you do in a room.

But everybody knows it's not the same. In person communication always works better.

But why?

Because eye contact and body language. When you only have voice, you have a small fraction of the data that you do in person. This makes it much harder to get your points across because you're unable to tell if your listeners are understanding, connecting, objecting, etc. Not to mention you don't have nice collaborative things like whiteboards.

But yet people are still resistant to video chats. Video chats allow a much greater percentage of communication data to flow (vs just voice) thus allowing much more productive meetings. But it puts people out of their comfort zone so they would sooner not do it.

This frustrates me (but I do empathize with those who object... more on that in bit).

We are human, people. We connect via eye contact and body language. This creates trust & connection.

Trust. This is probably the most important word for remote teams.

Without trust, a team is dysfunctional.

Why in the world in these days of modern technology would we cut off the most vital means of communication data when we don't have to?

To me, not using video in this modern age of 2016 is equivalent to being in your office with your door shut and talking to a person outside in the hall with just voice. How silly would that be? But yet we refuse to do video because it makes us feel uncomfortable.

Bottom line, it's because of fear. We are scared to look someone in the eye via video because it puts us out of our comfort zone. I've been there. There was a time when I was literally freaked to even think about turning on my video. I think I may have broke out in a sweat at times just thinking about it.

But eventually I started trying it. And it was super uncomfortable.

But eventually it got better. After you do this for a while, you'll begin to wonder why in the world you were so resistant.

However don't take my word for it... just google "successful remote teams" and see how many articles mention video chat. It IS a critical part of remote team success.

Conquer your fears. Do video chat with your remote co-workers. Build trust. You'll thank me someday. :)

Tuesday, January 26, 2016

Leadership == Communication

A question I've often pondered is "what is leadership"? Is it a state, is it making decisions, is it a title, is it management?

Turns out I don't think it's really fundamentally any of those. One thing I have discovered for certain, you don't need a "leadership" title to be a leader.

If you are reading this, you are a leader in some form. You might be the newest person in your company but you can lead in some way.

When I break down leadership to it's most basic element, I come up with one word.

Communication.

And really, what is communication? It is information sharing. As a leader, to facilitate communication, you have to ask questions. And then listen... like a lot. You have to deeply absorb what people around you are saying. Once you've done that, you will eventually get to the point where you begin to come up with suggestions and solutions to solving the problems around you.

Leadership is having the audacity to be vulnerable. It means once I've communicated with those around me and I think I have a solution, I'm going to throw my idea out there regardless of what my co-workers think. It means I will allow my "good" idea to be chewed on by the team and turned into something even better (or thrown away).

This is super hard, because when you think you have a good idea, and it gets ripped to shreds, it hurts. But, it's how you get better. You pick yourself up again, rinse and repeat. Eventually something will stick and you will begin to have an impact.

In fact I believe a properly functioning team is when everybody is leading. It means everybody initiates communication and is trying to solve the problems around them.

It's just not possible to lead without communication. If someone "leads" without communication, at best it is merely management.

So if you are still wondering if you are a leader (because you don't have the title), let me ask you... do you have the ability to ask questions? If your answer is yes, than yes you can be a leader.

So get up, get out of your comfort zone and go start asking some questions.

Question, listen, absorb, innovate. That's the leadership cycle. Go do it!

Saturday, April 25, 2015

Embracing JavaScript & the Web

So back in October 2014 I wrote a post about Managing Web Complexity. In that post I discussed web abstractions that allow you to write in a solid engineering language such as Java and take care of generating all the "web stuff". The main tools that I am familiar with that do this are GWT and Vaadin.

For years I have been on the side of the fence solidly FOR server side web abstractions. My main line of reasoning was something along the lines that "the web is too finicky & complex for large enterprise web applications, and JavaScript stinks". Over the last half year or so, I have changed course quite a bit. I have began a journey where I am learning the fundamental building blocks of the web. I've studied CSS and lately have been doing courses on JavaScript. Yes, you read that correctly, JavaScript. I've even dabbled with node.js (oh the thought!!). Believe me, node.js is not for the large enterprise app but it sure is a nice way to learn JavaScript. As an aside, I think node.js could work well in a microservices environment for certain kinds of microservice applications. If what you need is super fast responses with small data payloads, then it would be a good candidate. But, building a large monolith app with node.js? No way, that would not end well!

One of the big triggers for this change of course was learning about EXT.JS. While I had seen the name around before, I never took a good look at it. Turns out, it does quite a good job at managing web complexity cause all you do is write JavaScript, you don't have to deal with HTML & browser idiosyncrasies. EXT is really the jQuery of the enterprise. It's been around a long time, it's stable & mature, and it's not going away. These other JavaScript frameworks like Angular or React may just fade after a few years. Plus they just don't have the out-of-the-box UI toolkit that is needed for large enterprise web applications.

Another angle is separation of concerns. I think any UI needs to be completely separate from the server side & be completely back end implementation agnostic. In a world of many possible clients (Desktop web, mobile, integration clients, etc), this forces a good communication architecture with the back end.

With these server side frameworks, you are completely tied to the back end. In the case of Vaadin it's even worse because by default, any client side action has to go back to the server to do even the most simple UI interaction. That's bad.

By having server side agnostic clients, you could literally rewrite the back end if needed. Even though that would be a huge undertaking for any decent size application, it's totally doable, and probably easier than rewriting both the UI & back end at the same time. Easier because you have the front end "contract" and you just need to reimplement that "contract". It could also be a very necessary thing in this day and age with domains getting larger & the sheer size of data increasing rapidly. It's very possible that more & more relational database applications will need migrated to better storage systems.

So is JavaScript an awesome language? Not really, but once you get used to it, it's tolerable. As long as you are working with a framework such as EXT or jQuery, it's much better. It's easy to rat on JavaScript, but in reality it has won, so we should just stop trying to avoid it with these "compile to JavaScript frameworks". Cheers to the web!

Tuesday, November 11, 2014

Software Is Hard

NOTE: I wrote most of this blog post back in April 2014 but never published it. I am publishing it now because I think it has a certain amount of value. It might be perceived as somewhat pessimistic in tone but I'm really just trying to be realistic. Software development is not easy, and I think it's healthy to be open and honest about frustrations you face in any career.

---BEGIN POST----

Yes, software is hard. Programming isn't necessarily hard (although it can be till you've learned a language well enough) but working, functional software is hard.

Here are some reasons that make software hard (at least to me):
  1. Software is inherently not flexible. Software is incredibly precise. There is no ambiguity, even though the same actions may produce different results, it is only because settings or data somewhere were slightly different. As a result, it doesn't adapt very well.
  2. Understanding a business process and then translating into working code is incredibly difficult. Especially if you haven't worked directly in the industry your software is used.
  3. Software is abstract, it's an idea and not very visualizable.
  4. Communicating about abstractions is hard.
  5. Terminology varies, without good terminology we don't have "handles" to grab onto.
  6. Until you have a codebase entirely in your head, there are just things you don't know. You're never quite sure if you handled something in all the needed areas.
  7. Shortage of time. It seems there is usually not enough time to complete something or least a perceived shortage of time due to the massive amount of pending feature requests.
  8. Interruptions. As a developer, you have special insight into problems and you will be interrupted to help with production problems & questions. Interruptions really slow down the coding process and lead to errors and scattered thought processes.
  9. Other mental demands. Chances are you have a life outside of coding (hopefully) and there can be general life stress from other people and events in your life. These things also compete for your mental capacities and have an impact on your focus & productivity.
  10. Software doesn't stay constant. You may master a language or API, but there is always something new to learn. The principles stay the same but it is still a large task to master a new language or UI toolkit. Currently I have to be good with Java, SQL, BI, ActionScript, web services, integrations, etc let alone all the API's & frameworks such Spring, Hibernate, QuickBooks SDK, JasperReports, etc.
Of course there are many process related things that can make software hard but I'm not going to go into all of that in this post. Even if you have excellent processes the above points will usually still apply. It's just part of the business. You can go along way in helping yourself by just becoming aware of these pain points.

A good way to help is to maintain a good work life balance. Don't over work and don't over goof off. Learn to push through the resistance but also learn to recognize when you just need to take a break. Sometimes you just need something to break up the routine. Once in a while start earlier than normal or wait till after lunch to get started. Go out for a bike ride before work or maybe just take a random unplanned day off.

Ultimately I stay in software development mostly because it is a challenge, not because I love it everyday. Of course there are days when I'm having a complete blast but that is not the majority of the time. The point remains, software is hard.

Saturday, October 18, 2014

Managing Web Complexity

So the wonderful world of the web technologies is great. HTML5, CSS, JavaScript, you get the idea. You can build many wonderful things with these. Enter a tool like jQuery, and you can do things even easier and better. You have plethora of JS frameworks to help you build a website or a web app... Backbone, Angular, Ember, etc. I believe these tools & technologies work well for a large swath of web targets. However, not all. I'll breakdown how I segment the web to make this more understandable. I think the web falls into the following three categories with some overlap in between.

Websites
Websites are documents. In fact they are primarily what the web tech family was designed for. It fits well with the DOM concept. It was to facilitate a cross-platform way to display information to the world in a quasi standards based way. This does not mean the content is static but rather interact with the user to a certain extent. Dynamic content may displayed with JS or PHP. But the overarching idea of this category is to get information out. Examples are corporate websites & personal websites.

Websitey apps
This is when websites begin to function more like an application. I don't particularly like the term "websitey application" but that's all I can come up with now. In this category, the "websitey application" is very interactive. User's have accounts, they do persistent actions as in order stuff, change stuff, add content, etc. This is the beginning of where the concept of the UI being a document begins to get a little shaky but still holds up decently well. Examples would be Amazon, eBay, Drupal & Wordpress sites, and really any site with ecommerce. Web tech & tools hold up pretty well for most of these as the complexity is still limited however the document or DOM concept begins to show it's weaknesses when the application is larger. As I said before, the DOM was intended for displaying content, not really for complex dynamic & interactive scenarios. At this point JavaScript becomes somewhat of a hack to manipulate the DOM into doing what needs done.

Enterprise business apps
These are full screen business tools. The concept of "website" is just not adequate. The app takes the full browser area, no header or footer, just a menu system, and a work area. These are really web implementations of what back in the day would have been full screen desktop business applications. Examples would be QuickBooks Online, Xero Accounting, Flex Rental Solutions, and thousands of other applications that fly unknown under the cover of the enterprise. The idea of directly using many of the web tools & technologies really shows their weaknesses here. Why? Complexity. The large enterprise app has a tremendous domain size and structure. This results in large numbers of components on the UI side. It's important in this area that the UI layer be self documenting in the form of packages (namespaces), component names, inheritance, and lot of the concepts on the backend that make development sane. Without structure, it literally becomes hard to find the component that you need to work on. In JS, there is no concept of class per file, so you can have many "objects" embedded within one JS file. That leaves you to a basic text search & find to be able to find your data (DTOs) & UI components. Bottom line, the large enterprise can implode in this space if a way is not found to manage complexity. And no Backbone is not always sufficient for this.


So what's the solution for complex web apps? Honestly at this point I'm not totally sure. In abstract terms, the solution is a tool that abstracts away from web technologies that allow you to code to an API instead of the direct implementation. This might mean using an enterprise proven language to compile to JavaScript such as GWT.  I see web technologies as a low level "language" to be built on top of to unleash it's true power. Yes, there are drawbacks to this, but I think the good far out weights the bad. Just like in the sense that there are probably advantages to writing certain solutions in assembly language, but in general it's a given that higher level languages offer a tremendous productivity benefit.

However, I want to look further. I find the concept of web abstraction compelling but I'm squeamish if it's really the long long term future.  As in will it work well in 10 years. I feel that maybe the root problem of doing large Enterprise web apps is the DOM. I watched a youtube video (watch it, it's good) the other day about web frameworks and one guy made the comment something along the lines of "if you were to build a UI toolkit today, it would look nothing like the DOM". I find that fascinating. I feel the DOM is kinda a misfit for web applications but it has so much inertia going for it now, that it may be years before people start to seriously question the DOM.

Post-DOM is going to take a long time. I could easily see it being close to a decade before it reaches critical mass. In the meantime, I want to see awesome web abstraction tools. I think Vaadin & GWT have a compelling use case. I have reservations about Vaadin latency & performance but I think it's UI concept is solid.

To me, any good UI toolkit needs to be powerful but yet at the same time kinda just fade into the background and allow you to focus on solving business problems. You should not have to think about browser idiosyncrasies, compatibility issues, and structural complexity.

Cheers to awesome web abstractions, managed web complexity, and the post-DOM way of building internet applications whatever that may be!

Friday, April 25, 2014

The Programming Dilemma

Software is code. Code is structure. Structure is meaning. Meaning requires structure. Almost by definition, structure does not change or adapt or otherwise it would lose meaning. So, in order to simulate "adaptable-ness", you have to make it configurable. But there is still structure, because there are still unchanging structures that information must flow through.

You could in theory make software completely dynamic (as in get or send anything from or to a system) but then you would have to do some kind of "programming" on the client side in order to use it. For example, say I want to make it possible from the client side to be able to get any information possible out of my software system. One option is just make every piece of data available in every possible known combination (a herculean task). Or create some kind of "query" language to get the data out of my system. In essence, the application just becomes a domain specific language on top of your database.

Programming is supposed to make it easier to to do things. Users want to point and click to make things happen. They don't want to write code to make it happen. But then people want things to adapt to their needs. So you add configuration. The tradeoff is that now the system is harder to understand how to use. Taken to the extreme for flexibleness, your application becomes a maze of knobs and switches. Which oddly enough, to setup, feels a bit like programming something. So you end up programming the program. I thought the point of programming was to not have to program?

I've come to the conclusion that an application is really only 100% useful for one company max. As soon as you involve more than one company, there are differing opinions on how things should be done. There are differing opinions within a company, but a least at a company level, policies can be adjusted. To support more than one company, you have to either be dogmatic and say "you have to do it this way" or make a configuration option. As soon as you begin down the road of configuration, it becomes harder and harder to know how to use the software. It's a problem that if is not checked will bring the software to it's knees and it will die a slow hard death.

Programming doesn't really solve problems very well. It solves one problem and creates ten other ones. It is inherently self-defeating. I've said it before and I'll say it again, "software is hard", and yes it always will be.

P.S. I think it is wise to keep an eagle eye out for what NOT to program rather than what to program. There is no shortage of things to program. It takes gutsy, courageous leadership to say "we are not going to program that because it will create to many additional [programming] problems".