Tuesday, November 11, 2014

Software Is Hard

NOTE: I wrote most of this blog post back in April 2014 but never published it. I am publishing it now because I think it has a certain amount of value. It might be perceived as somewhat pessimistic in tone but I'm really just trying to be realistic. Software development is not easy, and I think it's healthy to be open and honest about frustrations you face in any career.

---BEGIN POST----

Yes, software is hard. Programming isn't necessarily hard (although it can be till you've learned a language well enough) but working, functional software is hard.

Here are some reasons that make software hard (at least to me):
  1. Software is inherently not flexible. Software is incredibly precise. There is no ambiguity, even though the same actions may produce different results, it is only because settings or data somewhere were slightly different. As a result, it doesn't adapt very well.
  2. Understanding a business process and then translating into working code is incredibly difficult. Especially if you haven't worked directly in the industry your software is used.
  3. Software is abstract, it's an idea and not very visualizable.
  4. Communicating about abstractions is hard.
  5. Terminology varies, without good terminology we don't have "handles" to grab onto.
  6. Until you have a codebase entirely in your head, there are just things you don't know. You're never quite sure if you handled something in all the needed areas.
  7. Shortage of time. It seems there is usually not enough time to complete something or least a perceived shortage of time due to the massive amount of pending feature requests.
  8. Interruptions. As a developer, you have special insight into problems and you will be interrupted to help with production problems & questions. Interruptions really slow down the coding process and lead to errors and scattered thought processes.
  9. Other mental demands. Chances are you have a life outside of coding (hopefully) and there can be general life stress from other people and events in your life. These things also compete for your mental capacities and have an impact on your focus & productivity.
  10. Software doesn't stay constant. You may master a language or API, but there is always something new to learn. The principles stay the same but it is still a large task to master a new language or UI toolkit. Currently I have to be good with Java, SQL, BI, ActionScript, web services, integrations, etc let alone all the API's & frameworks such Spring, Hibernate, QuickBooks SDK, JasperReports, etc.
Of course there are many process related things that can make software hard but I'm not going to go into all of that in this post. Even if you have excellent processes the above points will usually still apply. It's just part of the business. You can go along way in helping yourself by just becoming aware of these pain points.

A good way to help is to maintain a good work life balance. Don't over work and don't over goof off. Learn to push through the resistance but also learn to recognize when you just need to take a break. Sometimes you just need something to break up the routine. Once in a while start earlier than normal or wait till after lunch to get started. Go out for a bike ride before work or maybe just take a random unplanned day off.

Ultimately I stay in software development mostly because it is a challenge, not because I love it everyday. Of course there are days when I'm having a complete blast but that is not the majority of the time. The point remains, software is hard.

Saturday, October 18, 2014

Managing Web Complexity

So the wonderful world of the web technologies is great. HTML5, CSS, JavaScript, you get the idea. You can build many wonderful things with these. Enter a tool like jQuery, and you can do things even easier and better. You have plethora of JS frameworks to help you build a website or a web app... Backbone, Angular, Ember, etc. I believe these tools & technologies work well for a large swath of web targets. However, not all. I'll breakdown how I segment the web to make this more understandable. I think the web falls into the following three categories with some overlap in between.

Websites
Websites are documents. In fact they are primarily what the web tech family was designed for. It fits well with the DOM concept. It was to facilitate a cross-platform way to display information to the world in a quasi standards based way. This does not mean the content is static but rather interact with the user to a certain extent. Dynamic content may displayed with JS or PHP. But the overarching idea of this category is to get information out. Examples are corporate websites & personal websites.

Websitey apps
This is when websites begin to function more like an application. I don't particularly like the term "websitey application" but that's all I can come up with now. In this category, the "websitey application" is very interactive. User's have accounts, they do persistent actions as in order stuff, change stuff, add content, etc. This is the beginning of where the concept of the UI being a document begins to get a little shaky but still holds up decently well. Examples would be Amazon, eBay, Drupal & Wordpress sites, and really any site with ecommerce. Web tech & tools hold up pretty well for most of these as the complexity is still limited however the document or DOM concept begins to show it's weaknesses when the application is larger. As I said before, the DOM was intended for displaying content, not really for complex dynamic & interactive scenarios. At this point JavaScript becomes somewhat of a hack to manipulate the DOM into doing what needs done.

Enterprise business apps
These are full screen business tools. The concept of "website" is just not adequate. The app takes the full browser area, no header or footer, just a menu system, and a work area. These are really web implementations of what back in the day would have been full screen desktop business applications. Examples would be QuickBooks Online, Xero Accounting, Flex Rental Solutions, and thousands of other applications that fly unknown under the cover of the enterprise. The idea of directly using many of the web tools & technologies really shows their weaknesses here. Why? Complexity. The large enterprise app has a tremendous domain size and structure. This results in large numbers of components on the UI side. It's important in this area that the UI layer be self documenting in the form of packages (namespaces), component names, inheritance, and lot of the concepts on the backend that make development sane. Without structure, it literally becomes hard to find the component that you need to work on. In JS, there is no concept of class per file, so you can have many "objects" embedded within one JS file. That leaves you to a basic text search & find to be able to find your data (DTOs) & UI components. Bottom line, the large enterprise can implode in this space if a way is not found to manage complexity. And no Backbone is not always sufficient for this.


So what's the solution for complex web apps? Honestly at this point I'm not totally sure. In abstract terms, the solution is a tool that abstracts away from web technologies that allow you to code to an API instead of the direct implementation. This might mean using an enterprise proven language to compile to JavaScript such as GWT.  I see web technologies as a low level "language" to be built on top of to unleash it's true power. Yes, there are drawbacks to this, but I think the good far out weights the bad. Just like in the sense that there are probably advantages to writing certain solutions in assembly language, but in general it's a given that higher level languages offer a tremendous productivity benefit.

However, I want to look further. I find the concept of web abstraction compelling but I'm squeamish if it's really the long long term future.  As in will it work well in 10 years. I feel that maybe the root problem of doing large Enterprise web apps is the DOM. I watched a youtube video (watch it, it's good) the other day about web frameworks and one guy made the comment something along the lines of "if you were to build a UI toolkit today, it would look nothing like the DOM". I find that fascinating. I feel the DOM is kinda a misfit for web applications but it has so much inertia going for it now, that it may be years before people start to seriously question the DOM.

Post-DOM is going to take a long time. I could easily see it being close to a decade before it reaches critical mass. In the meantime, I want to see awesome web abstraction tools. I think Vaadin & GWT have a compelling use case. I have reservations about Vaadin latency & performance but I think it's UI concept is solid.

To me, any good UI toolkit needs to be powerful but yet at the same time kinda just fade into the background and allow you to focus on solving business problems. You should not have to think about browser idiosyncrasies, compatibility issues, and structural complexity.

Cheers to awesome web abstractions, managed web complexity, and the post-DOM way of building internet applications whatever that may be!

Friday, April 25, 2014

The Programming Dilemma

Software is code. Code is structure. Structure is meaning. Meaning requires structure. Almost by definition, structure does not change or adapt or otherwise it would lose meaning. So, in order to simulate "adaptable-ness", you have to make it configurable. But there is still structure, because there are still unchanging structures that information must flow through.

You could in theory make software completely dynamic (as in get or send anything from or to a system) but then you would have to do some kind of "programming" on the client side in order to use it. For example, say I want to make it possible from the client side to be able to get any information possible out of my software system. One option is just make every piece of data available in every possible known combination (a herculean task). Or create some kind of "query" language to get the data out of my system. In essence, the application just becomes a domain specific language on top of your database.

Programming is supposed to make it easier to to do things. Users want to point and click to make things happen. They don't want to write code to make it happen. But then people want things to adapt to their needs. So you add configuration. The tradeoff is that now the system is harder to understand how to use. Taken to the extreme for flexibleness, your application becomes a maze of knobs and switches. Which oddly enough, to setup, feels a bit like programming something. So you end up programming the program. I thought the point of programming was to not have to program?

I've come to the conclusion that an application is really only 100% useful for one company max. As soon as you involve more than one company, there are differing opinions on how things should be done. There are differing opinions within a company, but a least at a company level, policies can be adjusted. To support more than one company, you have to either be dogmatic and say "you have to do it this way" or make a configuration option. As soon as you begin down the road of configuration, it becomes harder and harder to know how to use the software. It's a problem that if is not checked will bring the software to it's knees and it will die a slow hard death.

Programming doesn't really solve problems very well. It solves one problem and creates ten other ones. It is inherently self-defeating. I've said it before and I'll say it again, "software is hard", and yes it always will be.

P.S. I think it is wise to keep an eagle eye out for what NOT to program rather than what to program. There is no shortage of things to program. It takes gutsy, courageous leadership to say "we are not going to program that because it will create to many additional [programming] problems".