Tuesday, November 11, 2014

Software Is Hard

NOTE: I wrote most of this blog post back in April 2014 but never published it. I am publishing it now because I think it has a certain amount of value. It might be perceived as somewhat pessimistic in tone but I'm really just trying to be realistic. Software development is not easy, and I think it's healthy to be open and honest about frustrations you face in any career.


Yes, software is hard. Programming isn't necessarily hard (although it can be till you've learned a language well enough) but working, functional software is hard.

Here are some reasons that make software hard (at least to me):
  1. Software is inherently not flexible. Software is incredibly precise. There is no ambiguity, even though the same actions may produce different results, it is only because settings or data somewhere were slightly different. As a result, it doesn't adapt very well.
  2. Understanding a business process and then translating into working code is incredibly difficult. Especially if you haven't worked directly in the industry your software is used.
  3. Software is abstract, it's an idea and not very visualizable.
  4. Communicating about abstractions is hard.
  5. Terminology varies, without good terminology we don't have "handles" to grab onto.
  6. Until you have a codebase entirely in your head, there are just things you don't know. You're never quite sure if you handled something in all the needed areas.
  7. Shortage of time. It seems there is usually not enough time to complete something or least a perceived shortage of time due to the massive amount of pending feature requests.
  8. Interruptions. As a developer, you have special insight into problems and you will be interrupted to help with production problems & questions. Interruptions really slow down the coding process and lead to errors and scattered thought processes.
  9. Other mental demands. Chances are you have a life outside of coding (hopefully) and there can be general life stress from other people and events in your life. These things also compete for your mental capacities and have an impact on your focus & productivity.
  10. Software doesn't stay constant. You may master a language or API, but there is always something new to learn. The principles stay the same but it is still a large task to master a new language or UI toolkit. Currently I have to be good with Java, SQL, BI, ActionScript, web services, integrations, etc let alone all the API's & frameworks such Spring, Hibernate, QuickBooks SDK, JasperReports, etc.
Of course there are many process related things that can make software hard but I'm not going to go into all of that in this post. Even if you have excellent processes the above points will usually still apply. It's just part of the business. You can go along way in helping yourself by just becoming aware of these pain points.

A good way to help is to maintain a good work life balance. Don't over work and don't over goof off. Learn to push through the resistance but also learn to recognize when you just need to take a break. Sometimes you just need something to break up the routine. Once in a while start earlier than normal or wait till after lunch to get started. Go out for a bike ride before work or maybe just take a random unplanned day off.

Ultimately I stay in software development mostly because it is a challenge, not because I love it everyday. Of course there are days when I'm having a complete blast but that is not the majority of the time. The point remains, software is hard.

Saturday, October 18, 2014

Managing Web Complexity

So the wonderful world of the web technologies is great. HTML5, CSS, JavaScript, you get the idea. You can build many wonderful things with these. Enter a tool like jQuery, and you can do things even easier and better. You have plethora of JS frameworks to help you build a website or a web app... Backbone, Angular, Ember, etc. I believe these tools & technologies work well for a large swath of web targets. However, not all. I'll breakdown how I segment the web to make this more understandable. I think the web falls into the following three categories with some overlap in between.

Websites are documents. In fact they are primarily what the web tech family was designed for. It fits well with the DOM concept. It was to facilitate a cross-platform way to display information to the world in a quasi standards based way. This does not mean the content is static but rather interact with the user to a certain extent. Dynamic content may displayed with JS or PHP. But the overarching idea of this category is to get information out. Examples are corporate websites & personal websites.

Websitey apps
This is when websites begin to function more like an application. I don't particularly like the term "websitey application" but that's all I can come up with now. In this category, the "websitey application" is very interactive. User's have accounts, they do persistent actions as in order stuff, change stuff, add content, etc. This is the beginning of where the concept of the UI being a document begins to get a little shaky but still holds up decently well. Examples would be Amazon, eBay, Drupal & Wordpress sites, and really any site with ecommerce. Web tech & tools hold up pretty well for most of these as the complexity is still limited however the document or DOM concept begins to show it's weaknesses when the application is larger. As I said before, the DOM was intended for displaying content, not really for complex dynamic & interactive scenarios. At this point JavaScript becomes somewhat of a hack to manipulate the DOM into doing what needs done.

Enterprise business apps
These are full screen business tools. The concept of "website" is just not adequate. The app takes the full browser area, no header or footer, just a menu system, and a work area. These are really web implementations of what back in the day would have been full screen desktop business applications. Examples would be QuickBooks Online, Xero Accounting, Flex Rental Solutions, and thousands of other applications that fly unknown under the cover of the enterprise. The idea of directly using many of the web tools & technologies really shows their weaknesses here. Why? Complexity. The large enterprise app has a tremendous domain size and structure. This results in large numbers of components on the UI side. It's important in this area that the UI layer be self documenting in the form of packages (namespaces), component names, inheritance, and lot of the concepts on the backend that make development sane. Without structure, it literally becomes hard to find the component that you need to work on. In JS, there is no concept of class per file, so you can have many "objects" embedded within one JS file. That leaves you to a basic text search & find to be able to find your data (DTOs) & UI components. Bottom line, the large enterprise can implode in this space if a way is not found to manage complexity. And no Backbone is not always sufficient for this.

So what's the solution for complex web apps? Honestly at this point I'm not totally sure. In abstract terms, the solution is a tool that abstracts away from web technologies that allow you to code to an API instead of the direct implementation. This might mean using an enterprise proven language to compile to JavaScript such as GWT.  I see web technologies as a low level "language" to be built on top of to unleash it's true power. Yes, there are drawbacks to this, but I think the good far out weights the bad. Just like in the sense that there are probably advantages to writing certain solutions in assembly language, but in general it's a given that higher level languages offer a tremendous productivity benefit.

However, I want to look further. I find the concept of web abstraction compelling but I'm squeamish if it's really the long long term future.  As in will it work well in 10 years. I feel that maybe the root problem of doing large Enterprise web apps is the DOM. I watched a youtube video (watch it, it's good) the other day about web frameworks and one guy made the comment something along the lines of "if you were to build a UI toolkit today, it would look nothing like the DOM". I find that fascinating. I feel the DOM is kinda a misfit for web applications but it has so much inertia going for it now, that it may be years before people start to seriously question the DOM.

Post-DOM is going to take a long time. I could easily see it being close to a decade before it reaches critical mass. In the meantime, I want to see awesome web abstraction tools. I think Vaadin & GWT have a compelling use case. I have reservations about Vaadin latency & performance but I think it's UI concept is solid.

To me, any good UI toolkit needs to be powerful but yet at the same time kinda just fade into the background and allow you to focus on solving business problems. You should not have to think about browser idiosyncrasies, compatibility issues, and structural complexity.

Cheers to awesome web abstractions, managed web complexity, and the post-DOM way of building internet applications whatever that may be!

Friday, April 25, 2014

The Programming Dilemma

Software is code. Code is structure. Structure is meaning. Meaning requires structure. Almost by definition, structure does not change or adapt or otherwise it would lose meaning. So, in order to simulate "adaptable-ness", you have to make it configurable. But there is still structure, because there are still unchanging structures that information must flow through.

You could in theory make software completely dynamic (as in get or send anything from or to a system) but then you would have to do some kind of "programming" on the client side in order to use it. For example, say I want to make it possible from the client side to be able to get any information possible out of my software system. One option is just make every piece of data available in every possible known combination (a herculean task). Or create some kind of "query" language to get the data out of my system. In essence, the application just becomes a domain specific language on top of your database.

Programming is supposed to make it easier to to do things. Users want to point and click to make things happen. They don't want to write code to make it happen. But then people want things to adapt to their needs. So you add configuration. The tradeoff is that now the system is harder to understand how to use. Taken to the extreme for flexibleness, your application becomes a maze of knobs and switches. Which oddly enough, to setup, feels a bit like programming something. So you end up programming the program. I thought the point of programming was to not have to program?

I've come to the conclusion that an application is really only 100% useful for one company max. As soon as you involve more than one company, there are differing opinions on how things should be done. There are differing opinions within a company, but a least at a company level, policies can be adjusted. To support more than one company, you have to either be dogmatic and say "you have to do it this way" or make a configuration option. As soon as you begin down the road of configuration, it becomes harder and harder to know how to use the software. It's a problem that if is not checked will bring the software to it's knees and it will die a slow hard death.

Programming doesn't really solve problems very well. It solves one problem and creates ten other ones. It is inherently self-defeating. I've said it before and I'll say it again, "software is hard", and yes it always will be.

P.S. I think it is wise to keep an eagle eye out for what NOT to program rather than what to program. There is no shortage of things to program. It takes gutsy, courageous leadership to say "we are not going to program that because it will create to many additional [programming] problems".

Tuesday, June 11, 2013

Getting your iPhone 4,4S, 5, or 5S fully working on Straight Talk

If you want to use your iPhone 4, 4S, 5, or 5S on Straight Talk, it's usually pretty easy, or I should say was easy (as of this writing I don't think Straight Talk is offering any AT&T compatible SIM cards for purchase on their website). All you normally would have to do is install the correct configuration profile from http://www.unlockit.co.nz/unlockit/ and voilĂ  you had data. But not full MMS, only iPhone to iPhone over iMessage.

Since I'm picky when it comes to my phone, not having full MMS functionality did not sit well with me. The solution I came across was to use a t-mobile sim to activate the cellular data settings. The process I used is detailed below.

First some preliminaries:
  1. Get your iPhone factory unlocked. Either through AT&T here or by purchasing a factory unlock service on eBay. I had to do the eBay unlock service because the AT&T unlock didn't work for me. Either way, once one of those unlock methods work, you'll need to restore you phone in iTunes for the unlock to be effective. It will tell you that your phone is unlocked when the restore completes. I believe it is required in order for the Cellular Data Settings to come up when you use the t-mobile SIM later.
  2. Get an inactive T-Mobile micro SIM. eBay is great way to get one, just make sure it is a micro SIM, not a standard size SIM. Otherwise you will have to cut down the standard size sim to micro sim size, so save yourself the trouble and purchase a MICRO SIM.
Now for the actual process:
  1. Turn off wifi. Keep your existing APN configuration profile in place. It's good to have it there just in case the APN settings reset randomly. This has happened to me a few times.
  2. Open the Notes app so that it is in the background for later.
  3. Insert T-Mobile SIM into iPhone. Wait a few seconds and go to Settings > General > Cellular > Cellular Data Network. If you don't see the Cellular Data Network menu item, it might be because your phone is not unlocked.
  4. Switch to Notes app. Do this by double tapping home button and select the Notes app. Stay in the Notes app.
  5. Insert Straight Talk SIM. It's important that you stay in the Notes app for at least a minute as different things flash up in the upper left corner of the screen. Wait till it says HOME and stays on HOME.
  6. Switch back to Settings. Do this by double tapping home button and selecting Settings app. Settings should still be on the Cellular Data Network screen.
  7. Enter Straight Talk settings. The settings here should do the trick.
  8. Go back to General. Hit the back button in the upper left corner to get back to the General screen. Your settings should now be saved.
  9. At this point your phone should be good to go but you may need to reboot the phone for everything to take effect.
Some other things to think about:

You may find that updating to a new version of iOS, even if it is only a minor version upgrade will reset the settings. If that happens, just follow the process above again. Also, I would be wary about upgrading to the next major version such as iOS 7. There's no guarantee this workaround will still work.

Hopefully this process works for you, it did for me. However, if it didn't, you can always revert back to installing the Straight Talk profile from http://www.unlockit.co.nz/unlockit/ and still have data & MMS iPhone to iPhone.

Update 11/05/2013:
As of the iOS 7.0.3 update my Straight Talk SIM just works, no APN settings or config profiles needed. I still keep the APN config profile installed just in case. This is for the newer LTE SIMs, I don't know if this works for the older ones.

Straight Talk has AT&T compatible SIMs for sell now.

Thursday, June 7, 2012

Font Extensions

One issue in a system that I work on that was kind of side stepped around for awhile is the issue of custom fonts in reports, specifically reports built with the Jasper Reports framework. For a while we were just building all our reports in a standard font which was included in the build so this worked fine.

However, when we build custom reports for customers, they sometimes want text to be in specific fonts to match their branding. This is a completely reasonable requirement, but we just needed a sure fire way to deal with it.

In past times we would just take the font and install it on the customer's server however this was brittle since as soon as you would move the customer to another server, it would break their custom reports. Also this approach was problematic in rendering the PDFs correctly on any machine.

So the best way is to just include the fonts in the build. Fortunately Jasper Reports includes something called font extensions that make this very easy. Basically you just put a properties file in the default package called "jasperreports_extension.properties" and Jasper will automatically find the file. This file just sets a couple properties and tells Jasper where to find the font XML configuration file.

Example jasperreports_extension.properties:


Next you create a "fonts" directory, place your True Type Format fonts in there, and create an XML file that configures the fonts.

Example XML font config file:

xml version="1.0" encoding="UTF-8"?>

   <fontFamily name="NewsGothicBT-Roman">
       <normal>fonts/News Gothic.ttfnormal>
       <bold>fonts/News Gothic Bold.ttfbold>

   <fontFamily name="Planer">


Jasper reads the above XML and registers the fonts. Whenever the above font names are referenced in a report, the TTF fonts are used and embedded in the PDF itself (notice pdfEmbedded=true above) so that the PDF will be rendered correctly on any system that opens it.

So from now on whenever a custom font is needed, we just drop the TTF font files in the fonts directory, configure them in frsfontsfamily.xml and they are available everywhere.

Our next step might be to create a way to dynamically load fonts onto a live system allowing for more flexibility but for now we are happy with being able to include fonts in the build guaranteeing that they will be available and embedded in the PDFs.

Monday, February 27, 2012

Transparent Field Encryption/Decryption

Recently I have been working on expanding FRS integration with QuickBooks to include the Online Edition. While working on the transport pipe for QBOE (QuickBooks Online Edition), a realized requirement was the need to store a QBOE connection ticket. Something like this is usually a very simple task involving adding a new field to the appropriate domain object.

However in this case, one of the security rules that QBOE requires of an application is to store the connection ticket in an encrypted state. This is a common security requirement. Since the ticket acts as a key to QBOE, we don't want anybody with database access to get their hands on the actual ticket.

The first thought was to simply create a utility class to handle the encryption/decryption, possibly in a getter or setter. Another thought was to create a custom hibernate type that would handle the encryption/decryption by making use of an encryption utility class.

Upon doing some searching I came upon this: http://www.jasypt.org/  Basically Jasypt is just a java library that makes encryption extremely easy at the data layer. In fact all you need to do is get the appropriate jars on the classpath and do just a little configuration in the hibernate configuration, and voilĂ  encryption/decryption just works.

Here is an example of setting up the custom type in the hibernate configuration:

<typedef name="quickbooksEncryptedString" class="org.jasypt.hibernate3.type.EncryptedStringType">
      <param name="algorithm">PBEWithMD5AndDESparam>
      <param name="password">encyption-passwordparam>
      <param name="keyObtentionIterations">1000param>

And the corresponding property mapping:

<property column="qb_online_connection_ticket" name="quickbooksOnlineConnectionTicket" type="quickbooksEncryptedString" length="1000"/>

Besides getting the Jasypt jars on the classpath, that is really all there is to it. With the above configuration the "quickbooksOnlineConnectionTicket" property is automatically encrypted when it is saved to the database and then automatically decrypted when it is retrieved. So from a coding perspective nothing changes. You can simply "set" the ticket and "get" the ticket whenever you need to.

Friday, May 13, 2011

School Update

I haven't blogged for a long time on the WGU school thing (in fact I haven't blogged much at all recently due to much busyness) but it has been coming right along. I am down to my final six classes and it appears I will be done this fall. It has been quite the ride. I have covered a ton of material in the last year and a half.

Two of the six classes remaining involves the final capstone project. My intention is to design & complete a small iPhone app to fulfill the capstone requirement. I am looking forward to getting into the world of mobile applications... it's been a long time coming.

Overall, WGU has been a good experience. I have gained a bunch of IT certifications one of which was the Sun Certified Java Programmer (SCJP). I had been wanting to achieve that one for a long time prior to even starting school. I have expanded my world quite a bit with general education studies. At times it was easy to fight their relevance, but I think they have a valid purpose... at least most of them anyway.

I am looking forward to completing the degree this fall. After that, I intend to utilize the degree as a foundation for my further pursuits in the field of software engineering. It's hard telling where the journey will lead.