“How many focused hours of work do you put in each day as a software engineer?”
The phenomena described in these stories ISN’T imaginary but it ISN’T unique to the “FAANGs” of the world either. It is happening across virtually all Fortune 500 companies that have any internal teams attempting to build and maintain HR systems, data warehouse systems, customer portal systems for support and online billing, and agent portals to provide customer service.
These inefficiencies are occuring for a combination of reasons.
First, classic Moore’s Law improvements in processing power and data storage densities have provided new data analytics capabilities against TERABYTES of data that were not economically practical 10-15 years ago. That produced a “step function” change in application requirements to make use of this power for bulk analytics and real-time processing. That created the need for new types of algorithms to provide solutions to use that power and languages and libraries to more optimally express those algorithms.
That sea change in computing power and algorithms created a nearly instant gap in developer / tester skills where existing staff famililar with writing .Net code in the 1990s or Plain Old Java Object (POJO) code of the 2000s had no experience with newer languages and frameworks. Whatever skill gap exists at the worker bee level was magnified in management ranks, generating 2-3 entire layers of management with ZERO grasp of the actual effort required to create useful functionality using these new tools. When faced with such a gap, management and worker bees alike err on the side of caution when asked to estimate how many person-hours and elapsed days it will take to accomplish X.
What is the impact of this knowledge gap on actual work? All labor estimates become inflated by EASILY a factor of TEN OR TWENTY. If you’re not sure how long it will take you to do X, you will want to make DAMN sure you understand what is meant by X so you insist on not only having the business provide you “requirements” but you pay your own department to develop requirements from the requirements (I’m not kidding…). And those requirements gatherers and “business analysts” typically have ZERO “domain expertise” in what ever is being built. They would be equally ill-suited to collect requirements for a new payments portal or a new Milky Way factory quality control system or a nuclear power plant.
Then you have architects review both sets of requirements and formulate a “High Level Design” to outline the entire solution. But do you trust the architects? Heck no, you have your own “solution” architects analyze the HLD and nail it down with even more specificity who then hand over the detailed design to the developers who haven’t been involved with the project since it started three months ago. Then they review the detailed design and come back with a development estimate.
How insane is the resulting process?
One of the more common tasks in software development is to create a “web service” – a small, standalone process that performs typical CRUD (create / retrieve / update / delete) functions against a table in a database housing information related to some business “object” like an account, a project, a payment, etc.
Web services have been predominately written in Java since about 2000 and the Java language has gone through various phases / fads with different libraries that simplify creating web services. The current favorite framework is something called Spring Web Services and a related library called Spring Boot that makes it absolutely child’s play to create a web service that can run in any modern, scalable environment via Docker, Kubernetes, etc.
Use of Spring boils most of the work of creating a web service down to simple template-based source files. An entire “source tree” – the list of source files that implement a service – of a functional web service for fetching project data looks like this:
wth@fedora1:~/gitwork/kuberdepends $ find ./src/main/java
./src/main/java
./src/main/java/com
./src/main/java/com/wthlabs
./src/main/java/com/wthlabs/depends
./src/main/java/com/wthlabs/depends/dao
./src/main/java/com/wthlabs/depends/dao/ProjectsDAO.java
./src/main/java/com/wthlabs/depends/DependsConfiguration.java
./src/main/java/com/wthlabs/depends/services
./src/main/java/com/wthlabs/depends/services/ProjectController.java
./src/main/java/com/wthlabs/depends/models
./src/main/java/com/wthlabs/depends/models/Project.java
wth@fedora1:~/gitwork/kuberdepends $
That looks like 11 different files but only the lines with a filename ending in .java reflect actual source code. How much code are we talking about?
wth@fedora1:~/gitwork/kuberdepends $ ls -lR ./src/main/java | grep "\.java"
-rw-rw-r--. 1 wth wth 2890 Mar 14 2022 DependsConfiguration.java
-rw-rw-r--. 1 wth wth 9717 Feb 28 16:49 ProjectsDAO.java
-rw-rw-r--. 1 wth wth 5687 Mar 14 2022 Project.java
-rw-rw-r--. 1 wth wth 5961 Feb 26 15:57 ProjectController.java
wth@fedora1:~/gitwork/kuberdepends $
A total of 24,255 bytes. About 652 lines of code and comments. (Full disclosure… No, this example does not include unit test code which “real” code should for automated testing. Shame on me.) For this service,
- Project.java – defines the “object” of a project as saved in a DB
- ProjectsDAO.java – defines logic used to interact with the DB
- ProjectController.java – defines the http endpoint of each action for create, retrieve, update, delete
- DependsConfiguration.java – defines shared variables identifying how to connect to the DB
A service like this took me two or three days to figure out how to write the first time I attempted to use Spring Web Services and Spring Boot. Once a service is written using these libraries, writing any other web service for a similarly complex object (account, payment, order, etc.) with similar create / retrieve / update / delete actions is primarily a matter of creating a new object class (Project.java in this example) then updating the SQL statements in the DAO file to match the new database table and column names for each field in the object. It should be HOURS of work. Maybe 10-20.
In my last job in Corporate America, developers and their “solution architects” would typically estimate TWO to THREE MONTHS of development time to create web services using this framework. And those estimates came after my team of enterprise architects would have already outlined the data model, identified all of the requirement methods and summarized the exact REST-standard HTTP endpoints to be implemented by the new service.
And these delivery estimates were ACCEPTED and incorporated into project plans because no one else in middle management of the “delivery team” or the “client organization” had any better understanding to know this work should have only taken a WEEK or two to deliver. And of course, when work takes that long to deliver, it increases the chance “the business” will change or expand requirements, creating more churn through design into development which becomes an excuse for MORE delay and the productivity spirals downward from there.
There is a trope within the software development field regarding something called the 10x developer. This is a person who is TEN TIMES more productive than not only the weakest developer on the team but even the AVERAGE developer. It seems like hyperbole but the reality is it is not. Things might have been slightly better at the FAANGs with their previously attractive / sexy work environments but, in the typical Fortune 500, a team with say 30 developers probably has 5-6 that are doing eighty percent of the real heavy lifting, another 5-6 might assist with coding tests and assisting with test automation and the rest will appear to provide “support” for older services still in use but will do virtally zero new development work. And often those unable to continue doing real development move UP to become “solution architects” and muddle communications between project teams and the few developers doing the real work.
Needless to say, for those who are actually competent and productive at what they do, it is frustrating beyond explanation to work in companies who have not properly trained both developers and managers on the proper current tools and continue to accept this level of incompetence.
WTH