Run Hard and Put Away Wet
This is a story that takes place in Q3-4 of 2012 and has to do with the creation of a reconciliation and effectuation system for a healthcare company.
Who were we?
4 XP developers who’d just finished a successful 6 month gig working together for another company.
1 XP developer from the same company who had the same working expectations and style.
1 new to the company senior tester
1 new to the company CSM
What was the task?
The healthcare provider needed an effectuation and reconciliation system to handle new subscribers coming in from the state run health care exchanges. They’d come up with some very high level specifications outlining four similarly sized workflows/pathways for data to flow from the exchanges into their internal systems. The seven of us were tasked with creating and getting it deployed onto their hardware by the end of the year. We started assembling and kicking off mid-July of 2012 with the full team assembled and working on the task at the beginning of August. Fixed scope, fixed deadline….
Pretty simple math. Four distinct blocks of work. Quick delivery of value means one workflow per month and a half, right? Wrong! It took us four months to deliver our first workflow through the system.
There were a lot of complications from the beginning. The requisition of hardware was slow and complicated: from the initial delivery of ‘office machines’ instead of developer systems, to no local admin privileges, to getting access to the dev/test/production environments that we would need to deploy our services. Not only did we need to get access to those environments, it was difficult to find people who knew who to contact about accessing these unknown environments.
There were complications around the workflows and the standard contracts between the healthcare exchanges too. The right people to answer our questions were not gathered and in some cases, didn’t even know they needed to talk/plan/work with us.
No one on the team had the domain knowledge of the healthcare providers internal systems, or database abbreviations for “standard healthcare terms”.
There were internal issues with the team as well. We stormed and formed and then stormed some more. We stormed around the role of test on a cross-functional XP team. We stormed around work being doled out to developers by the scrum master, the team’s ability to self organize, and create/pull work as needed to deliver tasks. We stormed around external pressure to have more design up front.
Add all those up and you start seeing this “hot shot team” that’s supposed to deliver a workflow every 1.5 months not looking so hot. Almost brown trouser time, as it’s a fixed deadline for the end of the year that the systems need to be up and running to accept policies from the healthcare exchanges. Us not being able to deliver a workflow incurred its own costs too. More oversight, more pressure to deliver, and behind means more meetings about why you are behind. Questions arise about if this whole XP thing will work (from the non-XP members of the team).
Needless to say, it got messy and there were heated discussions all around.
At the end of four months we delivered the first quarter of the project and heads exploded. It was the end of October and we only had two months left to get the rest of the work done. We tried to talk sense and bring up what we’d been saying all along...there are a lot of complications that are keeping us from delivering. In truth it was about two months of getting setup, requirements gathering and domain ramp-up and two months worth of work, all woven together in the previous four months. That knowledge didn’t make people feel much better because the math still didn’t work out. In the end, the ship had sailed and they were along for the ride anyway and had to trust the team when they said, “we got this”.
We got this...but…
We delivered our first workflow at the end of October. It had taken us approximately 15 weeks to deliver this first workflow. We’d done it TDD with emergent design, but there had been a few starts/stops/reworks in the process. It was solid code with complete test coverage and a full CI/CD system in place. We weren’t done with it though, we knew that there was cleanup that needed to happen and so we opted to do something that was very unpopular, to say the least. We requested and took two weeks to refactor and stabilize our code for the addition of the three upcoming workflows. To say that it was an unpopular choice is putting it lightly. We were behind schedule and there was no trust that we could deliver everything else in time, but the developers were strong and we got our two weeks to refactor. We did have to concede that a pair of us would break off and start gathering requirements on the next workflow though.
That two week refactor was tough. We did some of the standard refactoring removing duplication, extracting commonality, and renaming. Our biggest refactor involved a two-day discussion/fight between the developers about what pattern to implement which ended up being a Builder Pattern favoring composition over inheritance. It took us about a week to implement our first pass at the builder but in the end it resulted in a lot less code, so we knew that we were going along the right path.
After our two weeks of refactoring our next workflow took us only two weeks to deliver, we were now at the end of November. The switch to using a Builder and composition was paying off. We were able to leverage large parts of the existing codebase and infrastructure to do what needed to be done. The surprise on the product owners was amazing. They saw the math was finally on their side. In theory it would take us four more weeks to get the next two workflows and they were happy!
Of course we were happy, but not happy enough. We had to break it to the business that we needed another two days (or so) to do another refactor of the code we’d just written. They were confused at why we’d need to refactor again so soon, but given that it looked like we were going to hit our deadline, they were willing to try out another “refactor”.
This refactor was pretty straight forward. We were able to go through and extract, unify and segregate the two workflows very nicely. Which again reaffirmed our switch to a Builder Pattern.
We dove right into the next workflow and were able to deliver the next feature in two days. Which made the business even happier, because now we’d be delivering early and they’d have more time to test our work.
As you can probably guess, we weren’t happy with our code after that two day development. We dove into do another quick refactor and extraction that took about a day. This final refactor allowed us to deliver the final workflow in two hours of work.
We were done. We’d delivered our assigned task almost a month early. That time was spent with the code in front of their dedicated QA teams where they found a few minor defects, which were missed or unclear requirements, which aren’t actually code defects, they are missed requirements...and easily fixed. The QA team was definitely not used to bugs being fixed in minutes/hours and re-deployed so quickly. We even helped them a little and educated them about TDD and invited them to pair/collaborate with us so that they can see that their manual regression scripts were mostly useless for our code.
There was one major defect found, though. It was a performance issue under load. It took us one day to research, plan the restructure and four hours to implement. Essentially taking a single threaded process of reading XML and pushing that data structure into a database and making it multi-threaded.
The Cost
One of our XP developers was on antidepressants
Two XP Developers quit
One team member was let go
One happy customer
We’d delivered to the client. Yay! They were mostly happy. They were happy they delivered early, but they had a bad taste in their mouth from our four months to deliver the first workflow. They figure if we can deliver new workflows in two hours now, why did it take us four months for the first one?
Five years later developers in that company still use that code base as an example of how to write good code
The customer attributes success of that project to their management and extra attention given it
One of the most easy-going and level headed developers I know lost it with another teammate who was refusing to get on board with XP development. I had to physically interrupt the verbal altercation and go for a walk with that developer.
I am the only one of these seven people that talks about this project in public.
With the core XP developers on the team that I’m still in touch with, we agree we wrote some amazing code on that project.
I can still remember the deployment of the product to production. It happened the same time as the consulting company’s holiday party that was held at the Chateau Ste. Michelle Winery. There I was at the company holiday party, being billable, with a candle lit laptop on top of a wine cask table nursing a delivery/deploy. The only thing that required my attention during the 90 minutes of deployment was to remind the Ops crew to manually change the settings to “prod”. They had refused to automate this change and I warned them that making this manual change could lead to problems. Of course they forgot to make the change during deployment.
In Retrospect
It took us a long time to deliver that first workflow all the way through their system. Four months is way too long. We were trying like hell to get something delivered faster, but it was always the case of trying to reach the summit and coming to the top of the hill only to discover that there’s another hill you have to climb. There was no consideration to onboarding, learning the ecosystems and domains, getting permissions for those ecosystems in a highly regulated industry. How do you counter that? Be meticulous and take the time to create a master plan that takes every consideration and every conceivable issue into account. You know what they say about plans and first contact with the enemy right? No plan survives first contact with the enemy! - Helmuth von Moltke the Elder. We didn’t know what we didn’t know. Only by doing the work we know did we discover the work that needed to be done.
Stakeholder delivery/mindset shift?
Most stakeholders only care about tending the spreadsheet to make sure you are on track. They are not “bought in” on delivering small bits of value and iterating on it to expand functionality or make it better. They just want their designed bit of work done. They plant seeds in the garden and expect perfect produce to happen in 2 months with no extra work. Without the effort to tend and modify how the garden is growing in between.
