We released last night. A critical fuction failed today (the ability for agencies to opt consumers out of a particular piece of marketing), with today being the deadline to complete that opt out process. Significant business impact.
After some mad scrambling, and general confusion because none of the relevant code had changed between the last release and this one, we discovered that the problem was completely unrelated to the new release. It was just a coincidence of timing.
No, the problem was entirely a logic error on my count. I handled for when the commit deadline is later than today. I handled for when the commit deadline was earlier than today. What I failed to handle for was when the commit deadline *was* today. That's a rookie mistake if ever there was one, and I am thoroughly embarrassed to have made it.
The immediate fix in production was to change the deadline to tomorrow. The proper code fix is being compiled right now for a rapid hotfix deployment.
I'm going to go crawl under a rock now.
After some mad scrambling, and general confusion because none of the relevant code had changed between the last release and this one, we discovered that the problem was completely unrelated to the new release. It was just a coincidence of timing.
No, the problem was entirely a logic error on my count. I handled for when the commit deadline is later than today. I handled for when the commit deadline was earlier than today. What I failed to handle for was when the commit deadline *was* today. That's a rookie mistake if ever there was one, and I am thoroughly embarrassed to have made it.
The immediate fix in production was to change the deadline to tomorrow. The proper code fix is being compiled right now for a rapid hotfix deployment.
I'm going to go crawl under a rock now.
Tags:
From:
no subject
From:
no subject
O
L
From:
Stop it...
Do not beat yourself up.
There was no opportunity to transition as I was escorted out the door. I tried desperately to get Ryan to allow me to stay through last Friday so I could outline the pieces parts for whomever took over after I left, but was informed: 'We no longer need your ideas, they are too negative.'
While in the HR person's office, I gave them a quick laundry list, which he appeared to be writing down, but apparently something got lost in translation.
I am so sorry!
Please place the blame on me!
As I frequently told you, you are awesome!
Remember: we were always working with the 'pre-commit' delaying dealing with the 'post-commit' timeframe.
What I would anticipate is that there will be other gotchas that will be encountered as we were never allowed the opportunity to address the ongoing operational factors that are necessary for sustainment activities.
Just because I like you and care about the folks that truly matter, if there is something you think I might know from memory, feel free to contact me.
Best wishes as you move into your well-deserved new role!