![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
So the (insert boring technical description here) stopped working in production. It was working fine four weeks ago when the last version was released. The code had not changed in production, so it had to be something else. Having it broken is causing a serious blockage for the business owners, and it needed to be addressed rapidly. Of course, when the issue was raised yesterday it was Kathie who was coordinating the effort to fix the problem. My first thought was that a new version of the vendor data had been deployed, and probably some change there was effecting our code. I went to lunch thinking we had a solid plan to resolve the issue, and would touch base with Kathie in 90 minutes or so. Apparently thirty minutes later she was fired.
So pretty much nothing got done to fix it yesterday, and my first gut feeling proved to be wrong. I set about this morning to fix it. Unfortunately, fixing it entails stepping through it to narrow down which specific component is failing. Each pass takes a half hour or more due to the very long call to the database to get the list of tens of thousands of consumers. Each pass I eliminated one possibility, but the progress was tedious. Something that I thought I could have fixed in under an hour went on to consume six full hours of my day. In the end I figured out which column in the data was causing the problem, but couldn't tell why until I manually scanned through 21,500 records to examine that column and look for anything that stood out as... odd. It's hard to look for something when you don't know what you are looking for. You just have to let your mind glaze over and trust your instincts to take notice when something outside of the normal pattern shows up. It turned out that three records (out of twenty one thousand five hundred) were returning a null value.
Conferring with my database resource, I discovered that those particular records came from a new tool that was deployed (you guessed it) four weeks ago. Prior to that tool, the value in that column would never, ever be null. But all of a sudden it became a possibility, and over the course of four weeks a few odd records crept in. I adjusted the code to handle for a null, but I couldn't run it on my own machine to test because my vendor data was out of date. It was so late in the day, I don't even know whether or not it was able to be tested by QA. I suspect that it will be Monday before the business owners become unblocked. Oh, well. I did the best I could.
Unfortunately, by doing that I put myself a full day behind in the other work I was supposed to be doing today. I had hoped to complete the last of the image upload component today, so that QA could begin testing it on Monday. I mean heck, it's only supposed to deploy on Thursday. No rush, right?
Aside from that I rattled a few cages today. I had a very long conversation with someone in HR about the current situation in the tech department. I don't think the people in the upper levels of the company fully appreciate just how poisoned the atmosphere is in the tech department these days. Some of that is unavoidable, following a major layoff. But much of it is, in my opinion, *completely* avoidable just by proper communication. Right now everybody I work with on a daily basis is actively updating their resume and considering other options. There is a pervasive feeling that perhaps we are in fact being set up to fail spectacularly so that the entire department can be shut down and outsourced. It's ugly, and it's unfair.
Personally I love this company, and have every intention of retiring from here if they let me. It pains me to see things so poorly executed. I only hope that things will eventually improve. As far as my position is concerned, four more working days and I am done with my commitments to the existing projects. Once we deploy on Thursday I will be freed up completely to focus entirely on my new role in the R&D group. My new team includes a business analyst with whom I have greatly enjoyed working in the past, and a database person who has garnered my utmost respect. I'm a little iffy about the director for the team, but I recognize his technical skill and I am trying to give him an honest chance to make a better second impression on me. Once I dive into the new team, I expect that things will get much better for me. There is much to do before the end of the year, rolling out new tools and moving us into Visual Studio 2005 and .NET 2.0. I just fear that I am going to lose more valued friends along the way. I can think of two right off the top of my head who I pretty much expect to be gone by Thanksgiving. That pretty much sucks.
Oh, and it's after 11:30 at night and Ben is still awake. Probably for another two or three hours. My head hurts.
So pretty much nothing got done to fix it yesterday, and my first gut feeling proved to be wrong. I set about this morning to fix it. Unfortunately, fixing it entails stepping through it to narrow down which specific component is failing. Each pass takes a half hour or more due to the very long call to the database to get the list of tens of thousands of consumers. Each pass I eliminated one possibility, but the progress was tedious. Something that I thought I could have fixed in under an hour went on to consume six full hours of my day. In the end I figured out which column in the data was causing the problem, but couldn't tell why until I manually scanned through 21,500 records to examine that column and look for anything that stood out as... odd. It's hard to look for something when you don't know what you are looking for. You just have to let your mind glaze over and trust your instincts to take notice when something outside of the normal pattern shows up. It turned out that three records (out of twenty one thousand five hundred) were returning a null value.
Conferring with my database resource, I discovered that those particular records came from a new tool that was deployed (you guessed it) four weeks ago. Prior to that tool, the value in that column would never, ever be null. But all of a sudden it became a possibility, and over the course of four weeks a few odd records crept in. I adjusted the code to handle for a null, but I couldn't run it on my own machine to test because my vendor data was out of date. It was so late in the day, I don't even know whether or not it was able to be tested by QA. I suspect that it will be Monday before the business owners become unblocked. Oh, well. I did the best I could.
Unfortunately, by doing that I put myself a full day behind in the other work I was supposed to be doing today. I had hoped to complete the last of the image upload component today, so that QA could begin testing it on Monday. I mean heck, it's only supposed to deploy on Thursday. No rush, right?
Aside from that I rattled a few cages today. I had a very long conversation with someone in HR about the current situation in the tech department. I don't think the people in the upper levels of the company fully appreciate just how poisoned the atmosphere is in the tech department these days. Some of that is unavoidable, following a major layoff. But much of it is, in my opinion, *completely* avoidable just by proper communication. Right now everybody I work with on a daily basis is actively updating their resume and considering other options. There is a pervasive feeling that perhaps we are in fact being set up to fail spectacularly so that the entire department can be shut down and outsourced. It's ugly, and it's unfair.
Personally I love this company, and have every intention of retiring from here if they let me. It pains me to see things so poorly executed. I only hope that things will eventually improve. As far as my position is concerned, four more working days and I am done with my commitments to the existing projects. Once we deploy on Thursday I will be freed up completely to focus entirely on my new role in the R&D group. My new team includes a business analyst with whom I have greatly enjoyed working in the past, and a database person who has garnered my utmost respect. I'm a little iffy about the director for the team, but I recognize his technical skill and I am trying to give him an honest chance to make a better second impression on me. Once I dive into the new team, I expect that things will get much better for me. There is much to do before the end of the year, rolling out new tools and moving us into Visual Studio 2005 and .NET 2.0. I just fear that I am going to lose more valued friends along the way. I can think of two right off the top of my head who I pretty much expect to be gone by Thanksgiving. That pretty much sucks.
Oh, and it's after 11:30 at night and Ben is still awake. Probably for another two or three hours. My head hurts.