![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
So the (insert boring technical description here) stopped working in production. It was working fine four weeks ago when the last version was released. The code had not changed in production, so it had to be something else. Having it broken is causing a serious blockage for the business owners, and it needed to be addressed rapidly. Of course, when the issue was raised yesterday it was Kathie who was coordinating the effort to fix the problem. My first thought was that a new version of the vendor data had been deployed, and probably some change there was effecting our code. I went to lunch thinking we had a solid plan to resolve the issue, and would touch base with Kathie in 90 minutes or so. Apparently thirty minutes later she was fired.
So pretty much nothing got done to fix it yesterday, and my first gut feeling proved to be wrong. I set about this morning to fix it. Unfortunately, fixing it entails stepping through it to narrow down which specific component is failing. Each pass takes a half hour or more due to the very long call to the database to get the list of tens of thousands of consumers. Each pass I eliminated one possibility, but the progress was tedious. Something that I thought I could have fixed in under an hour went on to consume six full hours of my day. In the end I figured out which column in the data was causing the problem, but couldn't tell why until I manually scanned through 21,500 records to examine that column and look for anything that stood out as... odd. It's hard to look for something when you don't know what you are looking for. You just have to let your mind glaze over and trust your instincts to take notice when something outside of the normal pattern shows up. It turned out that three records (out of twenty one thousand five hundred) were returning a null value.
Conferring with my database resource, I discovered that those particular records came from a new tool that was deployed (you guessed it) four weeks ago. Prior to that tool, the value in that column would never, ever be null. But all of a sudden it became a possibility, and over the course of four weeks a few odd records crept in. I adjusted the code to handle for a null, but I couldn't run it on my own machine to test because my vendor data was out of date. It was so late in the day, I don't even know whether or not it was able to be tested by QA. I suspect that it will be Monday before the business owners become unblocked. Oh, well. I did the best I could.
Unfortunately, by doing that I put myself a full day behind in the other work I was supposed to be doing today. I had hoped to complete the last of the image upload component today, so that QA could begin testing it on Monday. I mean heck, it's only supposed to deploy on Thursday. No rush, right?
Aside from that I rattled a few cages today. I had a very long conversation with someone in HR about the current situation in the tech department. I don't think the people in the upper levels of the company fully appreciate just how poisoned the atmosphere is in the tech department these days. Some of that is unavoidable, following a major layoff. But much of it is, in my opinion, *completely* avoidable just by proper communication. Right now everybody I work with on a daily basis is actively updating their resume and considering other options. There is a pervasive feeling that perhaps we are in fact being set up to fail spectacularly so that the entire department can be shut down and outsourced. It's ugly, and it's unfair.
Personally I love this company, and have every intention of retiring from here if they let me. It pains me to see things so poorly executed. I only hope that things will eventually improve. As far as my position is concerned, four more working days and I am done with my commitments to the existing projects. Once we deploy on Thursday I will be freed up completely to focus entirely on my new role in the R&D group. My new team includes a business analyst with whom I have greatly enjoyed working in the past, and a database person who has garnered my utmost respect. I'm a little iffy about the director for the team, but I recognize his technical skill and I am trying to give him an honest chance to make a better second impression on me. Once I dive into the new team, I expect that things will get much better for me. There is much to do before the end of the year, rolling out new tools and moving us into Visual Studio 2005 and .NET 2.0. I just fear that I am going to lose more valued friends along the way. I can think of two right off the top of my head who I pretty much expect to be gone by Thanksgiving. That pretty much sucks.
Oh, and it's after 11:30 at night and Ben is still awake. Probably for another two or three hours. My head hurts.
So pretty much nothing got done to fix it yesterday, and my first gut feeling proved to be wrong. I set about this morning to fix it. Unfortunately, fixing it entails stepping through it to narrow down which specific component is failing. Each pass takes a half hour or more due to the very long call to the database to get the list of tens of thousands of consumers. Each pass I eliminated one possibility, but the progress was tedious. Something that I thought I could have fixed in under an hour went on to consume six full hours of my day. In the end I figured out which column in the data was causing the problem, but couldn't tell why until I manually scanned through 21,500 records to examine that column and look for anything that stood out as... odd. It's hard to look for something when you don't know what you are looking for. You just have to let your mind glaze over and trust your instincts to take notice when something outside of the normal pattern shows up. It turned out that three records (out of twenty one thousand five hundred) were returning a null value.
Conferring with my database resource, I discovered that those particular records came from a new tool that was deployed (you guessed it) four weeks ago. Prior to that tool, the value in that column would never, ever be null. But all of a sudden it became a possibility, and over the course of four weeks a few odd records crept in. I adjusted the code to handle for a null, but I couldn't run it on my own machine to test because my vendor data was out of date. It was so late in the day, I don't even know whether or not it was able to be tested by QA. I suspect that it will be Monday before the business owners become unblocked. Oh, well. I did the best I could.
Unfortunately, by doing that I put myself a full day behind in the other work I was supposed to be doing today. I had hoped to complete the last of the image upload component today, so that QA could begin testing it on Monday. I mean heck, it's only supposed to deploy on Thursday. No rush, right?
Aside from that I rattled a few cages today. I had a very long conversation with someone in HR about the current situation in the tech department. I don't think the people in the upper levels of the company fully appreciate just how poisoned the atmosphere is in the tech department these days. Some of that is unavoidable, following a major layoff. But much of it is, in my opinion, *completely* avoidable just by proper communication. Right now everybody I work with on a daily basis is actively updating their resume and considering other options. There is a pervasive feeling that perhaps we are in fact being set up to fail spectacularly so that the entire department can be shut down and outsourced. It's ugly, and it's unfair.
Personally I love this company, and have every intention of retiring from here if they let me. It pains me to see things so poorly executed. I only hope that things will eventually improve. As far as my position is concerned, four more working days and I am done with my commitments to the existing projects. Once we deploy on Thursday I will be freed up completely to focus entirely on my new role in the R&D group. My new team includes a business analyst with whom I have greatly enjoyed working in the past, and a database person who has garnered my utmost respect. I'm a little iffy about the director for the team, but I recognize his technical skill and I am trying to give him an honest chance to make a better second impression on me. Once I dive into the new team, I expect that things will get much better for me. There is much to do before the end of the year, rolling out new tools and moving us into Visual Studio 2005 and .NET 2.0. I just fear that I am going to lose more valued friends along the way. I can think of two right off the top of my head who I pretty much expect to be gone by Thanksgiving. That pretty much sucks.
Oh, and it's after 11:30 at night and Ben is still awake. Probably for another two or three hours. My head hurts.
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
From:
no subject
Eli has a huge credibility gap to bridge with many, if not most of us. Like you, i love the company, but it hurts having to support and being expected to blindly stand behind things which seem like bad decisions. If we weren't going to have real, talented technical visionaries in the directorial roles; we needed (in my opinion) people with the background to know how things work- and in what ways the business is different than most businesses and how Technology is different than a standard software shop. I don't think it's good enough that we have well intentioned folk of whom the CTO thinks highly- from what i've seen, aside from your director, the other two aren't broad enough in technical fields to be able to get things done from a sheer willpower and determination level and having been around three to six months don't have the breadth of understanding processes, even the ones which seem hopelessly fucked up and illogical to begin designing better ones. That they're both UI designers, working for a UI designer doesn't help; and it's also not helping me at least that Ryan won't listen to me but would rather be briefed on things by Jeffrey, yet another UI guy without an understanding of the guts of the process. Being uninformed is bad. Being willfully ignorant and refusing insight from someone who has been around long enough to know what's going on and who has written the documentation down to user level for the last few major marketing projects is painful.
I really don't want to have to leave. I'm still worried that my new position is going to be a paper tiger, and that i'm going to get to parcel out bugs to the two näifs which they'll proceed to prioritize right down into the morass, and i just don't see that being a happy place. Now, they might surprise me. The position might actually be something like a test manager role where i can coordinate DevQA and with the added weight of being able to be able to force teams to fix their bugs in a timetable which is determined through a discussion of equals, but i don't want to expect that and end up with something closer to the former.
Who [else?] do you expect to be gone by Thanksgiving?
From:
no subject
They actually were pissed at me because I came to work on Thursday after they said they wanted me to take a day off - on them.
I thought they meant Friday, and blah blah blah.
After initially reviewing, with Ed, how stuff got into production, I had just started to trek down the path of reviewing what had changed to the underlying structure.
In their infinite wisdom, the Consumer DB team being run by the lying, manipulative New Yorker, made changes that would directly effect this processing, but didn't think it relevant to share with anyone else. Go figure. BTW: none of it is documented.
Another one I would watch out for is Bill.
He is the one who threw me under the bus.
Then he lied to me again, this time on a personal matter.
Basically, prior to all of the chaos happening, we had plans to get together after the bball game tonight.
Today, I reached him about the plans and he said his had changed and he wasn't going because his game buddy had to work tomorrow.
Great, no worries, I had my buddy to go with.
Imagine my surprise when my little buddy and I are sitting at dinner at Michealangelo's in the Center House and guess who should walk by? Bill.
Granted, I am pretty cynical and my judgment is probably a bit clouded, but he acted incredibly sheepish and actually took a step back when he realized I was there.
Just everyone be careful. Watch what you say and who you say it to...the walls have ears and it appears no one can truly be trusted.
It was my pleasure to work with both of you. Just wish it could have gone on a bit longer as we were truly making progress, or so I thought....
Good luck...
From:
no subject
I think the database team is going to be short one person in the very near future.
From:
no subject
From:
no subject
As for the work thing... re-orgs/layoffs suck. The fact that they didn't have a definite plan into which to transition immediately (or within 4-7 days) after the "D-day" events has just made a necessarily uncomfortable situation unbearable.
While some ongoing projects need to be completed, people could have been transitioned into their new roles (at least given job descriptions, pay scales, team goals) while also finishing up the final details of the previous commitments. As it is, most people have no clue WHAT their new roles are (even if they have a title) and no plan for transition.
Eli needs to grow the fuck up and stop playing little-boy games of who he likes and who he fucks and take care of business before indulging in power plays. And Kristie and Matthew need to either decide that they want a functioning tech department and hire someone who is competent, emotionally stable, and scrupulously honest... or admit that they have no clue about the Tech side of things and don't really care about that aspect of the company and outsource it all - hell, the way things are being handled, maybe they deserve to have Hive do all their work.
Yup... i am beyond annoyed w/ K and M... like you have pointed out, they aren't stupid people... they KNOW who Eli is and what he is. And that is the kind of person (or Larry or whomever they had in the rotating CTO position) they want. If that is the case, then i wonder if the company is really what they appear. i know that V-O has treated you marvelously... and has been good to us, too - when things were so hard. But unless they decide to change their focus and start treating the Tech department employees with respect and honor, i don't know if they deserve any of the great employees they have.
i so don't want to go through the upheaval that a job change will cause for our family. i am encouraging my beloved to breathe, bear down and bear with it, and wait through this transition period. But someone needs to take control and make it a limited transition period, not an indefinite period of morale killing chaos. i saw what the poor conditions at M$ did to him (and to our family)... and i don't want to go through that again.
Damn, apparently i had a ramble/rant in me that i didn't know was there! Sorry.
From:
no subject
Seemed strange when I heard it last week.
I think it was a warning from Jeffrey.
'See how they treat complainers? Look at what they did to Kurt.'
Just something to keep in mind.....
From:
no subject
From:
no subject
They took his title. And they made him take a pretty hefty cut in pay.
He went for it because he is thinking of the medical coverage....
From:
no subject
Yup.
My heart-cockles are nice and toasty.
From:
no subject
Just be careful.
I cared a lot about you guys and we were doing great things.
Just be careful.
I think you all can trust one another, but beyond that, I would be skeptical.