Tales from Techsupport

[Merrick ap'Milandra]

My company switched to Libreoffice after they got rid of the IBM terminals in the early 90s. I love the suite. It is tremendous. THey have the ability to save permanently onto workstations completely disabled.

We have to download to a temp directory and then upload to our Microsoft One Drive, because the temp drive purges ever hour.

Hypothetically, this is to keep "sensitive" information and documents from being exposed to individuals on the networks/terminals that shouldn't see it. but it is a pain in the ass.

They could just as easily provide single directories on the server with each of our logins... but... we don't login to the terminals using a linux user. that is always automatically logged in as a generic, and the individual applications that run from it have our IDs tied to each of those programs.

But .. yeah..

for my same issue that i posted with the other day, they continue to try and circumvent various errors in pricing/inventory/invoicing by attempting to draw from different resource fields of pricing. MSP as opposed to cost... They just won't fix the problem, but instead circumvent making it a smart fix, and continue to break two additional modules of pricing/shipping/invoicing/ordering in order to fix one.

Part of the problem of our "step child" version of our location is that we don't have don't have any of their hotfixes in place that should work fo rthem at alternate locations. We are a beta product that no one properly designed from the ground up, for data logistics.

I'm now on level 12 of tech support, and who knows how many laterals that I've actually been placed on, because a 12 level tech support/development support hierarchy seems stupid. there's no fucking way. I couldn't be above level 3 or 4.

"Level 5" was me talking directly to the developer who wrote a given module for an application once. I'd gone through every level of their tech support and the senior, senior, specialist guy said "hang on, instead of relaying this over IM to the dev, he asked if he could just call you..." That was pretty cool. WHILE ON THE PHONE WITH ME he'd code a fix, compile it, throw it in dropbox, I'd download it, run it, and feed error information back to him.

Hard to believe that was only a decade ago. I can't imagine getting an actual developer on the phone for moderately sized product these days. I remember quipping with him when he said his boss probably wouldn't be happy that he was doing this directly that this was about as agile as development could get, so he was adopting the culture as thoroughly as humanly possible. :P

[Mileron]

Every month we have server maintenance and patching.

This past weekend I arrived at 6:55am, breakfast in hand.
Checked out the building electronics recycling. Didn't find anything cool.
Went about my tasks.

We had an onsite repair scheduled for two servers.
9:20 the tech calls me and asks if we'd gotten the parts. We hadn't. He says they were picked up at 7am by special courier. Okay, well they haven't shown up.
9:28 one of my guys shows up.
9:35 the other one comes in.
9:40 he calls back and says they're "ten minutes away" and comes on over.
10:10 he shows up. No parts.
10:45 he calls his dispatch and is on the phone with them for twenty minutes while they try to figure out what happened to the parts. They tell him that one of the batches is on its way.
Meanwhile I get a call from the second courier stating she'll be arriving after 11:30. I asked if she could show up sooner because the tech has already been onsite for the better part of an hour.
She arrives at 11:10.

Working with the tech we get one of the devices repaired.
At 12:15pm while trying to decide what we want for lunch, and while the tech is back on with his dispatch trying to find out what happened to Parts Batch_001, we get an alert that the AC unit in the server room has gone down and the temp is going up.
Keep in mind, it's one of the hottest days so far this summer - but we'd come to find out it's a new problem and our usual steps to fix it weren't working.
We set up our portable coolers and, fifteen minutes later as the unit isn't coming back on we start shutting down servers which we needed to complete patching/verification thereof.
By 12:45 we have our virtual hosts and two other servers running out of fifty. ETA on the HVAC emergency response is 60 minutes.
The tech gets with his dispatch again and they state that, as no one had been present to receive the package, it was sent back.
I send him on his way.

After, I call the mfr and rip them a new one as politely as I can, stating that after having watched our CCTV footage, there were no visits from anyone outside of the timestamps listed above, and all I get is "we're sorry for the inconvenience." I reschedule a new delivery for Monday because by that point the item was on its way back to depot.

3pm the hvac tech shows up and goes through a bunch of things. At this point four days later I can't remember what his initial diagnosis is, but it's some bullshit about how the unit "shut down due to inconsistent flow from the building water supply". If that were truly the problem, it would be having issues every weekend, not just this one. Plus if that were the case, there'd be building engineers crawling all over the place because we're not the only tenant with water cooled AC in our server room.
But it starts cooling by 4:30 or so, and slowly dropping.
We power up our servers and get our shit finished because it's getting late and we'd lost two guys due to time zone shenanigans.
5:30 rolls around and it drops low enough into the low orange that we button up and I head home, pulling out of the garage at 6pm.
6:20 I hear my phone ring, but no handsfree in car so I keep going home.
6:32 I get home, feed the cats, and remember that phone call. It was the emergency notification system stating the temps in the server room shot back up.
Shit.
I put my shoes back on and run out, getting out of the house by 6:40 and back into the building at 7:10 due to sportsball game traffic.
While I'm rushing around setting up the cooling units again two of the other guys are shutting down those same servers again.
Building engineer shows up at a few minutes after 8pm because he lives only a few blocks away and was on his well deserved weekend, but when one of the C level people in my org calls his boss, he's gonna show up.
HVAC tech shows up about 45 minutes later.
Turns out there's a condensate pan on the unit (as there should be) but the sensor that triggers cycling of the system is manual reset.

So what happened was humidity caused the pan to fill up with condensation, and the float sensor turned off the unit. Except for some reason some idiot installed a "manual override" float, instead of "automatic" - so if the unit ever shut off for this reason, it would need to be manually turned back on after a default cooling period of 15-20 minutes, plus any time required for the pan (currently bone dry) to drain.
The tech removed the trigger for the float and left after getting part information to order next business day.

I left at midnight after two conference calls - one with the tech and my boss, and the one after that with my boss and the C-level who called in the cavalry. I also set two rubbermaid totes under the pan for condensation catching purposes. Not ideal, but it would work.

I was back the next morning a little before 9 to check up on it. Still dry, still running.
Another of my guys went back in the afternoon. Same deal.

Turns out the mfr of the HVAC unit had no idea the device was shipped with a manual reset on that trigger. They had to special order the automatic part, which would require reprogramming of the unit - but as it's now Thursday morning and I've gotten no further calls from them about it, we don't know when it'll come in or what sort of downtime we're looking at (again).

to be continued...

[Bonlainy]

Wow, that sounds like it was SO much fun. Luckily I don't have to deal with that anymore now that I'm retired, but we used to have our A/C unit go down in our server room at least once every summer, and almost always on one of our many 100+ days, AND almost always on a weekend. Since I lived less than a mile from the office I was the one who had to come in and figure out what was going on, get our HVAC contractor on the way and wait to see if remedial actions would get the temp down enough to leave all the critical servers on.

We eventually went hunting and found a large portable A/C unit on wheels that fit in our server room, and it worked great. The only problem was it wasn't automatic, someone had to physically turn it on, and luckily for me they assigned that to our on-call techs, we would just program their badges to allow them access during non-business hours. Our main units weren't water-cooled, they use refrigerant.

Now my only tech support work is for my wife, who works from home, and it's all minor stuff. The problem is her company (Tetra Tech) locks her laptop down hard, so I am limited on what I can do.

[FilanFyretracker]

honestly makes me wonder if the drain is partially clogged, because you would hope the initial drip pan under the coil would be able to drain at the same rate that the coils can produce from the air.

[Mileron]

We eventually went hunting and found a large portable A/C unit on wheels that fit in our server room, and it worked great. The only problem was it wasn't automatic, someone had to physically turn it on, and luckily for me they assigned that to our on-call techs, we would just program their badges to allow them access during non-business hours. Our main units weren't water-cooled, they use refrigerant.

Yeah, we have two smaller units that can be setup if needed.

honestly makes me wonder if the drain is partially clogged, because you would hope the initial drip pan under the coil would be able to drain at the same rate that the coils can produce from the air.

They used CO2 canisters to blow it out from both ends and confirmed correct drainage.

Today they replaced the float trigger, but didn't confirm the reprogramming was complete. We'll follow up next week.

[Mileron]

Yesterday (August 15th) DUO Security had an issue where SSO logins were broken on about a third of their authentication systems.

We discovered it just a few minutes before 8am.
When we discovered it wasn't a transient problem, we implemented a temporary fix to allow usage we then sent out an email with a workaround at 8:10, and watched tickets roll in anyway.

This is one of the responses I received after that email went out.

tmp_b3da35a7-8c3e-493e-99d5-99b677bf1a8e.png

[Mileron]

Mine:

Our office is starting a "strongly recommended" hybrid situation - you must appear in the office two days a week. Some of the questions I've been getting:

Do I need to bring my laptop with me?
Do I still need a badge to get into the office?
Will you give me a (desk/voip) phone back when I come into the office?
What if I can't come in on Tuesdays?
If my internet goes out at home, how will I work in the office? Don't I need VPN? (yes that was all one question)


Sigh.


From my wife, whose boss had a deeply technical email question:

Can I CC more than one person at a time?

[Bonlainy]

Mine:

Our office is starting a "strongly recommended" hybrid situation - you must appear in the office two days a week. Some of the questions I've been getting:

Do I need to bring my laptop with me?
Do I still need a badge to get into the office?
Will you give me a (desk/voip) phone back when I come into the office?
What if I can't come in on Tuesdays?
If my internet goes out at home, how will I work in the office? Don't I need VPN? (yes that was all one question)


Sigh.


From my wife, whose boss had a deeply technical email question:

Can I CC more than one person at a time?

Wow, just wow.

[Mileron]

Wow, just wow.

I'd like to point out that I may have been unclear - my wife's boss had asked her if the CC'ing multiple recipients were possible.

New:

Had someone ship back a laptop after they left the company... and overwrote the hard drive with a fresh install of Windows 11.

[Tinthalas Tigris]

"Level 5" was me talking directly to the developer who wrote a given module for an application once. I'd gone through every level of their tech support and the senior, senior, specialist guy said "hang on, instead of relaying this over IM to the dev, he asked if he could just call you..." That was pretty cool. WHILE ON THE PHONE WITH ME he'd code a fix, compile it, throw it in dropbox, I'd download it, run it, and feed error information back to him.

Hard to believe that was only a decade ago. I can't imagine getting an actual developer on the phone for moderately sized product these days. I remember quipping with him when he said his boss probably wouldn't be happy that he was doing this directly that this was about as agile as development could get, so he was adopting the culture as thoroughly as humanly possible. :P

So this issue finally got its wheels rolling. But in the process internal IT (north carolina) and Bangalore (after hour updates, nighttime engineers, etc) got wind of me and interacted with our I -site third party IT team. The third party came down,three of them to talk to me. This was about 2 months ago. Got a job offer the first week of October with the IT company.

Fast forward. Tomorrow is my first day officially doing IT in a new career with a new company.

I am getting a company car, gas card, not to mention the true resume experience that has kept me from pursuing this field for over 2 decades. I am excite.

[Dale]

So this issue finally got its wheels rolling. But in the process internal IT (north carolina) and Bangalore (after hour updates, nighttime engineers, etc) got wind of me and interacted with our I -site third party IT team. The third party came down,three of them to talk to me. This was about 2 months ago. Got a job offer the first week of October with the IT company.

Fast forward. Tomorrow is my first day officially doing IT in a new career with a new company.

I am getting a company car, gas card, not to mention the true resume experience that has kept me from pursuing this field for over 2 decades. I am excite.

Congrats. I hope it continues as awesome ass it starts.

[Tinthalas Tigris]

Congrats. I hope it continues as awesome ass it starts.

Thank you so much!

But GOTT DAMN! Today has already gone off to a hilarious start. I have been a full time employee one way or another since I was 20 years old. For over 2 decades, every single employer I have had that I have been full time with has had problems with their on boarding for one reason or another. As far as I can tell, it has never been anything due to me making something happen incorrectly, like a bad application, or invalid references.

From payroll errors, to incorrect job description labeling, to going to an interview for a position and then being grouped into an onboarding, hr and procedures seminar for those already hired, to now today.

I mentioned everything above about how I got hired by this company. I had an assignment with my previous employer that I had to complete before I could leave. Thanksgiving was another blockade for me getting to start early, because it was only going to be a 2 day work week, and new hires need 2 weeks to complete all their related training.

All this is mentioned, because I completed my hiring data background checks and screenings by November 2nd. Essentially I was hired then, but would not start until today, the 28th of November, due to an assignment with my previous employer.

During all that, they had to extend some Homeland Security requirements that normally have to be completed within 72 hours. Also during that period, the major conglomerate that owns this company (or, now owned) has split the company off, and they are now a separate entity. All URLs for onboarding I had been completing were directing to the previous conglomerate and now that I am actually part of the company, all of my onboarding information did not import, because they no longer have those assets.

However, my remote onboarding and start date still was going to the previous conglomerate. They scheduled a remote orientation today while I was supposed to, initially go and train with my supervisor. I worked it out with him, and he was a little confused, but I have no idea about any of this background information until after my remote orientation meeting was already going, and I couldn't get my internal ID for the meeting to attach to the Microsoft Teams meeting.

Long story short, I am stuck between two systems.

After all was said and done with orientation, and me weaseling my way into the meeting using the provided token from the e-mail, and loading a stripped down version of teams onto my Linux running Laptop, I was able to get in, and follow all of the Onboarding information and orientation that they had on the docket for this morning.

After that, I got a call from my Boss Boss. The company services another very large company, as well as many others, as an on-site IT for several major retailers. I will be handling a district for one single retailer, and effectively am one of their employees; IT dude in Sheep's clothing.

We learned in that phone call with my Boss Boss that i am stuck between these two systems, and he has to import as much data as he can from the old system, no longer associated witht he company, and I had to resubmit everything again.

SO far everything is fine on my part, and I am getting paid to clean out my garage and repair my desktop I have in my workshop in the garage (because, I don't normally run windows, and it is going to take a while to get my company car, because they are all out of company cars, and they have to get me a rental, and before they can do that, they have to send me an american express card, which has to be attached to my login information, which has to be imported from the old system, and a new file built, etc etc etc).

Dude. I have never had a normal hire with any fucking company, and it just makes me laugh.... and laugh... and laugh.... because, what else am I going to do after 21 years of dealing with this?

[Tinthalas Tigris]

How dumb work technology rumors start:
Assoc: what are you doing?
Me: I am cataloging the model, asset tag, firmware and hardware versions of these credit card sig cap machines in every store.
Assoc:why are you doing that?
Me: we are never told exactly why we are surveying hardware, but in this case, it looks like they want to see how many of these units will be capable of firmware upgrade to next level technology, and which ones will require a physical hardware upgrade.
Assoc: what is next level technology.
Me: we aren't necessarily going this route, but it would be for options like phone touch apple or Samsung pay and thta kind of thing, if we have enough machines capable of it.
Assoc 2: what is he doing?
Assoc 1: he is upgrading it to apple pay
Me: I absolutely did not say that.

[FilanFyretracker]

basically the whole point made way back in elementary school with the game Telephone. Teacher hands a kid a note, they read it and then tell it to their neighboring student and so on across the class in a set order and then at the end one compares the note to what the final student in line heard.

[Mileron]

What a day.

1 - Sales guy comes up to me regarding an email we sent him a couple days ago. One of our security portals flagged him for DLP - which is weird, because the files that were twigged were all downloaded simultaneously. Turns out all he did was get an update to MS Teams on his phone at around the same time an update to our DLP system took place. However he had a more pressing concern - apparently he noticed in the last few weeks that certain... searches... were appearing in his work computer's browser, which we know he doesn't use for that purpose. Seems like Google's gmail account sync is a little too tenacious.

2 - We've been getting catered lunch every Tuesday since October. Additional incentive for people to return to the office. Today I'm in a meeting dealing with (#3) and the office manager runs over to me and says "Help the sterno is on fire!" I run into the main breakroom and apparently someone had knocked one of the chafing dish trays hard enough to slosh the fuel goop out onto the table. I smothered the sterno can with a coffee mug while someone else managed to grab one of the chafing dish lids and do the same with the wad of goo on the table. Table's a little burnt.

2b - this brings to light the fact that we don't have updated fire suppression capabilities outside of general systemic sprinklers, so I get voluntold that I'm one of the new fire wardens for the office, oh and I just might get signed up for first aid training as well. Cuz you know, in IT we're never busy.

3 - Have to provide an emergency replacement laptop for one of our Canadian people. Since we don't have a large physical presence in Canada, one of the managers up there has been storing extra devices for deployment. However the lady who needs the replacement is over five hours away and we had to decide whether we wanted to ship from the States, buy new, or ship from his house overnight to the tune of $300+ Canadian. We decided on overnight shipping for the transfer.

4 - There was an HR miscommunication for an employee transfer between locations. The transfer ticket for the incoming/destination office was submitted and assigned to me Monday night. Then the HR lead for the employee's location submitted a separation ticket for her instead - which one of the UK guys got to and processed Tuesday morning before I even got into the office. Needless to say I need to go through and basically re-onboard her from an IT perspective.

5 - We're sticking with Windows 10 22h2 for now, but a handful of my team have agreed to suffer with Windows 11. It's all shiny and pretty, they say, and annoying to get used to but once you do, it's basically alright. However two people have this ongoing issue where, if the device goes to screensaver/blank screen and isn't used for 30 minutes, it auto-disconnects the VPN. Can't figure out why. It's not Modern Sleep causing it. I get to spend time this/next week figuring that out.

[FilanFyretracker]

for the VPN id see if Windows is somehow pausing any programs when it goes to screensaver, It may not be sleep but don't discount various power saving features. Especially if they are laptops, If laptops that also means check anything related to the wifi for power saving. does the machine lock when it goes blank? could MS have some deeply buried security feature that discos all VPN links when a machine locks.

[Merrick ap'Milandra]

Assuming it's not already turned off, that 1000% sounds like windowskey+x -> device manager -> right click specific network adapter -> properties -> power management -> uncheck "Allow the computer to turn off this device to save power".

[Ackar]

For those that work tech retail...

Spoiler for image:

[Mileron]

Assuming it's not already turned off, that 1000% sounds like windowskey+x -> device manager -> right click specific network adapter -> properties -> power management -> uncheck "Allow the computer to turn off this device to save power".

I've updated the device drivers, turned off all power management, confirmed with a coworker that the firewall doesn't kick idle connections, uninstalled Dell Optimizer (which does some f'y shit with power and network connections).

It's weird, another guy on my team who has an older device running 11 doesn't have this issue. It's only the newest machines.

[Mileron]

3:30am - received notification call of cooling issue in datacenter
4:10am - arrived onsite
4:12am - set up portable cooling units
4:21am - called (HVAC Provider), got hung up on
4:21am - called (HVAC Provider) back, asked for a tech to be dispatched
7:35am - called (HVAC Provider) back, asked for update on ETA of tech
7:54am - started shutting down non-critical systems
8:37am - called (HVAC Provider) back, left voicemail
9:16am - got on brief bridge with two of my UK guys to shut down some more systems
9:36am - called (HVAC Provider) back, asked for update on ETA of tech. Was told he was nearby and would be arriving soon.
10:04am - Brief chat with (manager) regarding issue and tech ETA
10:10am - temperature issue seemed resolved; unit started to cool. (Based off temperature readouts in the (Datacenter environmental monitor) system)
10:18am - noticed the temperature dropping in the Comms room
10:20am - (HVAC Provider) tech arrived
11:11am - Temp in Comms room seemed stable, so started bringing devices back online
11:17am - after tech left, he called me back with a few questions related to the setup/issues / scheduling of the office AC units over the weekend possibly causing issues
12:09pm - all devices back online, room internal temp under 68 deg F
12:38pm - noticed temp started to rise again in Comms room
12:42pm - called (HVAC Provider) back, stated the issue had returned, please send tech back out
1:44pm - called (HVAC Provider) back asking for update. Different tech being sent out
2:20pm - lunch (otherwise, all I'd eaten til now was swedish fish candy)
2:38pm - received call from (HVAC office manager)@(HVAC Provider), tech should be here in 40-45
2:53pm - brief chat with (manager) regarding tech ETA, and request to get tech in contact with CIO / (HVAC office manager) / (manager) after the tech arrives and examines issue
3:04pm - call from (HVAC Provider) to request building security give the HVAC tech approval to enter building
3:05pm - called building security to approve HVAC tech arrival
3:09pm - (HVAC Provider) onsite
3:49pm - temperature dropping after poking at the control panel and thermostat. Additional testing being done on thermostat
5:00pm - per new tech, states the issue may be the thermostat. He's trying to trigger it to fail but as yet has been unsuccessful. He will try to bypass it to ensure the hvac unit remains online through the remainder of Sunday
6:38pm - had a chat with tech. He's still leaning towards issues with thermostat.
7:51pm - chat with tech and CIO. I need to be back onsite Monday early to receive new wave of techs to replace thermostat.
8:25pm - in car
9:00pm - home

I don't know what our emergency SLA is for support with the HVAC company but I think they're probably in breach. 6 hours seems excessive.

Last August I spent a day in the office - it was 17 hours - 7am to 6pm, feed cats, back by 7pm, left by 1am
Today it was just under 17 door to door.

I'm calling it a tie.