Tales from Techsupport

Anyone else with MSCE find it amusing that the new version of windows will be lease licensed, and effectively streamed to terminals via IPXE?

It is so humorous, the number of retail/consume products that rely on linux servers and other *nix based systems, and now the new windows will be run via a linux distribution server to provide a Windows GUI for their clients.

This is now year 6 of me running (almost) exclusively linux.

I do have to say, the "One Note" system within Windows is pretty awesome. We used windows on our laptops with work, and having my work phone running OneNote while I run around a site taking notes to have al that data automatically sync with my work laptop is amazing.

Linux already has great programs capable of doing the same thing, but I know very little about OneNote. It cracks me up that it is such a powerful program (OMG, the Image Capture and Insert into Notes is amazing. I can take skewed angles of screens and it will immediatley crop to relevant data!), and it took how frickin' long for Microsoft to package it?

Quote:

---

Originally Posted by Tinthalas Tigris

Anyone else with MSCE find it amusing that the new version of windows will be lease licensed, and effectively streamed to terminals via IPXE?

---

What? I haven't been keeping up on my news-of-upcoming-stuff, apparently.

Quote:

---

Originally Posted by Tinthalas Tigris

It is so humorous, the number of retail/consume products that rely on linux servers and other *nix based systems, and now the new windows will be run via a linux distribution server to provide a Windows GUI for their clients.

---

I'd love more information on this. It almost sounds similar to some things I've set up to allow access to expensive 3D rendering hardware in countries that have internet connections but larger barriers to entry for obtaining physical computer parts. Provision something out of an Azure data center and have it serve up a workstation OS with access to the computational power of the rack it's sitting in.

Not really that different than running VMWare or Citrix on a local set of servers, except it's geographically somewhere else and there's not a ton of low-level access to the rack itself.

Quote:

---

Originally Posted by Tinthalas Tigris

I do have to say, the "One Note" system within Windows is pretty awesome. We used windows on our laptops with work, and having my work phone running OneNote while I run around a site taking notes to have al that data automatically sync with my work laptop is amazing.

Linux already has great programs capable of doing the same thing, but I know very little about OneNote. It cracks me up that it is such a powerful program (OMG, the Image Capture and Insert into Notes is amazing. I can take skewed angles of screens and it will immediatley crop to relevant data!), and it took how frickin' long for Microsoft to package it?

---

I started using it on and off like, a decade ago. I still don't utilize everything it's capable of (I probably use ctrl+1 more than anything) but it's definitely handy.

For basic note taking I still defer to google's keep on my android phone/pc's web browser, but that's just for stuff that amounts to the functional equivalent of a post-it note in my back pocket.

For anything more serious, I still whip out OneNote.

It's improved a lot in the last decade, but the idea was there quite a while back and proof-of-concept stuff was shown off by Gates back when EverNote was brand spanking new almost a decade before OneNote actually came out.

I kinda wish MS would buy/create/fund a piece of included-with-windows/office mind mapping software similar to https://thebrain.com/ for a much needed boost to their whole productivity ecosystem.

Maybe it's just me, but I find that particular piece of software to be far more useful than, say, a knowledge base. I stood one up for my department at an old job ages ago back before IT Glue was a thing and found it to be ridiculously helpful and easy to use.

Certainly miles better than any combination of any of the myriad of KB/ERP/CRM/SalesForce/Sharepoint implementations I've ever used, but that might just be due to how my brain works.

Quote:

---

Originally Posted by Merrick ap'Milandra

What? I haven't been keeping up on my news-of-upcoming-stuff, apparently.



I'd love more information on this. It almost sounds similar to some things I've set up to allow access to expensive 3D rendering hardware in countries that have internet connections but larger barriers to entry for obtaining physical computer parts. Provision something out of an Azure data center and have it serve up a workstation OS with access to the computational power of the rack it's sitting in.

Not really that different than running VMWare or Citrix on a local set of servers, except it's geographically somewhere else and there's not a ton of low-level access to the rack itself.

---

I listen to a podcast from TuxDigital (a couple actually), and I believe it was on the episode of This Week in Linux from 2 weeks ago. If I get a chance, I'll try and find the exact episode, time stamp, etc and post it here.

But yep. They are going to start charging $10/mo per license in the new version.

I've been trying to startup my own domestic NAS, because monthly subscriptions are getting out of hand from everyone. I plan on backing up all our physical media to it so we can stream through it, and we will setup a monthly queue in the house on a Note system, once I decide which one, for ordering new serires, etc, and putting them on there.

Quote:

---

I started using it on and off like, a decade ago. I still don't utilize everything it's capable of (I probably use ctrl+1 more than anything) but it's definitely handy.

For basic note taking I still defer to google's keep on my android phone/pc's web browser, but that's just for stuff that amounts to the functional equivalent of a post-it note in my back pocket.

For anything more serious, I still whip out OneNote.

It's improved a lot in the last decade, but the idea was there quite a while back and proof-of-concept stuff was shown off by Gates back when EverNote was brand spanking new almost a decade before OneNote actually came out.

I kinda wish MS would buy/create/fund a piece of included-with-windows/office mind mapping software similar to https://thebrain.com/ for a much needed boost to their whole productivity ecosystem.

Maybe it's just me, but I find that particular piece of software to be far more useful than, say, a knowledge base. I stood one up for my department at an old job ages ago back before IT Glue was a thing and found it to be ridiculously helpful and easy to use.

Certainly miles better than any combination of any of the myriad of KB/ERP/CRM/SalesForce/Sharepoint implementations I've ever used, but that might just be due to how my brain works.

---

6 years ago, I pretty much wiped Windows from everything I owned. Not a rage quit or anything, My laptop just wasn't working very well, my desktop was neither, both didn't want to cooperate with their older versions of windows, and I decided to try Linux Mint. I didn't use it much, because my line of work at the time did not have any true need for it.

Once QUarantine went full effect, I doubled down, and went through multiple distributions, until finally settling on Pop!_OS. It is my favorite distro, and I install it on almost everything, now.

My kids do have windows on their machines, but they also have KDE PLasma on a separate drive (which I encourage them to use, even though they still prefer windows).

OneNote was something I only came across for the first time when I got my new work laptop back in December of 2022, and I needed a note taking software, because most of the Google Suite is locked out of the device's allowed URLs, etc. It works, and I dig it, but I really want to either develop or find something that is a tiny bit more practical for my whole *nix world.

Guy walks up to me working on a printer on a mobile printer cart. Small space, I am right there, multiple tools and parts scattered around the cart.
"Can I print off of this one?"
"I will be only 2 more minutes."
"you're good"
::moves my tools, drops a part::
"as I said, I am only going to be two more minutes and then you can use the entire cart."
"I only need this printer,and this is the last thing I need to print for the day, thank you."
"well, you are absolutely welcome, in two minutes."
"it's okay, I can just do it now"
:proceeds to print:

I show up to the same site as above, today.

I show 5 parts I ordered have come in, and I have tracking. THey've all been signed for, and the instructions on them say to take to the MDF room.

I arrive, and have one at the front desk. It has been there for a week. Since the day after I left, previously.

There is one in the General Manager's office. It says "Take to MDF Room/Give to Field Service Technician IT."

One is in the Assitant Manager's Office. It says, "Take to MDF Room/Give to Field Service Technician IT"

One is in the MDF room. Where it should be. Right on the bin labeled, "New Freight and Parcel - Rec'd And Ready for IT"

It takes me 1 hour and 15 minutes to track down who signed for it from the tracking, which was less than an hour ago. The other person in Receiving says it went to the Assistant Manager's Office. ASM says "didn't you pick it up?" I show them my laptop screen which says just signed for, shipped via UPS and show them the box that was in there that says FedEx. This game goes on for that whole time. I finally walk into the MDF room, and see it behind the door behind the MDF rack. Genius.

While our office is still pretty hybrid (with a good contingent being present on Tuesdays, due to free lunch, less on Weds, even less on Thurs, and nearly zero on Monday/Friday) we've got a handful of people who, for various reasons, are permanent WFH.

One recent addition to this list has complained repeatedly since right after the 4th of July - roughly two tickets a month - about his VPN stability.

"It keeps dropping," he says.

He's in an apartment building with dozens of other tenants. Naturally, his wireless list has a couple dozen unique SSID.

Reboot. Restart router. Try different room. All seem to temporarily alleviate the issue.

He states he cannot use an ethernet cable, due to distance from the router and, apparently, the fact that it's in a roommate's room and they don't want a cable strung through the residence.

But that's not the point of this post.

The last ticket we received started out as a VPN issue ticket. But when emailing back and forth with the assigned tech, he also stated "the shocking is really making it difficult to type on the laptop."

:wtf:

I'm off the day he brings it in to swap. My counterpart (the ticket owner) gives him an (undeserved/unnecessary) upgrade during the replacement - before he examined the original device because he wanted me to look at it as well.

I pull the battery and bottom case off and find this.

Attachment 6813

Everywhere underneath. Including the edges where the wireless activation switch and the power port are.

:banghead:

My mental order of operations are:

The guy is absolutely addicted to takeout iced coffee - Starbucks or convenience store, he doesn't seem to have a preference - and he is left handed, so his mouse goes to the left of the computer.
The power port and wifi switch are on the right side - where his drink of choice sits and condenses until empty.
Which means that most of this liquid is just water condensation.
For this go-around. There are many indications of it having dried and encrusted underneath. Which could explain the ongoing VPN issues, especially if water encroached on the WLAN card that early on.
Even the dirt/dust in the fan is showing evidence of clumping from moisture.

It's too late to re-swap him down to a previously-similar model, but he's been informed that in the event of a future swap, evidence of liquid damage will be grounds for compensating the company for damage/repair.

We also told him to keep his coffee the fuck away from the laptop.

Deployment for nationwide HR offices. New PCs, as the new vx Network layering just doesn't run on the old Windows PCs. We're installing a batch of Lenovo Thin Clients (m80qs) And they have been stupid buggy.
Deployment initially began the last week of November. 2023.
Their DP out does not like mating with the DVI on the monitors we already have deployed. They postponed it.
They rereleased a fix (hold power button to "de-energize the bios/cmos."
Imaging of Office Suite did not work (Hello, can we just rewrite the bash script ourselves?)
My region's deployment manager finally, last week at our monthly meeting, said simply, "Just deploy it, we'll figure out what is wrong with it as we go."
I took monday of last week off, so I was behind by a day, but I wasn't too concerned, as this was our only deployment scheduled for this week.
I went Wednesday to Site A. Brought the new m80q and informed all administration some things:

"This deployment will take 2 hours. I will need unfettered access, and no one interrupting me while I perform this deployment. Periodically, I need to see what is happening on the display monitor, in order to push various modules from the network onto the new terminal if it hangs."
I have everything deployed, hardware wise, one half-hour in, I have removed the old terminal from the DHCP tool, and the new one is on. I have it maging almost immediately.
I step away to use the restroom. I am away for 10 minutes.
I come back, and the door is closed and locked. One of the administrators says I can't go in there because an associate "needs a safe space right now."
I said I needed in, and they refused.
For 2 fucking hours, that employee was in their safe space, while I just waited outside the door/in my remote office, just pinging the unit to see if it was on the network. It isn't. for 2 hours, I just wasted fucking time.
The crew in charge of monitoring deployment was supposed to leave at 2pm PST. The guy working alongside me in North Carolina said he could stick around for an extra hour.
By the time the door opened again, it was 1pm. PST, and I had just barely the amount of time to restart it if necessary.

Little did I know, that in the process of getting the DHCP server to avoid conflicts for what sometimes remains on its port triggering/port forwarding software, the VoIP phone that the terminal plugs into was unplugged, and the VoIP phone plugged into the network adapter (I mean, i probably should have looked to see if it was live, because that is normal procedures, but why shoudl someone unplug a terminal and plug in a phone in their safe space? I mean, one would think that they woudl tell me that they were going to do this, did this, or asked me why it was unplugged in the first place).

For half an hour I trouble shooted. For an hour I trouble shooted. I looked under the paper that says, "DO NOT PLUG IN," covering the screen, and the VoIP phone is one. It is now 2 o'clock PST.

Fuck it. I'm getting two hours of OT. Messaged my deployment manager the above information and his first response was,

"..."

About 5 minutes later he said, "take the 2 hours OT today, and whatever amount of time it takes you to finish this when you come back next, take that whole amount of time, and bill it as OT for the same day alongside whatever else you do that day. Even if you are only there for 15 minutes outside of the 2 hour imaging. Bill them the whole day, plus the amount of time it takes for this."

Separate Site.
We manage the networking of the Lot Cops (you see these often at Home Depot, Lowe's, Walmart and other big box stores in the parking lot. Every 15 minutes they remind you to lock your car, store your valuables, etc) and have cameras at the top of a mast on the head unit. We occasionally maintain the batteries and the solar system, but they go mostly to the technicians employed by the LVT company themselves, but it costs us less for our contract to deal with the head unit and resituation of solar panels, and periodic voltage checks of the batteries.

We also move them, when the client goes through the motions and requests the move properly.

I had an e-mail forwarded to me from 6/18 on 7/5 requesting that I move the two trailers at Site B immediately. THey are having their parking lots repaved that following morning. I informed them that I would be there first thing Monday morning at 7am.

No response.

The pavers were there at 6am, and already scraped the area around one of the LVTs, but the other had not yet been scraped.

I wrassled a couple of site associates that could drive a forklift, prepped the unit for movement, and we moved it from one side of the building to the other. THe intended site of transfer was already paved, and they wouldn't let me move it over there, because the asphalt was fresh and had not yet been striped.

I informed them verbally, as well as an e-mail to their entire Administration and Management team that they would need to cordon off the locations they would be moved back to (very active parking lot) SUnday evening, 7/14, so that I could move them first thing Monday morning. I spoke directly to the individual who would be closing the site Sunday evening who said they would take care of cordoning off the area.

I arrived Monday morning, and nothing had been cordoned off. I had an itinerary for the day and had to be 4 cities and 23 miles away at 10am. By the time I had pictures, spoke to admin, and cleared it with my bosses, it was 845, and time for me to head to the next location. I informed them that I would be back on Friday to do the move, e-mailed all parties, attached pictures to the e-mail for where it needed to go, and they said verbally they would accomodate me.

NO one from admin responded to the e-mail.

Tonight at 6pm (I complete my day at 3:15 pm and i didn't check my e-mail until 9pm when I was plugging my phone for the night) , I got an e-mail from the Admin who was supposed to cordon things off on Sunday evening VIA PRIVATE REPLY (Not Reply All to the entire thread), saying that I moved all the LVTs to the wrong location, and the contractor needed them moved to an alternate location by 6am tomorrow morning.

FUCK THAT SHIT. That means i would have to be there at 5am. Not fucking happening. less than 24 hour notice? less than *12 hour notice?* I don't care how much i'm getting paid hourly. THat's a $1,000 charge in my book - PER MOVE. That shit is going to my pocket.

I can't stand muggles.

To those of you having a bad day today because of Crowdstrike: good luck, and I look forward to hearing the stories here!

Attachment 6963

We got rid of Crowdstrike a year and a half ago because the device-resident agent was drastically impacting the usability of half of our fleet.

So I'm glad for that.

That said, I do feel for those who were impacted. (Heck I couldn't log into my bank until after 8pm.)

https://www.yahoo.com/entertainment/...182703060.html

My company uses first-initial-lastname. We've got one higher up with a name that would make for a bad combo. We've also had some women (three, as I understand it; all occurred before I joined) who were getting married who submitted tickets to get their email changed to match their new married name but were managed to be convinced that using it wouldn't be professional for them and they stuck with their maiden names (or got a uniquely patterned alias).

https://www.yahoo.com/entertainment/...182703060.html

My company uses first-initial-lastname. We've got one higher up with a name that would make for a bad combo. We've also had some women (three, as I understand it; all occurred before I joined) who were getting married who submitted tickets to get their email changed to match their new married name but were managed to be convinced that using it wouldn't be professional for them and they stuck with their maiden names (or got a uniquely patterned alias).

We do firstname dot lastname

given this does not seem to be an uncommon problem you think companies across the board would just use something else.

Quote:

---

Originally Posted by FilanFyretracker

given this does not seem to be an uncommon problem you think companies across the board would just use something else.

---

Part of it is inertia. Also, a heaping load of "we've never had this problem, it shouldn't be one now". And "this policy has been in place since before 2010".

It's even more troublesome, for example, with people who change their names. (1,2)

Network ID = email address = computer login, i.e. JPICARD (3,4)

1 - Regarding those that changed their names when getting married, for example, I've had one ... very ... opinionated ... user who hates the fact that her maiden name is still in use as her computer login. Hates. It. And I hear about it all the time. However the amount of time to recreate her network account, transition everything over to the new one, let alone email, documents, attached accounts, etc, it's not feasible to complete building her a new account.

2 - Also: we have one user who specifically requested before being onboarded that her full name be used in her user ID. So let's say her name is Danica Patrick Stewart. Upon being told that we create user logins with first initial last name, she requested that her "middle name/prior maiden" be used as the second character, so "DPSTEWART". It was rejected, because it's not policy, and instead is just DSTEWART. And she complains about it all the time...

3 - Similarly, we've had a handful of people who prefer to be addressed by a hyphenated first-middle name, and requested to use the middle initial as part of their login (i.e. JLPICARD) - but it was rejected, because it's not policy.

4 - It's even more annoying when you need to onboard someone whose new network username/email address already exists. So the policy is to use the next letter of the first name, instead of middle initial or a differentiating number. GFOREMAN exists, so the second becomes GEFOREMAN (instead of GFOREMAN2)... and the third GEOFOREMAN... which is policy...

All of which would be easier if a different identification method were to be used.

For example, in my last position, the network user ID was the employee's HR database ID. i.e. EN123456 The email address then became an alias to that ID, so changing the "display name" and email address for the respective user would be SO much easier.

Quote:

---

Originally Posted by Mileron

4 - It's even more annoying when you need to onboard someone whose new network username/email address already exists. So the policy is to use the next letter of the first name, instead of middle initial or a differentiating number. GFOREMAN exists, so the second becomes GEFOREMAN (instead of GFOREMAN2)... and the third GEOFOREMAN... which is policy...

All of which would be easier if a different identification method were to be used.

---

Wait, so if you have a name collision because your company hires a fifth "John Smith", how does that get resolved?

When I asked that, the response was "when it happens, we'll figure it out." :rolleyes:

Quote:

---

Originally Posted by Mileron

1 - Regarding those that changed their names when getting married, for example, I've had one ... very ... opinionated ... user who hates the fact that her maiden name is still in use as her computer login. Hates. It. And I hear about it all the time. However the amount of time to recreate her network account, transition everything over to the new one, let alone email, documents, attached accounts, etc, it's not feasible to complete building her a new account.

---

These Karens suck. Thankfully they are rare.

Usually most are satisfied with leaving their login the same as long as they can get email with their new name (make new name the primary and old name an alias).

But every once in a while...

Quote:

---

Originally Posted by Mileron

1 - Regarding those that changed their names when getting married, for example, I've had one ... very ... opinionated ... user who hates the fact that her maiden name is still in use as her computer login. Hates. It. And I hear about it all the time. However the amount of time to recreate her network account, transition everything over to the new one, let alone email, documents, attached accounts, etc, it's not feasible to complete building her a new account.

---

This one does not seem unreasonable to me. Would you be fine with using a login that wasn't accurate spelling of your legal name?

You say it's not feasible... but it is obviously a standard task associated with new hires. Why would it not also be considered a standard task for name changes by marriage?

Quote:

---

Originally Posted by Nerkahia

This one does not seem unreasonable to me. Would you be fine with using a login that wasn't accurate spelling of your legal name?

You say it's not feasible... but it is obviously a standard task associated with new hires. Why would it not also be considered a standard task for name changes by marriage?

---

For email, yes. We add an alias, so if Reba McEntire gets married and becomes Reba Rogers, then emails to both rrogers and rmcentire will get delivered to the user's email box.

However the computer login will not change, as it's tied to the user's network account, and a number of systems that (according to the dev teams whose responsibility they are) cannot be changed for various reasons, which are solidly tied to the user's network account..

While things are what they are, I will say the fact they cannot change shows that whoever wrote that stuff was an idiot. I mean marriage happens and one would think they would consider such a thing and allow one to with proper levels of system access change a username and hit apply and it would fire across the system.

Quote:

---

Originally Posted by FilanFyretracker

While things are what they are, I will say the fact they cannot change shows that whoever wrote that stuff was an idiot. I mean marriage happens and one would think they would consider such a thing and allow one to with proper levels of system access change a username and hit apply and it would fire across the system.

---

Yeah those are kind of my thoughts as well. And I know I'm terribly jaded in this regard, but for me, at the company I personally work at, when an IT systems specialist says it's not feasible, -it translates to "I'd have to Alt-Tab out of Call of Duty for longer than a few minutes."

JOKING!!! OMG I'm just joking guys and gals.

:rofl:

Quote:

---

Originally Posted by Nerkahia

Yeah those are kind of my thoughts as well. And I know I'm terribly jaded in this regard, but for me, at the company I personally work at, when an IT systems specialist says it's not feasible, -it translates to "I'd have to Alt-Tab out of Call of Duty for longer than a few minutes."

JOKING!!! OMG I'm just joking guys and gals.

:rofl:

---

:banplz:

Quote:

---

Originally Posted by Eremius

:banplz:

---

Probably completely fair ~

:rofl:

Quote:

---

Originally Posted by Eremius

:banplz:

---

:eyes:

Sometimes... yeah.

And all in one day.

I'm on my way out to lunch, and there's two Amazon delivery guys standing in our elevator vestibule, arguing. I'm waiting for my car when they start walking back and forth, trying both sets of doors into our suite.

Me: Can I help you guys?

#1: Yeah, is this 1701?

Me: Yes, this building is 1701, and you're on the 26th floor

#1: visibly confused, looking at all the signage But is this 1701?

Me: Yes, this is 1701 Enterprise Street, and right now you're on the 26th floor.

#1: still staring around, looking back and forth between signage and the scanpad in one hand, and package in the other

#2 pipes up: We're looking for a Gene Luck Pikkerd?

Me: You're in the right place. Suite 3700 or 37th floor?

#2: Yeah

Me: Thanks, I'll sign for it.

#1: Okay so does this next one that says "25" on it mean we need to go to the 25th floor?

:wtf:

I'm finally out at lunch, because I left it on the kitchen table when I was prepping. I'm on my way back to the building when in front of me on the escalator are two young ladies with polling surveys for some PA thing.

They walk up to a random guy as I'm trailing behind them trying to get out to the street and say

Voter1: Hey, can we bother you for a few minutes? We have some questions about your opinion on (whatever voter issue) in Pennsylvania

Random Guy: No, I'm from Jersey

I slowed my walk, I had to hear this

Voter2: Okay so you work in the city right? How do you think (whomever will do whatever)

Random Guy, slower: I'm from New Jersey...

:ohreally:

It's around 4pm and I'm making my 4pm rounds, making sure the coffee machine is off, checking conference room TVs/projectors, things like that. I barely sit down at my desk when one of the younger sales guys who is present mid-week brings a DHL driver over to me.

Sales Guy: Delivery dude wanted a signature and I didn't know what to do so I brought him to the ol... IT guy because I figured you'd know what to do.

Me: sticks out my hand for the sign-pad I got it, thanks

Delivery Guy: Did he really just call you old?

Me: He also needed to check with IT-Dad if the mailman could deliver a package, so probably.

:fishhit:

Mondays and Fridays are the emptiest in my office, with four or less people both days.

Today I was fully solo(asterisk definitions below) for the first time in a couple months. I've got music on when I'm at my desk. Random music scores. This morning it was Alan Silvestri's "Flight of the Navigator".

I'm doing my start of day/start of week stuff, including my morning office rounds, and hear music, and smell food, but there's no one present. Check the door logs - no one present except myself.

Get back to my desk and the food smell is stronger. I follow my nose and find a guy who set up a device replacement on Friday - when I was out of office, and for which my junior tech didn't inform me of. I ask him how he got in.

"Oh, the front door was unlatched."

I go and check it. He's right, the magnetic strike plate on the doors isn't working (again). I shoot an email to the office manager, and handle his issue.

About two hours after that, I hear a random phone alert through my headphones. It's not mine, it's on silent. I turn around and there's another random dude, just looming the f behind my chair.

RD: So I'm working at home Friday morning and I realize I can't get logged in to my computer
Me: Did you put in a ticket?
RD: I can't get logged into the computer
Me: You can still access the helpdesk site with a mobile device
RD: How is that going to help me?
Me: Instead of wasting a day of PTO because you can't get logged in, you could have had someone that was available help you
RD: Do you know how nice it was on Friday?
Me: No, I was in the hospital all day
RD: Well sucks to be you. You gonna help me with my password problem?

:ohreally:

Turns out when he changed his password earlier in the week, he "forgot how many exclamation points" he used.

:so:

User: I have half an hour free for my computer swap
Me: I'm ready when you are
User: Here you go, I'm going to pump breast milk
Me: Um... TMI, but I need you here for this

-==--==-

After fixing a different issue for a different user for the second time:
Me: Okay so we're done for now. Let's restart, and you can continue working. Tomorrow morning we'll check to see how it's behaving.
User: You mean this restart won't let me work until tomorrow morning?
Me: Not at all. The last time this issue presented, it took twelve hours from the time we fixed and confirmed, until the issue returned. So I'll schedule a meeting for tomorrow at 8:30am so we can double check, but until then you can continue using your stuff.
User: Are you sure I don't need to wait overnight?
Me: Yes

^^ So this was due to a Windows driver update (and a Realtek driver update) causing the user's 23 day old laptop's webcam to stop working. As a proactive measure after the above exchange I contacted Lenovo who stated that the driver provided by Realtek (which they host on the Lenovo support site, and is available through their automagic update downloader) is in fact bad, and they've known it for two weeks, but for some reason no one decided it would be a good idea to pull it from their site.

So, I can't believe this hasn't happened before in multiple decades of working in IT, but the other day...

I'm on a teleconference session with another person from my team, a software engineer at the vendor, and an end user, trying to troubleshoot an extremely weird issue. (Why more vendors don't build in "Impersonate user" capabilities in their software so we could test this stuff without inconveniencing people, I will never know. Probably because they can't be arsed to allocate budget to actually SECURE anything these days...)

It's taken months to get this end user and the vendor together so the vendor can tick the box on their stupid policy that says they actually saw the issue happen or whatever flaming hoops that vendor forces their poor support staff to jump through in order to actually do their jobs.

We let the vendor basically run the call and at one point, they want to see the network requests the browser is making, so the vendor engineer advises the end user to hit F12 to bring up browser dev tools...

...and we watch in horror as the "Airplane Mode" icon slides up onto the screen and freezes there.

While I have no idea what it looked like for that end user, thankfully the end user was smart enough to figure out what happened, undo it, and get back on the call, but that totally could have gone a different direction.

Uff da.

8.

8 laptops since December first.

Dead and pretty much gone. All from different users.

1 - User dumped coffee on it, and wanted me to overnight her a replacement. Um, no. You live within 12 miles of the office. Get off your (couch) and come in for it. After drying it for two weeks, while I was able to get it to power on, it won't POST. Issues with active warranty are causing delay in repair/replace.
2 - User is in meeting. Fan ramps up. Screen goes black. Machine shuts off. Still isn't powering back on. Out of warranty.
3 - User is in meeting. Computer gets really slow, screen flashes. Computers gets fast, and a checkerboard of video corruption appears. We tell him to shut it down, let it cool off, and try again. A week later, the screen flashing thing happens again, but this time half of the screen vanishes and Windows somehow scrunches a usable resolution into the top half of the screen. (Pending return of this one. I can't wait.)
4 - User's house burned down.
5 - User is in meeting and randomly shuts off. Receive it, evidence of liquid damage. However, the entry point is the track pad and the bottom edge of the unit. Turns out the user has a medical condition where they sweat profusely, and after hearing that I remember that there was a previous machine that had (only) the trackpad killed by it. However after several weeks of drying and some judicious isopropyl alcohol cleaning, I've got it booting up. Haven't decided if I want to re-deploy it.
6 - Machine randomly "loses" half its installed RAM. When I use it, it works fine. When my intern uses it, the active RAM slots randomly switch with every reboot.
7 - Machine spontaneously shut off. Bring into office, works fine. Takes home, works fine. Sits in a different room, spontaneously shuts off. Tries a different power adapter. Fuse blows, laptop died.
8 - Machine blue-screens whenever the right shift key is used. Replaced keyboard. Replaced RAM. Works fine with a stress test or with someone like me who doesn't use the right shift key (grade school injury makes it hard for the right pinky to twist in that direction, which caused my keyboarding teacher no end of conniptions) however after the last stress test, it stopped powering on at all.

And that's not even getting into the dozen or so units less than a year old that keep "losing" their webcam and microphone drivers.

Quote:

---

6 - Machine randomly "loses" half its installed RAM. When I use it, it works fine. When my intern uses it, the active RAM slots randomly switch with every reboot.

---

Magnetic bracelet?

Quote:

---

Originally Posted by Merrick ap'Milandra

Magnetic bracelet?

---

Good thought, but no. We never did figure out what was causing it, so long as he didn't use it. We tried different RAM. Different power adapters. Different desks. The common denominator for reproducing the issue was the intern.
We had our network engineer test it out, works for him, he uses it now as his permanent WFH device.

Here's two recent ones

Sales guy (ticket): I'm trying to dial Canada, but none of the calls are going through
Me: (replying/discovery) We need some example numbers
SG: +62...
Me: That's Indonesia...
SG: No it's not



and

Ticket from a different sales guy: My email is showing two identical signatures. Can this be fixed? This can often impact direct response rates because people may think my emails are spam.

Maybe I need to dig out my Pakled for this...

Weird summer.

1 - Had the HVAC company come out for proactive maintenance. Hours after they left, the temp in my main comms/server room jumped 4 degrees (F) and sat there. Also they discovered a leak in one of our secondary units and, when the guy came out to repair it, told me the 4 degree (F) jump wasn't a problem. Come to find out (thanks to a random comment by my junior tech) that the vents coming in from the primary unit had the angles of the directive flaps changed. Instead of blowing down the front of the server racks, they were changed instead to blow across the top. Needless to say, that's less than useless, so we re-aligned them and resolved that problem.

2 - Ticket: I can't reset MS MFA after getting a new phone. Result: User wasn't using MS MFA. They were using DUO.

3 - Had a ticket come in, "My headset USB cord isn't fitting into the USB port anymore." Come to find out the inner portion of the plastic part of the port snapped off inside the headset cable.

4 - Three laptops received with corrosion from liquid damage

5 - Had a very one-sided conversation (as in, one side of the conversation understood the question, and the other didn't) with another tech about why we don't need to wipe systems when they're used by people who don't utilize the local file system of the computer (focusing on web storage, i.e. Sharepoint).

6 - Had a user complain about getting an email from their own email address. With an attachment and calendar invite. Opened the attachment. At least they didn't scan the QR code inside. Had to quarantine that system for a month.

7 - Over two dozen tickets of people who forgot their passwords. Within two hours of changing it.

8 - Webchat with a third-party contact management site. One of their "helper apps" keeps causing MS Defender to throw malware alerts. Their answer: "Oh, your users aren't licensed, which is why you're getting those alerts."

9 - User: You know when you're on vacation, and you just need to log into check on things? Me: No, because this summer was my first vacation in 6 years while at this job, and I definitely didn't check in.

10 - Same user as #9 - by the way, I'm driving, can you remote into solve my problem?

Oh yes, it's so nice to be retired and not have to worry about those kinds of things anymore! :cool:

Quote:

---

Originally Posted by Mileron

Weird summer.

1 - Had the HVAC company come out for proactive maintenance. Hours after they left, the temp in my main comms/server room jumped 4 degrees (F) and sat there. Also they discovered a leak in one of our secondary units and, when the guy came out to repair it, told me the 4 degree (F) jump wasn't a problem. Come to find out (thanks to a random comment by my junior tech) that the vents coming in from the primary unit had the angles of the directive flaps changed. Instead of blowing down the front of the server racks, they were changed instead to blow across the top. Needless to say, that's less than useless, so we re-aligned them and resolved that problem.

2 - Ticket: I can't reset MS MFA after getting a new phone. Result: User wasn't using MS MFA. They were using DUO.

3 - Had a ticket come in, "My headset USB cord isn't fitting into the USB port anymore." Come to find out the inner portion of the plastic part of the port snapped off inside the headset cable.

4 - Three laptops received with corrosion from liquid damage

5 - Had a very one-sided conversation (as in, one side of the conversation understood the question, and the other didn't) with another tech about why we don't need to wipe systems when they're used by people who don't utilize the local file system of the computer (focusing on web storage, i.e. Sharepoint).

6 - Had a user complain about getting an email from their own email address. With an attachment and calendar invite. Opened the attachment. At least they didn't scan the QR code inside. Had to quarantine that system for a month.

7 - Over two dozen tickets of people who forgot their passwords. Within two hours of changing it.

8 - Webchat with a third-party contact management site. One of their "helper apps" keeps causing MS Defender to throw malware alerts. Their answer: "Oh, your users aren't licensed, which is why you're getting those alerts."

9 - User: You know when you're on vacation, and you just need to log into check on things? Me: No, because this summer was my first vacation in 6 years while at this job, and I definitely didn't check in.

10 - Same user as #9 - by the way, I'm driving, can you remote into solve my problem?

---

Friday morning: Get into the office and see messages about how one of the primary SQL DB servers flipped overnight, but was made OK quickly by one of the guys in the UK office. He also fails over to the secondary server. While attempting to compile a RCA I contact Dell support. Tech#1 response: Well, it's working now, so we have nothing to look at.
:confused:
Saturday morning at 0017: I get a call from our audible alert system. The same SQL DB server flipped again. I rush down into my kitchen, grab my kit, head back up to my office, and start troubleshooting. It's back online; it was apparently just a reboot. I'm thinking, "huh, this reminds me of the time one of our ESXI hosts went TU due to a bad NIC firmware." I gather logs, upload to a new Dell ticket (Tech#2, who sends some "next steps"), monitor it for another hour, and go back to bed (1:55am)

Saturday morning at 3:05: I get another call from the audible alert system. Same thing, it has bounced itself. I'm too tired to diagnose at this point, so I turn off the audible alert system and go back to bed (3:10).

0900 I eat something small at home, pick up donuts for the building security guys, and I'm finally in the office by 10, at which point I discover that server has been boot looping for a couple hours.
First steps: Record the boot loop two times through.
Then I follow up on the recommendations sent to me by Tech#2.
This includes a full power removal, flea drain (unplugging all cables and holding the power button for 30+ seconds to drain all capacitors), and reseating the NIC that is being reported by the logs as having a problem (gee, why does this sound familiar).
I power the server back on, which is on for maybe an hour before it flips again into a boot loop.

11:15 Then because I don't want to wait to play email tag, I call into Dell support and get a guy with a strong French accent, who did not introduce himself. I have to ask him to repeat himself several times when I determine that he wants me to do another flea drain. I tell him I've just done it and the server boot looped. He insists. I tell him I have to set him down as the server room is too loud to hear someone on a phone, I go do so, and come back, to hear him talking in French to (someone who is not me) and after I try to interrupt him, the line goes dead.
:mad:
I upload more logs.

1145 I call back and I get a courteous dude named Mario. He tells me it can't be the NIC causing the boot loops. He asks for more logs, which I provide again. I tell him that I've got records going back four years from NICs causing boot loops on various servers. He insists that cannot be it, starts pointing me to the "unusual" layout of RAM in the server (320GB, with four 8s, and the rest 16s). As this is how it was shipped from Dell (it's in the freaking product specs when you look up the service tag) I tell him if it were in fact a memory issue based on the layout, it would have appeared after we installed the damn thing, NOT TO MENTION all the logs that I sent over specifically show a NIC failure right before the flip.
Also, I tell him that while reseating the NIC I discovered that the fan on the NIC does not spin gracefully or easily. (Gee, maybe the stupid customer was right)
So I tell Mario this.
He says, and I try to quote verbatim, "The fan won't spin easily unless it has power."
:wtf:
He tells me to reseat the RAM (which I don't do, because it's now 1215pm, twelve hours after the initial onset, and the server hasn't boot looped - it just full on shut down while I was trying to read Windows event logs. Mario tells me to flea drain it and bring it back up to get more logs. :confused: Like, what? I've put in four sets of logs over the last 12 hours and they all show the same thing. I get back to my desk from resetting the unit, which is now not boot looping at all, but just getting to a black screen then shutting off, and find my call has been disconnected.

:angry:

1230 I manage a few more things to get the server up and running and so far, it's behaving. No errors, no boot looping.

1330 my manager calls me to check up on me. Offers to have one of the other guys in my local team tag me out. I tell him that's not fair, and I'll see it through.

1430 The server is still toodling along nicely. After snacking on some Chinese fried noodles that I keep in my desk for migraine relief (to relieve the migraine I have), I call back. I get Charlie.

Charlie takes four minutes to read the logs and says "OK so this NIC... tell me about it."
I tell him about the errors.
"Yes, I see those"
I tell him I reseated it and yet it still eventually boot looped. That I've had this issue with bad NICs in other servers which caused boot loops. That the fan doesn't spin. He reviews internal KB articles and sees that the error in the logs has been documented as being a NIC problem.
So while I tell him that there's a fan issue, we're poking around and looking at the logs together, and I'm showing him my concerns.

1555 We're coming up on two hours since the last reboot (which seems to be roughly the time of effect) and having a fantastic conversation about movies, TV, Marvel vs DC entertainment, and Halloween decorations when the server flips and, instead of booting, simply shuts down.
He orders me a NIC for 4 hour turnaround delivery.
yay
1610 I update my team, make plans to get food, and just as I'm about to get up from my desk I get a call from the courier dispatch confirming that I'm present, because they'll have it out to me in under 45 minutes. The building security portal is undergoing a transition so I go downstairs 40 minutes after this call to meet the courier.

1707 I get a call from the courier. "I'm (several blocks away from the delivery), can you come meet me?"
:wtf: Motherfucker, NO. You're supposed to come to ME. I give him landmarks/cross streets.
Fifteen minutes later I get a followup call. "I'm outside your building in a Honda Civic, with my flashers on."

Keep in mind yesterday 10/11 was the start of the Nor'easter rolling up the East Coast US, so I step out of the building and it is fucking pouring.
I look up and down the street and see him a block and a half away, in front of the wrong building. I don't want to wait for this MFer to roll around the block to the correct address so I literally run out and meet him. He lets me sit in his car for a couple minutes to breathe and sign the receipt of package (which was nice of him) and I run back to the building.
The girl behind the security desk offers me their heater to dry off, but I decline and go back upstairs.
After drying off as best I can, I find a random jacket left behind at a departed employee's desk and go into the server room (one foot and my shirt still soaked) and install the new card.
The fan on it spins freely. Imagine that.

I boot it up, grab a fresh set of logs, upload to the ticket, and finally (1810) go out to get myself something to eat. The weather is leading me to need Pho, so I now that the rain stopped briefly I go to the Pho restaurant two blocks down from my building. It's closed. Like, closed closed. :doof:

I end up getting Ramen from a place on the other side of my building's block.

After the two-hour mark has elapsed from the initial bootup, and the server is still humming along nicely, I pack it up and finally get home by 2115.

It's now just about 1000 and the server is green across the board and has been online for the better part of sixteen hours. Since I did all the legwork I'll have to write the RCA tomorrow but so far... knocks wood

Huh i never would have thought an overheating NIC could crash a system instead of just a loss of network connectivity. But lol at it needs power to spin freely. No a good fan can always spin, this is why you wedge something in them when blowing out a computer so that they do not just spin up. I saw a video once where someone overspun a fan with a blower and it threw a blade.

Yesterday got kudos from my manager (and a free day off, which I've requested Halloween but has yet to be approved), and most of my team.

Then my manager gives me a new project - build out a dictionary of words that we don't want people to use as part of their passwords. (I'm hoping this is a step towards longer-than-90-day resets).

So far I've got about a hundred and fifty words on that list.

Out of the "focus test group" of 20, 18 complained that they couldn't come up with a password in the first few minutes of being asked to reset.

Maybe that's a good thing.

Non-network Device security has become a large weekly task of mine.

I've been back and forth with our SIEM provider about bending their system's limitations into a way to automatically prune our tenancy for various things.
1 - To automatically remove devices that have not been scanned within 28 days
2 - To automatically remove devices that are duplicates OR do not correlate to itself.
2b. For example: If the on-asset scan agent reports an IP of 10.100.10.100 then the internally hosted scan server finds that device on another IP (such as if the VPN of the user dropped) there should not be a "new" asset entry added - it should correlate to the prior asset and autoclear within minutes, if not no longer than 24 hours

We've implemented changes to the way that we handle spares patching.
As in, now we don't. (Finally, after me asking for that for almost four years.)
And I don't mean "we never patch the spares" - I mean, we're no longer going leave them out and online to be patched "whenever" but instead in the event of a scheduled deployment we bring them online 24+ hours in advance in order to facilitate those patches.
Additionally, we're now limiting hot spares to one per device level on the totem pole.
Generally this means three hot spares.
This reduces our reportable vulnerability surface.
Also, having 10+ spare laptops sitting out on desks is a security hazard anyway.

Because ideally, if a machine is online, it should be getting all of its patches within a couple hours of being online, right.

Except about two weeks ago, we received a raft of over a dozen new and refurb devices.

Some return shenanigans occurred, and all has been sorted now about that.

When we receive refurbs, we immediately wipe the drive for security reasons(1), login to the base OS, generate the hardware hash, upload it into Intune, re-trigger OOBE, and it goes through Autopilot.
A full refurb setup should generally take an hour, including app installs.

However out of the refurbs, six are absolutely refusing to receive Windows patches.

These six seemed great. Until they didn't patch.

I spent all day Thursday the 19th with one next to me trying to troubleshoot it.

When running Check for Updates, it shows "downloading" then "installing" then fails with some random 0x800..... error.

Each of these errors basically says "C4U had a canary. Delete the C4U files and reboot and try again."

So I did that.

I also have a couple scripts in our RMM that reset Windows Background Updates Services (BITS, etc) and repair the OS cache (dism, which has helped a fair few times!), which leads into a reboot, which did not work.

I even tried unencrypting the drives and doing a flea-drain on the motherboard (Dell Latitudes) which did not work.

Normally after 3-5 failures the C4U window throws a "Hey we noticed we can't update correctly, but we can reset the OS for you and save all your settings!" except that isn't appearing.

My next step planned was to reset the OS from the device (Reset this PC) and Keep All Settings - just to refresh the OS build (again).

Other tasks became more important: We had our monthly maintenance period on the 21st, then I got the flu, so they were farthest from my mind for the last few days.

Earlier today I get a message from one of my overseas counterparts.

CP: I've noticed these six devices have high vulnerability counts
Me: Yeah, they're the recently received refurbs. We can't get them to patch, so I have notes on them to be OS reset, and if worse comes to worse, completely reinstalled from USB ISO.
CP: Did you try to reboot them?

Look, I get he's trying to help.

I'm still too sick to deal with someone(2) questioning my methods when I could see if I forgot to reboot ONE out of the six, but if a reboot fixed six machines randomly, without someone touching them for five days, I'd be ecstatic.
(Notwithstanding the fact that my junior tech rebooted them just yesterday when he was working on them.)

I'd be happy to admit (because I've done this recently) that sometimes leaving the damn thing alone for 24+ hours might result in something positive occurring(3)
Because then I'd know I could stop spending time on them when I've got more important things to tackle.

Footnote 1: We received a half dozen laptops from a refurbisher, a couple with evidence someone must have either taken them home to play with or was doing something shady at "the office" because of the cracked software we found on them.
Footnote 2: This is from a tech who I watched (via webcam) try to use a screwdriver pry off the battery connector from a laptop motherboard because he didn't listen to my instruction that "gently pull by the plastic loop on the battery cable" meant the plastic loop attached to the battery cable.
Footnote 3: For some reason, both the batches of new and refurb laptops (the ones that are patching correctly, anyway), a couple Macs I've worked on, and a few other changes that I've needed to review, have needed 24-48 hours to "rest and reset". For example, we had one device that, no matter what, we couldn't trigger the encryption. Kept giving a policy error. Even when refreshed from Intune. Even when retriggered through gpupdate. It just. wouldn't. apply. But I told the tech working on that device to just let it sit online (making sure to disable Sleep) for 24 hours. The next time he tried it, it encrypted just fine.

But it's been several hours now that the original Counterpart is still messaging me the lack of success.
He's been working on them for most of HIS shift today, and still nothing has changed.

I told him he could try to reset the OS remotely, but as I'm not in the office I'm not guaranteeing they won't get stuck on a Bitlocker Recovery screen or have some other unexpected failure that would prevent him from continuing.

My team supports a large, nationwide retail chain.

Spring is always chaos. It always comes out of a nearly silent winter, because even though our market in the west is driven all year long, the rest of the nation doesn't do much sales between November and February.
Instead of having us do heavy lifting then, they get us to raise our blood pressure like crazy during the 3 weeks before solar seasonal spring.

We had a customer ergonomics upgrade come through the pipeline for our paint counters.
Lots of upgrades. Tablet kiosks, wifi label printers, internet order integration. Hardware arms, etc.

Instructions were beautifully written. Two very critical steps were written highly in accurately. One step calls for calibration of the new installs to confirm proper integration to the paint tint dispensers (tinters).

I had a single location out of the whole west coast that was an alpha deployment at the end of last sprinf. It was easy enough. Engineer written installation, but I figured it out.

Now we have enterprise deployment with a 10 day cutoff for project completion. Support agents aplenty at support room. But critical step in the instructions call for the tech to grab a can of paint (not the one that the tinters actyally requires). Huge hang ups on this that no one realized. The entire enterprise had new barcode imaging scanners (scan guns) back ordered to replace "defective" ones.

I noticed the error and noted it.

The agter correcting the issue, first one to do it I was the first one in who managed to scan it accurately.

Nexr step says, "press dispense button and confirm label printer has dispensed label."

Only, this also dispenses extremely deep color tint out of the tinter. No warn or mention to out either the paint can (open) beneath the dispenser, Or to put a purge can underneath either.Paint tint splashed all over my jeans.

I called the deployment manager and explained a need for revision.

"what kind of a person is going to think dispense button isn't going to dispense tint?"

"the one who reads that the printer is going to dispense a label."

Dude was pissed. "do you know how many revisions we went through to send out these instructions?"

"well, it doesn't realky matter becuase right now you have three thousand eight hundred eighty four backordered barcode imaging scanners being placed because your revised instructions are telling field service technicians to scan the wrong paint can, and your support team thinks everyone has a defective scan gun. So... How much do each of those scan guns cost *just to ship via overnight?*"

" okay, I've revised it. "

Me, early 2024: Windows 10 EOL is on the horizon. We should look into testing and schedule rolling laptop purchases to remove incompatible devices from the fleet because MS won't support Win10 updates after October 2025
... and late 2024, and early 2025
Me, October 2025: Windows 10 is now EOL. We're going to get alerts every month about devices not getting patched.
Me, November 2025: Please remember, Win10 is EOL, these devices cannot be patched because we did not pay for the extended support
... and December, January 2026, February, March

Team, yesterday: Why are all of these Windows 10 machines unpatched?