Page 102 of 103 FirstFirst ... 252929899100101102103 LastLast
Results 2,021 to 2,040 of 2041

Thread: Tales from Techsupport

  1. #2021
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Quote Originally Posted by Nerkahia View Post
    This one does not seem unreasonable to me. Would you be fine with using a login that wasn't accurate spelling of your legal name?

    You say it's not feasible... but it is obviously a standard task associated with new hires. Why would it not also be considered a standard task for name changes by marriage?
    For email, yes. We add an alias, so if Reba McEntire gets married and becomes Reba Rogers, then emails to both rrogers and rmcentire will get delivered to the user's email box.

    However the computer login will not change, as it's tied to the user's network account, and a number of systems that (according to the dev teams whose responsibility they are) cannot be changed for various reasons, which are solidly tied to the user's network account..

  2. #2022
    Ancient Arcanist
    FilanFyretracker's Avatar
    Join Date
    Jun 2003
    Posts
    23,418

    Re: Tales from Techsupport

    While things are what they are, I will say the fact they cannot change shows that whoever wrote that stuff was an idiot. I mean marriage happens and one would think they would consider such a thing and allow one to with proper levels of system access change a username and hit apply and it would fire across the system.
    Today we sail
    On the Solar Rail
    For there's much we just don't know
    So farewell with a kiss
    Then it's fast for the mist
    Till we're sleeping in the cold below

  3. #2023
    Buckeye Wizard
    Nerkahia's Avatar
    Join Date
    Sep 2003
    Posts
    5,413

    Re: Tales from Techsupport

    Quote Originally Posted by FilanFyretracker View Post
    While things are what they are, I will say the fact they cannot change shows that whoever wrote that stuff was an idiot. I mean marriage happens and one would think they would consider such a thing and allow one to with proper levels of system access change a username and hit apply and it would fire across the system.
    Yeah those are kind of my thoughts as well. And I know I'm terribly jaded in this regard, but for me, at the company I personally work at, when an IT systems specialist says it's not feasible, -it translates to "I'd have to Alt-Tab out of Call of Duty for longer than a few minutes."

    JOKING!!! OMG I'm just joking guys and gals.

    Nerkahia
    Retired 85 Wizard of Ascentia, The Nameless

  4. #2024
    Elder Arcanist

    Join Date
    Jun 2003
    Posts
    5,222

    Re: Tales from Techsupport

    Quote Originally Posted by Nerkahia View Post
    Yeah those are kind of my thoughts as well. And I know I'm terribly jaded in this regard, but for me, at the company I personally work at, when an IT systems specialist says it's not feasible, -it translates to "I'd have to Alt-Tab out of Call of Duty for longer than a few minutes."

    JOKING!!! OMG I'm just joking guys and gals.


  5. #2025
    Buckeye Wizard
    Nerkahia's Avatar
    Join Date
    Sep 2003
    Posts
    5,413

    Re: Tales from Techsupport

    Quote Originally Posted by Eremius View Post
    Probably completely fair ~

    Nerkahia
    Retired 85 Wizard of Ascentia, The Nameless

  6. #2026
    Elder Arcanist
    Bonlainy's Avatar
    Join Date
    Mar 2004
    Posts
    5,475

    Re: Tales from Techsupport

    Quote Originally Posted by Eremius View Post
    'This world may be another planet's hell.'{Aldous Huxley}
    'After silence, that which comes nearest to expressing the inexpressible is music.'{Aldous Huxley}

  7. #2027
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Sometimes... yeah.

    And all in one day.

    I'm on my way out to lunch, and there's two Amazon delivery guys standing in our elevator vestibule, arguing. I'm waiting for my car when they start walking back and forth, trying both sets of doors into our suite.

    Me: Can I help you guys?

    #1: Yeah, is this 1701?

    Me: Yes, this building is 1701, and you're on the 26th floor

    #1: visibly confused, looking at all the signage But is this 1701?

    Me: Yes, this is 1701 Enterprise Street, and right now you're on the 26th floor.

    #1: still staring around, looking back and forth between signage and the scanpad in one hand, and package in the other

    #2 pipes up: We're looking for a Gene Luck Pikkerd?

    Me: You're in the right place. Suite 3700 or 37th floor?

    #2: Yeah

    Me: Thanks, I'll sign for it.

    #1: Okay so does this next one that says "25" on it mean we need to go to the 25th floor?



    I'm finally out at lunch, because I left it on the kitchen table when I was prepping. I'm on my way back to the building when in front of me on the escalator are two young ladies with polling surveys for some PA thing.

    They walk up to a random guy as I'm trailing behind them trying to get out to the street and say

    Voter1: Hey, can we bother you for a few minutes? We have some questions about your opinion on (whatever voter issue) in Pennsylvania

    Random Guy: No, I'm from Jersey

    I slowed my walk, I had to hear this

    Voter2: Okay so you work in the city right? How do you think (whomever will do whatever)

    Random Guy, slower: I'm from New Jersey...



    It's around 4pm and I'm making my 4pm rounds, making sure the coffee machine is off, checking conference room TVs/projectors, things like that. I barely sit down at my desk when one of the younger sales guys who is present mid-week brings a DHL driver over to me.

    Sales Guy: Delivery dude wanted a signature and I didn't know what to do so I brought him to the ol... IT guy because I figured you'd know what to do.

    Me: sticks out my hand for the sign-pad I got it, thanks

    Delivery Guy: Did he really just call you old?

    Me: He also needed to check with IT-Dad if the mailman could deliver a package, so probably.


  8. #2028
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Mondays and Fridays are the emptiest in my office, with four or less people both days.

    Today I was fully solo(asterisk definitions below) for the first time in a couple months. I've got music on when I'm at my desk. Random music scores. This morning it was Alan Silvestri's "Flight of the Navigator".

    I'm doing my start of day/start of week stuff, including my morning office rounds, and hear music, and smell food, but there's no one present. Check the door logs - no one present except myself.

    Get back to my desk and the food smell is stronger. I follow my nose and find a guy who set up a device replacement on Friday - when I was out of office, and for which my junior tech didn't inform me of. I ask him how he got in.

    "Oh, the front door was unlatched."

    I go and check it. He's right, the magnetic strike plate on the doors isn't working (again). I shoot an email to the office manager, and handle his issue.

    About two hours after that, I hear a random phone alert through my headphones. It's not mine, it's on silent. I turn around and there's another random dude, just looming the f behind my chair.

    RD: So I'm working at home Friday morning and I realize I can't get logged in to my computer
    Me: Did you put in a ticket?
    RD: I can't get logged into the computer
    Me: You can still access the helpdesk site with a mobile device
    RD: How is that going to help me?
    Me: Instead of wasting a day of PTO because you can't get logged in, you could have had someone that was available help you
    RD: Do you know how nice it was on Friday?
    Me: No, I was in the hospital all day
    RD: Well sucks to be you. You gonna help me with my password problem?



    Turns out when he changed his password earlier in the week, he "forgot how many exclamation points" he used.


  9. #2029
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    User: I have half an hour free for my computer swap
    Me: I'm ready when you are
    User: Here you go, I'm going to pump breast milk
    Me: Um... TMI, but I need you here for this

    -==--==-

    After fixing a different issue for a different user for the second time:
    Me: Okay so we're done for now. Let's restart, and you can continue working. Tomorrow morning we'll check to see how it's behaving.
    User: You mean this restart won't let me work until tomorrow morning?
    Me: Not at all. The last time this issue presented, it took twelve hours from the time we fixed and confirmed, until the issue returned. So I'll schedule a meeting for tomorrow at 8:30am so we can double check, but until then you can continue using your stuff.
    User: Are you sure I don't need to wait overnight?
    Me: Yes

    ^^ So this was due to a Windows driver update (and a Realtek driver update) causing the user's 23 day old laptop's webcam to stop working. As a proactive measure after the above exchange I contacted Lenovo who stated that the driver provided by Realtek (which they host on the Lenovo support site, and is available through their automagic update downloader) is in fact bad, and they've known it for two weeks, but for some reason no one decided it would be a good idea to pull it from their site.

  10. #2030
    Poof make squid!
    Merrick ap'Milandra's Avatar
    Join Date
    Jan 2004
    Posts
    14,500
    Blog Entries
    1

    Re: Tales from Techsupport

    So, I can't believe this hasn't happened before in multiple decades of working in IT, but the other day...

    I'm on a teleconference session with another person from my team, a software engineer at the vendor, and an end user, trying to troubleshoot an extremely weird issue. (Why more vendors don't build in "Impersonate user" capabilities in their software so we could test this stuff without inconveniencing people, I will never know. Probably because they can't be arsed to allocate budget to actually SECURE anything these days...)

    It's taken months to get this end user and the vendor together so the vendor can tick the box on their stupid policy that says they actually saw the issue happen or whatever flaming hoops that vendor forces their poor support staff to jump through in order to actually do their jobs.

    We let the vendor basically run the call and at one point, they want to see the network requests the browser is making, so the vendor engineer advises the end user to hit F12 to bring up browser dev tools...

    ...and we watch in horror as the "Airplane Mode" icon slides up onto the screen and freezes there.

    While I have no idea what it looked like for that end user, thankfully the end user was smart enough to figure out what happened, undo it, and get back on the call, but that totally could have gone a different direction.

    Uff da.
    For copyright purposes, all of my posts are covered under the "Do What The Fuck You Want To Public License"
    http://sam.zoy.org/wtfpl/
    Noone should sue or be sued ambiguously.

  11. #2031
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    8.

    8 laptops since December first.

    Dead and pretty much gone. All from different users.

    1 - User dumped coffee on it, and wanted me to overnight her a replacement. Um, no. You live within 12 miles of the office. Get off your (couch) and come in for it. After drying it for two weeks, while I was able to get it to power on, it won't POST. Issues with active warranty are causing delay in repair/replace.
    2 - User is in meeting. Fan ramps up. Screen goes black. Machine shuts off. Still isn't powering back on. Out of warranty.
    3 - User is in meeting. Computer gets really slow, screen flashes. Computers gets fast, and a checkerboard of video corruption appears. We tell him to shut it down, let it cool off, and try again. A week later, the screen flashing thing happens again, but this time half of the screen vanishes and Windows somehow scrunches a usable resolution into the top half of the screen. (Pending return of this one. I can't wait.)
    4 - User's house burned down.
    5 - User is in meeting and randomly shuts off. Receive it, evidence of liquid damage. However, the entry point is the track pad and the bottom edge of the unit. Turns out the user has a medical condition where they sweat profusely, and after hearing that I remember that there was a previous machine that had (only) the trackpad killed by it. However after several weeks of drying and some judicious isopropyl alcohol cleaning, I've got it booting up. Haven't decided if I want to re-deploy it.
    6 - Machine randomly "loses" half its installed RAM. When I use it, it works fine. When my intern uses it, the active RAM slots randomly switch with every reboot.
    7 - Machine spontaneously shut off. Bring into office, works fine. Takes home, works fine. Sits in a different room, spontaneously shuts off. Tries a different power adapter. Fuse blows, laptop died.
    8 - Machine blue-screens whenever the right shift key is used. Replaced keyboard. Replaced RAM. Works fine with a stress test or with someone like me who doesn't use the right shift key (grade school injury makes it hard for the right pinky to twist in that direction, which caused my keyboarding teacher no end of conniptions) however after the last stress test, it stopped powering on at all.

    And that's not even getting into the dozen or so units less than a year old that keep "losing" their webcam and microphone drivers.

  12. #2032
    Poof make squid!
    Merrick ap'Milandra's Avatar
    Join Date
    Jan 2004
    Posts
    14,500
    Blog Entries
    1

    Re: Tales from Techsupport

    6 - Machine randomly "loses" half its installed RAM. When I use it, it works fine. When my intern uses it, the active RAM slots randomly switch with every reboot.
    Magnetic bracelet?
    For copyright purposes, all of my posts are covered under the "Do What The Fuck You Want To Public License"
    http://sam.zoy.org/wtfpl/
    Noone should sue or be sued ambiguously.

  13. #2033
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Quote Originally Posted by Merrick ap'Milandra View Post
    Magnetic bracelet?
    Good thought, but no. We never did figure out what was causing it, so long as he didn't use it. We tried different RAM. Different power adapters. Different desks. The common denominator for reproducing the issue was the intern.
    We had our network engineer test it out, works for him, he uses it now as his permanent WFH device.

    Here's two recent ones

    Sales guy (ticket): I'm trying to dial Canada, but none of the calls are going through
    Me: (replying/discovery) We need some example numbers
    SG: +62...
    Me: That's Indonesia...
    SG: No it's not



    and

    Ticket from a different sales guy: My email is showing two identical signatures. Can this be fixed? This can often impact direct response rates because people may think my emails are spam.

    Maybe I need to dig out my Pakled for this...

  14. #2034
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Weird summer.

    1 - Had the HVAC company come out for proactive maintenance. Hours after they left, the temp in my main comms/server room jumped 4 degrees (F) and sat there. Also they discovered a leak in one of our secondary units and, when the guy came out to repair it, told me the 4 degree (F) jump wasn't a problem. Come to find out (thanks to a random comment by my junior tech) that the vents coming in from the primary unit had the angles of the directive flaps changed. Instead of blowing down the front of the server racks, they were changed instead to blow across the top. Needless to say, that's less than useless, so we re-aligned them and resolved that problem.

    2 - Ticket: I can't reset MS MFA after getting a new phone. Result: User wasn't using MS MFA. They were using DUO.

    3 - Had a ticket come in, "My headset USB cord isn't fitting into the USB port anymore." Come to find out the inner portion of the plastic part of the port snapped off inside the headset cable.

    4 - Three laptops received with corrosion from liquid damage

    5 - Had a very one-sided conversation (as in, one side of the conversation understood the question, and the other didn't) with another tech about why we don't need to wipe systems when they're used by people who don't utilize the local file system of the computer (focusing on web storage, i.e. Sharepoint).

    6 - Had a user complain about getting an email from their own email address. With an attachment and calendar invite. Opened the attachment. At least they didn't scan the QR code inside. Had to quarantine that system for a month.

    7 - Over two dozen tickets of people who forgot their passwords. Within two hours of changing it.

    8 - Webchat with a third-party contact management site. One of their "helper apps" keeps causing MS Defender to throw malware alerts. Their answer: "Oh, your users aren't licensed, which is why you're getting those alerts."

    9 - User: You know when you're on vacation, and you just need to log into check on things? Me: No, because this summer was my first vacation in 6 years while at this job, and I definitely didn't check in.

    10 - Same user as #9 - by the way, I'm driving, can you remote into solve my problem?

  15. #2035
    Elder Arcanist
    Bonlainy's Avatar
    Join Date
    Mar 2004
    Posts
    5,475

    Re: Tales from Techsupport

    Oh yes, it's so nice to be retired and not have to worry about those kinds of things anymore!

    Quote Originally Posted by Mileron View Post
    Weird summer.

    1 - Had the HVAC company come out for proactive maintenance. Hours after they left, the temp in my main comms/server room jumped 4 degrees (F) and sat there. Also they discovered a leak in one of our secondary units and, when the guy came out to repair it, told me the 4 degree (F) jump wasn't a problem. Come to find out (thanks to a random comment by my junior tech) that the vents coming in from the primary unit had the angles of the directive flaps changed. Instead of blowing down the front of the server racks, they were changed instead to blow across the top. Needless to say, that's less than useless, so we re-aligned them and resolved that problem.

    2 - Ticket: I can't reset MS MFA after getting a new phone. Result: User wasn't using MS MFA. They were using DUO.

    3 - Had a ticket come in, "My headset USB cord isn't fitting into the USB port anymore." Come to find out the inner portion of the plastic part of the port snapped off inside the headset cable.

    4 - Three laptops received with corrosion from liquid damage

    5 - Had a very one-sided conversation (as in, one side of the conversation understood the question, and the other didn't) with another tech about why we don't need to wipe systems when they're used by people who don't utilize the local file system of the computer (focusing on web storage, i.e. Sharepoint).

    6 - Had a user complain about getting an email from their own email address. With an attachment and calendar invite. Opened the attachment. At least they didn't scan the QR code inside. Had to quarantine that system for a month.

    7 - Over two dozen tickets of people who forgot their passwords. Within two hours of changing it.

    8 - Webchat with a third-party contact management site. One of their "helper apps" keeps causing MS Defender to throw malware alerts. Their answer: "Oh, your users aren't licensed, which is why you're getting those alerts."

    9 - User: You know when you're on vacation, and you just need to log into check on things? Me: No, because this summer was my first vacation in 6 years while at this job, and I definitely didn't check in.

    10 - Same user as #9 - by the way, I'm driving, can you remote into solve my problem?
    'This world may be another planet's hell.'{Aldous Huxley}
    'After silence, that which comes nearest to expressing the inexpressible is music.'{Aldous Huxley}

  16. #2036
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Friday morning: Get into the office and see messages about how one of the primary SQL DB servers flipped overnight, but was made OK quickly by one of the guys in the UK office. He also fails over to the secondary server. While attempting to compile a RCA I contact Dell support. Tech#1 response: Well, it's working now, so we have nothing to look at.

    Saturday morning at 0017: I get a call from our audible alert system. The same SQL DB server flipped again. I rush down into my kitchen, grab my kit, head back up to my office, and start troubleshooting. It's back online; it was apparently just a reboot. I'm thinking, "huh, this reminds me of the time one of our ESXI hosts went TU due to a bad NIC firmware." I gather logs, upload to a new Dell ticket (Tech#2, who sends some "next steps"), monitor it for another hour, and go back to bed (1:55am)

    Saturday morning at 3:05: I get another call from the audible alert system. Same thing, it has bounced itself. I'm too tired to diagnose at this point, so I turn off the audible alert system and go back to bed (3:10).

    0900 I eat something small at home, pick up donuts for the building security guys, and I'm finally in the office by 10, at which point I discover that server has been boot looping for a couple hours.
    First steps: Record the boot loop two times through.
    Then I follow up on the recommendations sent to me by Tech#2.
    This includes a full power removal, flea drain (unplugging all cables and holding the power button for 30+ seconds to drain all capacitors), and reseating the NIC that is being reported by the logs as having a problem (gee, why does this sound familiar).
    I power the server back on, which is on for maybe an hour before it flips again into a boot loop.

    11:15 Then because I don't want to wait to play email tag, I call into Dell support and get a guy with a strong French accent, who did not introduce himself. I have to ask him to repeat himself several times when I determine that he wants me to do another flea drain. I tell him I've just done it and the server boot looped. He insists. I tell him I have to set him down as the server room is too loud to hear someone on a phone, I go do so, and come back, to hear him talking in French to (someone who is not me) and after I try to interrupt him, the line goes dead.

    I upload more logs.

    1145 I call back and I get a courteous dude named Mario. He tells me it can't be the NIC causing the boot loops. He asks for more logs, which I provide again. I tell him that I've got records going back four years from NICs causing boot loops on various servers. He insists that cannot be it, starts pointing me to the "unusual" layout of RAM in the server (320GB, with four 8s, and the rest 16s). As this is how it was shipped from Dell (it's in the freaking product specs when you look up the service tag) I tell him if it were in fact a memory issue based on the layout, it would have appeared after we installed the damn thing, NOT TO MENTION all the logs that I sent over specifically show a NIC failure right before the flip.
    Also, I tell him that while reseating the NIC I discovered that the fan on the NIC does not spin gracefully or easily. (Gee, maybe the stupid customer was right)
    So I tell Mario this.
    He says, and I try to quote verbatim, "The fan won't spin easily unless it has power."

    He tells me to reseat the RAM (which I don't do, because it's now 1215pm, twelve hours after the initial onset, and the server hasn't boot looped - it just full on shut down while I was trying to read Windows event logs. Mario tells me to flea drain it and bring it back up to get more logs. Like, what? I've put in four sets of logs over the last 12 hours and they all show the same thing. I get back to my desk from resetting the unit, which is now not boot looping at all, but just getting to a black screen then shutting off, and find my call has been disconnected.



    1230 I manage a few more things to get the server up and running and so far, it's behaving. No errors, no boot looping.

    1330 my manager calls me to check up on me. Offers to have one of the other guys in my local team tag me out. I tell him that's not fair, and I'll see it through.

    1430 The server is still toodling along nicely. After snacking on some Chinese fried noodles that I keep in my desk for migraine relief (to relieve the migraine I have), I call back. I get Charlie.

    Charlie takes four minutes to read the logs and says "OK so this NIC... tell me about it."
    I tell him about the errors.
    "Yes, I see those"
    I tell him I reseated it and yet it still eventually boot looped. That I've had this issue with bad NICs in other servers which caused boot loops. That the fan doesn't spin. He reviews internal KB articles and sees that the error in the logs has been documented as being a NIC problem.
    So while I tell him that there's a fan issue, we're poking around and looking at the logs together, and I'm showing him my concerns.

    1555 We're coming up on two hours since the last reboot (which seems to be roughly the time of effect) and having a fantastic conversation about movies, TV, Marvel vs DC entertainment, and Halloween decorations when the server flips and, instead of booting, simply shuts down.
    He orders me a NIC for 4 hour turnaround delivery.
    yay
    1610 I update my team, make plans to get food, and just as I'm about to get up from my desk I get a call from the courier dispatch confirming that I'm present, because they'll have it out to me in under 45 minutes. The building security portal is undergoing a transition so I go downstairs 40 minutes after this call to meet the courier.

    1707 I get a call from the courier. "I'm (several blocks away from the delivery), can you come meet me?"
    Motherfucker, NO. You're supposed to come to ME. I give him landmarks/cross streets.
    Fifteen minutes later I get a followup call. "I'm outside your building in a Honda Civic, with my flashers on."

    Keep in mind yesterday 10/11 was the start of the Nor'easter rolling up the East Coast US, so I step out of the building and it is fucking pouring.
    I look up and down the street and see him a block and a half away, in front of the wrong building. I don't want to wait for this MFer to roll around the block to the correct address so I literally run out and meet him. He lets me sit in his car for a couple minutes to breathe and sign the receipt of package (which was nice of him) and I run back to the building.
    The girl behind the security desk offers me their heater to dry off, but I decline and go back upstairs.
    After drying off as best I can, I find a random jacket left behind at a departed employee's desk and go into the server room (one foot and my shirt still soaked) and install the new card.
    The fan on it spins freely. Imagine that.

    I boot it up, grab a fresh set of logs, upload to the ticket, and finally (1810) go out to get myself something to eat. The weather is leading me to need Pho, so I now that the rain stopped briefly I go to the Pho restaurant two blocks down from my building. It's closed. Like, closed closed.

    I end up getting Ramen from a place on the other side of my building's block.

    After the two-hour mark has elapsed from the initial bootup, and the server is still humming along nicely, I pack it up and finally get home by 2115.

    It's now just about 1000 and the server is green across the board and has been online for the better part of sixteen hours. Since I did all the legwork I'll have to write the RCA tomorrow but so far... knocks wood

  17. #2037
    Ancient Arcanist
    FilanFyretracker's Avatar
    Join Date
    Jun 2003
    Posts
    23,418

    Re: Tales from Techsupport

    Huh i never would have thought an overheating NIC could crash a system instead of just a loss of network connectivity. But lol at it needs power to spin freely. No a good fan can always spin, this is why you wedge something in them when blowing out a computer so that they do not just spin up. I saw a video once where someone overspun a fan with a blower and it threw a blade.
    Today we sail
    On the Solar Rail
    For there's much we just don't know
    So farewell with a kiss
    Then it's fast for the mist
    Till we're sleeping in the cold below

  18. #2038
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Yesterday got kudos from my manager (and a free day off, which I've requested Halloween but has yet to be approved), and most of my team.

    Then my manager gives me a new project - build out a dictionary of words that we don't want people to use as part of their passwords. (I'm hoping this is a step towards longer-than-90-day resets).

    So far I've got about a hundred and fifty words on that list.

    Out of the "focus test group" of 20, 18 complained that they couldn't come up with a password in the first few minutes of being asked to reset.

    Maybe that's a good thing.

  19. #2039
    Elder Arcanist
    Junior Alt-a-holic
    Mileron's Avatar
    Join Date
    Jun 2003
    Posts
    8,120
    Blog Entries
    26

    Re: Tales from Techsupport

    Non-network Device security has become a large weekly task of mine.

    I've been back and forth with our SIEM provider about bending their system's limitations into a way to automatically prune our tenancy for various things.
    1 - To automatically remove devices that have not been scanned within 28 days
    2 - To automatically remove devices that are duplicates OR do not correlate to itself.
    2b. For example: If the on-asset scan agent reports an IP of 10.100.10.100 then the internally hosted scan server finds that device on another IP (such as if the VPN of the user dropped) there should not be a "new" asset entry added - it should correlate to the prior asset and autoclear within minutes, if not no longer than 24 hours

    We've implemented changes to the way that we handle spares patching.
    As in, now we don't. (Finally, after me asking for that for almost four years.)
    And I don't mean "we never patch the spares" - I mean, we're no longer going leave them out and online to be patched "whenever" but instead in the event of a scheduled deployment we bring them online 24+ hours in advance in order to facilitate those patches.
    Additionally, we're now limiting hot spares to one per device level on the totem pole.
    Generally this means three hot spares.
    This reduces our reportable vulnerability surface.
    Also, having 10+ spare laptops sitting out on desks is a security hazard anyway.

    Because ideally, if a machine is online, it should be getting all of its patches within a couple hours of being online, right.

    Except about two weeks ago, we received a raft of over a dozen new and refurb devices.

    Some return shenanigans occurred, and all has been sorted now about that.

    When we receive refurbs, we immediately wipe the drive for security reasons(1), login to the base OS, generate the hardware hash, upload it into Intune, re-trigger OOBE, and it goes through Autopilot.
    A full refurb setup should generally take an hour, including app installs.

    However out of the refurbs, six are absolutely refusing to receive Windows patches.

    These six seemed great. Until they didn't patch.

    I spent all day Thursday the 19th with one next to me trying to troubleshoot it.

    When running Check for Updates, it shows "downloading" then "installing" then fails with some random 0x800..... error.

    Each of these errors basically says "C4U had a canary. Delete the C4U files and reboot and try again."

    So I did that.

    I also have a couple scripts in our RMM that reset Windows Background Updates Services (BITS, etc) and repair the OS cache (dism, which has helped a fair few times!), which leads into a reboot, which did not work.

    I even tried unencrypting the drives and doing a flea-drain on the motherboard (Dell Latitudes) which did not work.

    Normally after 3-5 failures the C4U window throws a "Hey we noticed we can't update correctly, but we can reset the OS for you and save all your settings!" except that isn't appearing.

    My next step planned was to reset the OS from the device (Reset this PC) and Keep All Settings - just to refresh the OS build (again).

    Other tasks became more important: We had our monthly maintenance period on the 21st, then I got the flu, so they were farthest from my mind for the last few days.

    Earlier today I get a message from one of my overseas counterparts.

    CP: I've noticed these six devices have high vulnerability counts
    Me: Yeah, they're the recently received refurbs. We can't get them to patch, so I have notes on them to be OS reset, and if worse comes to worse, completely reinstalled from USB ISO.
    CP: Did you try to reboot them?

    Look, I get he's trying to help.

    I'm still too sick to deal with someone(2) questioning my methods when I could see if I forgot to reboot ONE out of the six, but if a reboot fixed six machines randomly, without someone touching them for five days, I'd be ecstatic.
    (Notwithstanding the fact that my junior tech rebooted them just yesterday when he was working on them.)

    I'd be happy to admit (because I've done this recently) that sometimes leaving the damn thing alone for 24+ hours might result in something positive occurring(3)
    Because then I'd know I could stop spending time on them when I've got more important things to tackle.

    Footnote 1: We received a half dozen laptops from a refurbisher, a couple with evidence someone must have either taken them home to play with or was doing something shady at "the office" because of the cracked software we found on them.
    Footnote 2: This is from a tech who I watched (via webcam) try to use a screwdriver pry off the battery connector from a laptop motherboard because he didn't listen to my instruction that "gently pull by the plastic loop on the battery cable" meant the plastic loop attached to the battery cable.
    Footnote 3: For some reason, both the batches of new and refurb laptops (the ones that are patching correctly, anyway), a couple Macs I've worked on, and a few other changes that I've needed to review, have needed 24-48 hours to "rest and reset". For example, we had one device that, no matter what, we couldn't trigger the encryption. Kept giving a policy error. Even when refreshed from Intune. Even when retriggered through gpupdate. It just. wouldn't. apply. But I told the tech working on that device to just let it sit online (making sure to disable Sleep) for 24 hours. The next time he tried it, it encrypted just fine.

    But it's been several hours now that the original Counterpart is still messaging me the lack of success.
    He's been working on them for most of HIS shift today, and still nothing has changed.

    I told him he could try to reset the OS remotely, but as I'm not in the office I'm not guaranteeing they won't get stuck on a Bitlocker Recovery screen or have some other unexpected failure that would prevent him from continuing.

  20. #2040

    Re: Tales from Techsupport

    My team supports a large, nationwide retail chain.

    Spring is always chaos. It always comes out of a nearly silent winter, because even though our market in the west is driven all year long, the rest of the nation doesn't do much sales between November and February.
    Instead of having us do heavy lifting then, they get us to raise our blood pressure like crazy during the 3 weeks before solar seasonal spring.

    We had a customer ergonomics upgrade come through the pipeline for our paint counters.
    Lots of upgrades. Tablet kiosks, wifi label printers, internet order integration. Hardware arms, etc.

    Instructions were beautifully written. Two very critical steps were written highly in accurately. One step calls for calibration of the new installs to confirm proper integration to the paint tint dispensers (tinters).

    I had a single location out of the whole west coast that was an alpha deployment at the end of last sprinf. It was easy enough. Engineer written installation, but I figured it out.

    Now we have enterprise deployment with a 10 day cutoff for project completion. Support agents aplenty at support room. But critical step in the instructions call for the tech to grab a can of paint (not the one that the tinters actyally requires). Huge hang ups on this that no one realized. The entire enterprise had new barcode imaging scanners (scan guns) back ordered to replace "defective" ones.

    I noticed the error and noted it.

    The agter correcting the issue, first one to do it I was the first one in who managed to scan it accurately.

    Nexr step says, "press dispense button and confirm label printer has dispensed label."

    Only, this also dispenses extremely deep color tint out of the tinter. No warn or mention to out either the paint can (open) beneath the dispenser, Or to put a purge can underneath either.Paint tint splashed all over my jeans.

    I called the deployment manager and explained a need for revision.

    "what kind of a person is going to think dispense button isn't going to dispense tint?"

    "the one who reads that the printer is going to dispense a label."

    Dude was pissed. "do you know how many revisions we went through to send out these instructions?"

    "well, it doesn't realky matter becuase right now you have three thousand eight hundred eighty four backordered barcode imaging scanners being placed because your revised instructions are telling field service technicians to scan the wrong paint can, and your support team thinks everyone has a defective scan gun. So... How much do each of those scan guns cost *just to ship via overnight?*"

    " okay, I've revised it. "

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •