Tales from Techsupport

[Woodja]

Other Tech: :Printer up in this office wasn't working, people had the wrong address on the driver.

Yes, I had to change the IP address on it Monday - Sent you an email about it.

I didn't get it...

Well, I sent it.

Well I didn't pay attention to it then.

You don't say.

We get this all the time from the first line team that also sit in our office:

Tech - Application x isn't working for this user, can you help?

Us - have you checked the emails?

Tech - I don't have time for that

Us - but you have the time to wander over to us and whinge about it?

/facepalm

[Balthis]

I am responsible for auditing who has access to our Data Center on a monthly basis and once a year I have to send out a Policy Review that they need to respond to.

I send out this e-mail 4 times over a 2 month period. "Please read attached documents and respond to me that you have read and understand them. If you do not respond by May 1st your Data Center access will be removed and you will have to go through the full request process to get it reactivated.".

May 2nd rolls around:
Network Tech: Hey, I can't get into the Data Center, my card doesn't work.
Me: Did you respond to the Yearly Policy Review?
Network Tech: What policy review?
Me: Here are the forms you need to have your manager fill out and send to his boss to get approval for you to get your access to the Data Center.
<after some debate, the technician leaves>
Phone Call from Network Tech's Manager: Hey! My guy says you won't let him in the Data Center, what's wrong with you?
Me: ...........

[Merrick ap'Milandra]

HAHAHAHA.

I've had to take to "Yellow" subject lines on stuff like that in order to get people to read them.

Subject: "YOUR DATA CENTER ACCESS HAS BEEN REVOKED"
Body: Please fill out these forms to re-establish access. This e-mail sent from May 2nd 2015 using my time machine.

[Melcar]

I sent an email to the Tech who said he couldn't be bothered to read my email asking him to install a software package on the computer he's already working on - cc'ed the IT director. As I was leaving say to the boss "Sent tech an email just to see if he'd read it".

Boss just chuckles and says : "Ah I wondered why you cc'ed me on that; so did he?"

Don't know yet -- Let you know when he turns the computer over to the user.

[Merrick ap'Milandra]

So I get this ticket last week.
"Users cannot print from (application) to printer. HIGH PRIORITY! URGENT"

Odd, usually not urgent from billing until end of month. Whatever.

I recognize the name of the application, it's one we CAN'T manage or support. We don't even have user logins, let alone admin logins, but I figure, at the very least, I can verify the printer is fine.

It's a shitty little undercutter black and white laserjet. Major brand, but at least a decade old. I should mention this is at a remote location where we only have occasional staff and I'm not one of them.

The problem "just started".

For the fuck of it, I look for documentation. It was the sheerest fucking optimism, really, and was re-inforced. No documentation, nobody knows anything about it, it just "works"...except for once every few years.

This is going to be a loooong haul.

Me: Do you have another printer that's hooked up to this (TERMINAL-BASED FROM 1980-SOMETHING) application?
Them: Well, yes, but it's across the building and we have thousands of these to print.
Me: Could you possibly start printing them there right now while I work on this for you, just in case it takes a while to fix this one?
Them: *frustrated sigh* I guess I have to...hold on...Let me make sure THAT one still works TOO.
(You didn't think to check as soon as your super urgent prints wouldn't come out? Way to care about the time-sensitive work you have to do)

So I start looking into this by first determining if there's an escalation chain for this application, it'll at least tell me what I'm looking at.

There is.

It's the UNIX team. (Except I find out later that it's not, because they know nothing about it. Go-Go-Gadget Workflows in the ticketing system)

Great. Okay, before I escalate anything, I'm making absolutely certain that everything works on my end.
Printer is reachable on print server. Test pages come out from all windows print jobs. Just for good form I send a straight PJL job, a .txt file through the web interface of the printer, and a ps job.

Them: Yeah, I see your prints, it's just when we try to print from the application, nothing comes out.
Me: Okay, let me see what else I can do, I'm just eliminating all of the possibilities under my control, first.

In the meantime, "They" have screamed uphill enough (DESPITE BEING ABLE TO WORK) that someone went "Oh, THAT APPLICATION! Let me put you in touch with the person who knows how to handle this"

Okay:

Expert: What are you doing with {MyApplication}?!?!?!
Me: Nothing, I don't have access to do anything. This one department is having printing trouble, I'm ruling out the windows side.
Expert: Pfft, don't be stupid, just have the UNIX team reset the thing that holds the prints.
Me: Will they know what I mean?
Expert: Yes, we have to do it all the time. Bye.

So I escalate the ticket with what notes I have. I figure at this point it's probably a CUPS Spooler issue, but I have no way to know or tell. Ticket is still high/urgent.

The rest of the week passes.

Monday passes.

I ask for an update to the ticket(s) BECAUSE THERE'S NOW A DUPLICATE.

Them: We're working with UNIX to figure it out.
Me: Okay, let me know what you discover.
*90 seconds passes*
*The ticket has been assigned to your queue by {UNIX PERSON}*

The details read "This is clearly a windows printer issue, because I can print a test page from UNIX".

Cue 20 minutes of me following up with various people.

Them: Oh, yeah, in the meantime we stopped being able to print from windows too.
Me: ohhh-kaaay...

Someone has messed with the windows printer, no idea how, and changed the IP and the Port details from the working IP of the JetDirect to a NON-PINGABLE IP, and changed the queue from RAW:9100 to LPR somethingsomething with SNMP byte count enabled.

WHAT IN THE ACTUAL FUCK?!?!?!

I change it back to the right IP and queue settings and LO AND MOTHERFUCKING BEHOLD, Windows printing works again.

I contact the UNIX guy after *ahem* "obtaining" an enumeration of his SOCKS mappings and said "you activated this printer and pointed it at this IP, it should be this *other* IP, the one in the ticket.

UNIX guy goes and fixes it. "okay, done" and I silently wonder how he actually got a test print or what he as printing to, no worries, he only knows his little silo and this is not something that comes up often. Let's face it, he's on a team of few, supporting legacy applications for hundreds upon hundreds of users. He has bigger fish to fry and I don't believe in raking fellow techs over the coals unless they do so to me first.

Me: Hi! If you could, try to print from {Application} again, please.
Them: We're only getting the first line...and it's in the wrong font.
Me: Okay, it's better than zero prints at all, so it's progress, but it doesn't really help you, let me see what I can do.

STILL.
NOT.
DONE.

It turns out that the application in question, the windows print server, and the UNIX CUPS instance have to be 100% in sync and NOBODY KNOWS WHAT ANYTHING SHOULD BE.

So, several hours of work later and getting berated by people (especially EXPERT who spent most of the time berating me about how noone should be able to touch {ThisApplication} but EXPERT and how confused EXPERT was that I could have solved the problem (THANKS FOR HELPING IF YOU *KNEW* THAT!!!), it was finally taken care of.

I don't know how to break it to you, kiddo, but nobody WANTS to touch your fucking application.

Now I get the duty of documenting how to do all of this shit, because nobody else seems to give a shit.

I hate ticket ping-pong.

Me: "Hey I think this might be something in your wheelhouse, could you take a loo..."
Everyone:

No_u_lovecraftian.jpg

[Mileron]

Me: Sent email back at the beginning of December that our print server was going offline at the end of Dec 2014 to be replaced by some sort of web app that processes print jobs and that any users who have problems with installing a simple app that allows them to search for printers geographically to contact me. I think I helped a half dozen out of the 29 in my group.

CW1 in January: I can't print
Me: Resends email to CW1 - got her set up

CW2 in February: I can't print
Me: Resent email to CW2
CW2: OH yeah I got this but I didn't bother reading it

CW3 in March: I can't print
Me: Resent email to CW3, and also to the rest of my group with reminder. Got CW3 set up.

All of April and May went by with no issues, because Tax Season and cleanup.

CW4 on Monday last week: I can't print
Me: Resent email
CW4: How do I do this?
Me: Go to her desk and go through the steps. Find out the interface between the driver install app and the actual web server is down so I tell her to call the help desk. She's not back in the office until Monday, so she emails her PDF to a coworker who prints it for her successfully.

CW5(one of my two primary problem users) on Wednesday: I can't print
Me: I tell him that there's a problem with the printing process and he should print to PDF and send it to a coworker who has successfully printed and call the Help Desk to indicate he can't set up his printer.
CW5 on Friday: I still can't print
Me: Send it to your coworker again
CW5: No one's in the office to print for me
Me: Call the Help Desk
HDT: None of your printers are set up
Me: Walks around to his desk and said "Look at his email from December 6th. Oh look instructions. He chose not to follow them or contact me in the last 6 months about them."
HDT who I just then found out is good friends with CW5: Dude really? You should know better than that
Me: Finally, someone else witnesses what I've been going through for the last ten years
CW5's manager, in the background: Nodding and giggling to herself (thanks for being helpful)
HDT: Workaround fixes CW5 and I ask him to also fix CW6, who had just come back from maternity leave, which covered the entire period since that email had been sent

Today
CW7 (my other big problem user): I can't print, I'm selecting (old print-server-printer)

....

Maybe it's time for another reminder email

[Melcar]

Maybe it's time for another reminder email

Maybe it's time to print that email out for all to see... Print it to the underbelly of a wet fish - duct-taped to a 2x4....

[FilanFyretracker]

Maybe it's time to print that email out for all to see... Print it to the underbelly of a wet fish - duct-taped to a 2x4....

and then launch that 2x4 from an air cannon through the employee that is a constant problem? naturally the fish should be a sword fish with the email printed on poster sized paper. and stuck on the sword part of the fish before launching.

[Merrick ap'Milandra]

Me: Sent email back at the beginning of December that our print server was going offline at the end of Dec 2014 to be replaced by some sort of web app that processes print jobs

What the fuck manner of wizardry is this?

What's the environment like? Do users have workstations or thin clients? Mostly windows or mostly "other"(linux, unix, mac, os/2, etc)?

Where does the webapp live if not on some sort of print server? Is it cloud-based? Did they have to give all printers public IP's and tunnel traffic or is there a mid-server on prem that...jesus, I can't even wrap my head around this.

Please pass on as many gory details as you feel comfortable with, it'll likely make me feel better about the shit technologies I sometimes have to support.

[Mileron]

What the fuck manner of wizardry is this?

Please pass on as many gory details as you feel comfortable with, it'll likely make me feel better about the shit technologies I sometimes have to support.

Sadly, due to my office move in October 2014, I'm no longer privy to nearly as much of the nuts and bolts as I used to be.

All users have Windows 7 Pro workstations. Some have Mac, but those are primarily developers for other areas that I'm not involved with.

So far as I understand it:
The "host app" gets installed on the user's PC. It's a tiny executable with a bunch of DLLs and no configuration files.
Host App queries a database of available printers (company wide!) and allows you to pick a geographically local device
When you select the printer, you can install drivers for that printer, which results in the "usual" printer object in Devices&Printers.

I actually just looked it up, cuz I've been curious too but my office habitually uses such fly-by-night solutions I never expected them to "go commercial"... hah

http://www.cirrato.com/how-it-works/

So apparently there IS one server that takes care of it all.
I'm 100% certain it's not a local-to-my-office server.
Which also could explain why it seems to shit the bed when more than 2 people try to send a job to a single printer simultaneously.

[FilanFyretracker]

that is an absolutely horrible system if it has the look at he whole company and then find a close printer. Would it not make more logical sense for each building to have a small print management server. I mean what if the internet goes down? does everybody in the building lose track of the local printers?

[Mileron]

As part of the "welcome package" when we moved into our new office, we all received a 4-inch USB mini-fan.

The A/C was... lacking... last week so a few people who had taken theirs home brought them back in this week.

One woman brings it to me and says "it won't spin".

I plug it into USB, flick the power switch on the back, and whaddya know, it turned on.

[Mileron]

that is an absolutely horrible system if it has the look at he whole company and then find a close printer. Would it not make more logical sense for each building to have a small print management server. I mean what if the internet goes down? does everybody in the building lose track of the local printers?

I would think so too - even a low-RAM virtual server! would do the job.

But no.

Instead, it seems the local install of the driver defaults to the server-based spooling.
If the server's unavailable (which would be bad, given we have VoiP phones) it delays print by 2 minutes before using its local settings.

[Melcar]

Would it not make more logical sense for each building to have a small print management server. I mean what if the internet goes down? does everybody in the building lose track of the local printers?

WHAT? Innernets Go DOWN? That's unimpossible!


Can I guess you also use complete cloud document storage? Maybe even all your apps in the cloud?

[Mileron]

Can I guess you also use complete cloud document storage? Maybe even all your apps in the cloud?

Nope, they're steering away from that for now.

[Merrick ap'Milandra]

Sadly, due to my office move in October 2014, I'm no longer privy to nearly as much of the nuts and bolts as I used to be.

All users have Windows 7 Pro workstations. Some have Mac, but those are primarily developers for other areas that I'm not involved with.

So far as I understand it:
The "host app" gets installed on the user's PC. It's a tiny executable with a bunch of DLLs and no configuration files.
Host App queries a database of available printers (company wide!) and allows you to pick a geographically local device
When you select the printer, you can install drivers for that printer, which results in the "usual" printer object in Devices&Printers.

I actually just looked it up, cuz I've been curious too but my office habitually uses such fly-by-night solutions I never expected them to "go commercial"... hah

http://www.cirrato.com/how-it-works/

So apparently there IS one server that takes care of it all.
I'm 100% certain it's not a local-to-my-office server.
Which also could explain why it seems to shit the bed when more than 2 people try to send a job to a single printer simultaneously.

Wow. I'd be printing config pages for the devices closest to me, then adding them directly with tcp ports going to raw:9100.

I'd still have to make sure I had the right drivers, but that's a totally messed up system.

[Mileron]

Me: Hi, I'm having problems with my physically local server. I can ping it, but any application that attempts to access it (such as RDP or any of our database apps) get time out errors. Additionally, mapped drives aren't connecting. I have rebooted and have the same issue. 29 other people are having the same issue.
Helpless Desk: Where is the server located?
Me: Local to me, in the same office
Helpless Desk: So the server is in the same location as you?
Me: Yes it's in the same office.
Helpless Desk: Do you know if the server is near you or if it's elsewhere?
Me: ... It's in the same office as me, and 30 people are having problems accessing it
Helpless Desk: So it's in the same office as you then?
Me: Yes
Helpless Desk: Why don't you see if anyone else is having the same problem as you?
Me: 29 other people are having the same problems
Helpless Desk: And that's your local server, yes?

[Melcar]

Me: Hi, I'm having problems with my physically local server. I can ping it, but any application that attempts to access it (such as RDP or any of our database apps) get time out errors. Additionally, mapped drives aren't connecting. I have rebooted and have the same issue. 29 other people are having the same issue.
Helpless Desk: Where is the server located?
Me: Local to me, in the same office
Helpless Desk: So the server is in the same location as you?
Me: Yes it's in the same office.
Helpless Desk: Do you know if the server is near you or if it's elsewhere?
Me: ... It's in the same office as me, and 30 people are having problems accessing it
Helpless Desk: So it's in the same office as you then?
Me: Yes
Helpless Desk: Why don't you see if anyone else is having the same problem as you?
Me: 29 other people are having the same problems
Helpless Desk: And that's your local server, yes?

Yes, exactly and that's when I shot him your Honor...

[Ronaan]

I got an email today from someone who works six hours per week that she needs pdfcreator or something similar so she can send word 2007 documents via email and they don't look all messed up on the other end.

Plot twist: word 2007 can save pdf documents out of the box.

[FilanFyretracker]

plot twist twist... maybe its time to update from a version of word that is not closing in on being a decade old.