Tales from Techsupport

[Melcar]

Is it bad that I do almost wish the world had ended today?

Not after that conversation.

[FilanFyretracker]

wow how can that person be a tech? seems pretty simple to me unhide the drive or move the program to a drive that everybody who needs it has the permissions to always see.

[Merrick ap'Milandra]

I completely forgot about this until I posted in Jonas' windows 8 thread.

So one of our clients has a fully virtualized environment with both on-site SAN backups and offsite cloud backups.

Veeam/VMWare/vSphere/vcenter/whatever that beta tool is that allows granular exchange mailbox restores from Veeam is called.... with a side order of Citrix for the dumb terminals, 10 or 12 NIC cards on each ESXi box (one in, one out - for each function, fully VLAN'd network to keep the virtual machine backups from interfering with or being interfered with by regular LAN traffic, not a bad setup for a small-to-medium business. It has plenty of room to scale and it's nice and easy to work with except for the occasional Veeam annoyance.)...except for...well, you'll see...

I get a call about an app that won't work inside of citrix on a thin client. No biggie there, probably just something about the user having their permissions locked down nice and tight, as I always prefer to do within citrix setups.

I hop onto the citrix farm and start peeking around.

Huh. It's not a published app. What's it called again?

{NameOfApp} We use it to monitor the security doors.

A quick search reveals nothing on the farm.

So, I pull up the Wyse Device Manager and shadow the guy's session.

*blink*

There's a big huge connection button labelled RDP to Terminal Server.
And another one for the Citrix session he usually uses.

"RDP?" ...I think to myself... "I thought they moved away from terminal services when we installed their citrix farm and virtualized everything?".

So, we sign him into the RDP session.

800x600 screen resolution...and that tell-tale light blue background.

It's a...windows 2000 server?

Sure enough, there's the application, right on the desktop. Crappy looking icon though. Whatever. I shrug and figure "enterprise software" and run the app so I can figure out what, exactly, is failing to launch on the citrix side. I briefly wonder why it's not installed on the citrix servers if it's such a necessary application for building security...



...and watch in horror as I am presented with a 16 bit application...



...that is now testing the COM ports for connectivity...

Yep, this application came straight out of the windows 3.1.1/95/98/NT era and hadn't been upgraded since.

Now that I have the error, I get off of the phone with the user so I can "fix it" (he's impatient and doesn't want to be on the phone anyway) and walk around the department in a brief daze, having not seen a 16 bit application since I tried to make Dungeons & Dragons: Fantasy Empires work on a win2k install back in late 2000, TWELVE YEARS AGO and well before I knew about DOSbox. (Totally unplayable on a 1ghz p3. No CPU speed correction, but I digress)

So a few minutes later, I stumble on one of the coworkers involved in the deployment of said virtualized environment.

"Oh yeah, {NameOfApp}? Yeah, that was a pain. The servers in the citrix farm are all 2008 R2 64 bit instances. I ended up having to limp an old 2k box to life because the vendor said: 'Do not under any circumstances upgrade to our .NET version. It doesn't work. At all.'. I had to find something that'd replicate COM ports over an Ethernet connection. It took me the better part of a week to reconfigure all of the Wyse Terminals to allow them into both TS and Citrix, get the COM port app up and running, and then train them on the differences. I'll show you where the COM port software is, it probably just quit working, then you can explain, again, that they can only access {NameOfApp} from their old terminal services connection. Only 2 people need access, they just both need post-it notes or something."

I never thought I'd see a 16-bit-ONLY app in the wild in today's day and age with how our setups normally work, but even the VENDOR thinks it's the only workable version of their application.

I knew the vb->.NET porting hurt a lot of software developers with long-established code trees, but I figured that was mostly over with half a decade ago...

Uff Da.

Full-on production environment, 16 bit app necessary for Sarbanes-Oxley compliance.

[Merrick ap'Milandra]

Oh, and I should add that part of the reason my coworker took so much time to switch them over is that we ripped out NEARLY EVERYTHING they had when we put in the virtualized environment.

...because they were running their entire LAN on public addresses.

192.168.*?
172.16.*-172.32.*?
10.*?

NOT GOOD ENOUGH FOR US, WE'RE GOING TO USE..uhhh... 173! (or 72 or something, they either upped it by one or dropped the 100, I forget) YEAH! NOBODY WILL EVER GUESS THAT! (To be fair, I don't think 173 used to be public addressing space, but it's been at least a few years since that opened up. One would think that "we just randomly can't hit these websites" would prompt some level 1 pinging or perhaps a few inside/outside tracert commands)

He had to go dig up old switches and figure out what was where to even FIND the machine that was running the app. I don't remember for sure if he found it or just built one from parts but in the last 7 months I've seen far too many places using publicly addressed space for their LAN IP scheme.

[FilanFyretracker]

what type of security needed a 16bit app for SOX compliance? you mentioned building security but I know there must be 32bit and maybe even 64bit applications that can talk to door locks. Actually kinda surprises me that SOX would even care how the building is secured as long as it has fixed access points that can control access.

[Merrick ap'Milandra]

what type of security needed a 16bit app for SOX compliance? you mentioned building security but I know there must be 32bit and maybe even 64bit applications that can talk to door locks. Actually kinda surprises me that SOX would even care how the building is secured as long as it has fixed access points that can control access.

If they need building ingress and egress tracked with badges and they need to have logs going back X far of who entered where and when, then they pretty much have to keep track of it.

I know there are 32 bit apps that do this because I've used them before but they're wedded to this particular app.

[FilanFyretracker]

ahh so its like the IE6 problem on Intranets, They have an ancient app and are stuck with it.

I am guessing that to upgrade to software from this century they would have to upgrade the physical layer too(RFID units and the badges themselves.)

[Mileron]

Same guy from the arbitrary backup post

Me: I need a replacement disk for my server's data raid array - it's a 72GB 10K U320 drive.
Him: Okay, Let me see what I have

I receive a box but due to the pre-holiday rush am unable to open it let alone install it. Once I finally DO get around to opening it, I find that it's a 72GB U320 drive but 15K instead.

Me: Hi, I got the drive, thanks, but unfortunately it's a 15K drive. I need a 10K drive.
Him: You can use that
Me: Um, no, I can't. RAID requires identical drives. Capacity, connection, rotation speed.
Him: You can use that.
Me: No, I can't.
Him: Dude, yes you can. Just plug it in, the array will rebuild your data.
Me: And then it'll crash.
Him: Just use it.

[Charbok]

Speaking of drives...

We had a NetApp located out of state for DR purposes. We're a small IT/hosting company, so we don't have full time (hell, or even part time) staff down there. So when a drive fails, we hire a contractor to perform the physical swap.

NetApp ships the drive. The drive is delivered. We call the contractor, give him the location and contact info, and have him take possession of the drive and perform the swap.

The datacenter then tells the contractor that they have misplaced the drive. They'll search for it, just give them a few hours.

A day later, we call them and speak to a receptionist. She'll check into it for us. An hour later, we get an e-mail stating they have found the drive. We call the contractor, and while speaking with him, receive a second e-mail stating that Oops, they were wrong, and the drive is still misplaced.

So we talk to NetApp. They offer to ship a second drive to the datacenter. Agreed, and done. It should be there in a few hours. During this period, the datacenter found the original drive, and put a big note on it explaining what it was, who it was for, and to not lose it again.

They ask us what they want us to do with it, since drive #2 is on its way. We arrange for the contractor to pick it up the next day since he's unavailable now and instruct datacenter to deny delivery of drive #2 so it goes back to NetApp. No problem.

Receptionist never gets the word. Drive #2 is delivered and signed for before contractor arrives. Fine, no problem, just get the return label and ship it back to them. Receptionist does so.

Tech flunkie sees the note saying "ship back NetApp drive" and ships drive #1 alongside drive #2. Net result: zero drives at facility when contractor arrives.

And that's where we're at now. NetApp is shipping drive #3. This time, UPS is holding the drive at their shipping facility so contractor can cut out the middle men.

[Melcar]

And that's where we're at now. NetApp is shipping drive #3. This time, UPS is holding the drive at their shipping facility so contractor can cut out the middle men.

Ick... Next time - Call the contractor first, arrange shipment straight to Contractor.

We've had warranty repairs that went like that: Company IBM, HP whoever, "OK, we'll send parts & tech..."

Tech arrives. "Where's the parts?"

Not here yet.

[Mileron]

Customer: I need to e-file a return
Me: They stopped accepting e-filing of all forms for that tax form on December 1, 2012
Customer: I need to e-file a return
Me: The IRS stopped accepting those returns at the beginning of December
Customer: I need to e-file a return
Me: You can't
Customer: Well why not?

[FilanFyretracker]

Customer: I need to e-file a return
Me: They stopped accepting e-filing of all forms for that tax form on December 1, 2012
Customer: I need to e-file a return
Me: The IRS stopped accepting those returns at the beginning of December
Customer: I need to e-file a return
Me: You can't
Customer: Well why not?

Sounds like the kind of person that would ask a bridge operator why they cannot close the drawbridge while there is a bigass container ship clearly going by.

Also they are a year late if they are trying to file a return. It is almost time to think about this year's tax returns.

[Mileron]

Also they are a year late if they are trying to file a return. It is almost time to think about this year's tax returns.

Depending on the form, some are accepted for a longer period than 1040 income tax returns.

Here's today's:

Netops: We sent you a replacement (read: refurb) voice gateway and we want you to set it up
Me: We are moving office in 60 days, and we don't want to put it in play, so we are holding it in reserve for mid-move usage.
Netops: We want you to set it up
Me: No
Site Move Contact: They don't have to set it up, in fact it works out in our favor, this way we don't have to apportion/requisition new gear for their new site in advance; we can just use this one for testing
Netops: Fine, but we need you to run IOS commands for config info etc before it can be used for testing.
Me: It has to wait til (today) as I have no idea where all of that stuff is, since the last time we used it was five years ago

Counterpart: It's simple, all you have to do is Hyperterm into it
Me: I've never used Hyperterm, don't know any IOS commands, and hey, you put the only laptop with a DB9 port somewhere I can't find it.
Cp: Oh. Yeah. I'll be in (today.)

Today.
Cp: Here's how you set up Hyperterm.
Cp: (runs a bunch of IOS commands, which we try to copy and paste the results into Notepad.)
Me: Something doesn't look right.
Cp: What do you mean?
Me: It's truncated and scrambled.
Cp: Huh?
Me: (separating the HT window from Notepad) Look, shit's missing and scrambled in the logged output

        Part Numberde
    

If you requi


 478  2E87C5B0ance please co

Processor

or

-------------- Top 100 allo                           
 476  2E87C5A0  2E6CC484   00  0 0TP Ms
 19  

0x60FAB4A8    8891304---------- Mempool st


 485  2E87C5E8  2E6CFF04   0    1 483  2E87C5layed first 2048 Allocator PCs only

Me: So that's not normal to me
Cp: Eh, that's what they asked for, that's what they'll get
Me: Wouldn't this mean there's a major problem with the unit?
Cp: No

Me: wtf?

[Melcar]

It was missing info in the Hyperterm window? Or notepad after you pasted it?

[Mitebe]

yeesh, use putty at the very least. Hyperterminal is such a pita.

[Mileron]

Info was missing in Hyperterm.

So we try Putty. Same thing.

Ended up being a bad console cable, which was brand new from Cisco. At least we had an extra.

[Melcar]

Me: Wouldn't this mean there's a major problem with the unit?
Cp: No

Me: wtf?

So your tech was right then.

[Mileron]

So your tech was right then.

Eh, the last two we got, he swore there was no problem with, but in one, the power supply was bad and the other had bad memory (both of which I called.)

So it's okay to be partly wrong once in a while, but still, he should have clued me in a little, train of thought kind of thing. Maybe "Oh I've seen this before with a bad cable." or something.

After all, it could have gone the other way - it could have been a good cable and bad memory/IOS load/whatever

[Mileron]

Customer: I'm having trouble installing your software.
Me: Okay, what is happening?
Customer: Well we got new Windows 7 workstations. And I'm able to use both the OS and the program with no problem. But when I go to your website, hit "Download", then choose "Save," I pick the folder I want - on my network drive - and hit Save. It downloads quickly - under a minute for a ten MB file - but then a window comes up that indicates "Copying Program_Setup.EXE from C:\Users\Temp to X:\Downloads... with a 9+ minute estimated time.
Me: Okaaaayyy...
Customer: This just started happening Tuesday.
Me: What happened Tuesday?
Customer: I was out of the office. Maybe it missed me and got jealous.
Me: Anthropomorphizing computers aside, are you aware of any system updates?
Customer: No. However our server is ailing, and with less than 2GB free on the data drive (which has a max capacity of 60GB), we're expecting a replacement in the next 60 days.

So I remotely connect into her computer and do the downloads she needs, then try to apply them. Even with Full Control access to the destination folder, she's getting disk space warnings.
I delve into my program's structure, delete a few older installers, some unneeded temp files, that sort of thing, (freeing up about 250MB)and go about downloading all of the update installers to her local C: drive.

So I try the installer again - same issue.

I try to "backdoor" it, and instead install to her C: drive, and go about copying the executables and other files that the update included.

At this point, I discover that copying a 7.72MB file from C: to X: will take over 3 minutes. A 13MB file took over 11. Windows 7's "more details" window displays a 39KB/sec transfer rate.

Me: Well, everything's in place, however you're having a terrible problem with your server. Information is transferring at under 40KB/second, where it should be transferring at over 30 times that (probably more.) You may want to ask your tech guy to either troubleshoot the issue, or expedite the replacement, because this sort of issue is never a good thing...
Customer: Here's my tech
Tech: What's the problem now?
Me: Your gig-ethernet capable server is transferring files at 39KB/second
Tech: That doesn't sound too bad, it's gig-e after all
Me: Maybe you didn't hear me right. I said 39 KILOBYTES per second.
Tech: Yeah, that's riii... wait, what?

[FilanFyretracker]

39kbit/sec? I would consider that unacceptable on my cable modem connection that is only 12mbit.