Tales from Techsupport

[Mileron]

So then I get a response from him with a dozen lines of code.
Dev: If the DvB table is not found in the shared app directory (inside the program folder structure) then Program4-Sub-Print looks for DvOld. If DvB is present, then Sub-Print uses DvC. If Sub-Print is using DvOld, then it isn't finding DvB.

Hmm. He doesn't get it.

Me:
I understand from your explanation how the Sub-Print touches all three sets of the various login tables.
It's unexpected, and it doesn't make sense as to why, and it didn't happen in prior versions to the point where there were problems.

Based on your information in (the developer escalation owned by my coworker), we told the customer to rename their DvB files.
It didn't fix their error due to the DvOld files being zero-sized and resulted in the missing file error.
Easy enough to replace the those tables with blank/default login tables.

However, that also doesn't answer (my escalation).
If I use a prior version of the Program4-Sub-Print with the current/street Program4, and use all three of the customer's login tables, there is nothing preventing me from correctly accessing the Sub-Print program's functions.
If I use the current version of Program4-Sub-Print and remove DvC, Program4 creates new copies immediately upon opening the program. Upon inspection these files include identical information to those located in the DvOld tables. However, I still am unable to utilize the customer's login name (which is listed in the DvB under the shared folder structure) and use the Sub-Print.
Further testing:
(screenshot of the identical information between DvOld and DvB)
As you can see, the (shared folder structure\DvB login file) has identical information to (Program4 folder structure\DvOld login file).

I compared these to the (shared folder structure\DvC login file) table.

Between the two different tables we have:

DvB table has 8 unique values
DvC table has 4 unique values
There are 7 values in common with both tables

When I try to log in to Program4 using the unique logins from the Program4's DvC table, there is no Sub-Print functionality, as per (my escalations') "grayed out" option buttons.

When I try to log into Program4 using the logins from the DvB table, the Sub-Print works fine.

When I try to log into Program4 using the logins that are common to both tables, the following occurs:

Logins from the DvB all are able to access the Program4-Sub-Print correctly.
Login D, which is a security-level in both files (to allow for general use) is getting the inability to edit progrma options depicted by the grayed out button. This is expected behavior.
Login M, which is a security-level 1 (view only user) in the (shared folder structure\DvC), is bizarrely a senior user in the (shared folder structure\DvB) and (Program4 structure\DvOld) files. While this is an unusual configuration (I can't even figure out how they would have managed this) the user login into Program4 prevents them from adding information but has full capabilities to print. This is NOT expected behavior.

Circling back around to the DvC's logins which don't work - Why don't they?
Ostensibly from this testing, it's because they, for some reason, are not listed in the (Program4 structure\DvOld) file, and thus not listed in the (shared folder\DvB) file.
Additionally, each of the respective users (missing from the DvB and DvOld) are senior level, but per your explanation, since the Sub-Print is sticking its fingers into the peanut butter jar that is (shared folder\DvB), it's not seeing those logins and assumes "worse than view-only user".

So there's the explanation for the problem…
But how do we fix (coworker's escalation)?
Removing the shared\DvB tables will cause Program4 to recreate them - incorrectly, in this case.
Removing the Program4\DvOld tables will cause missing file error, because the Sub-Print needs them and Program4 won't recreate blanks/default value files.
So instead we need to remove the shared\DvB tables AND replace the Program4DvOld with blank/default copies.

Is this ideal?
No, but yes
Yes - Because the Sub-Print does not find, for example, login S in Shared\DvB or Program4\DvOld, it defaults to the permissions which had been assigned to the user upon Program4 login.
No - because removing the previously existing user logins affects the the primary Program1 and any Program2 and Program3 programs prior to 2008.


Circling back up to testing logins, this time with the Program4-Sub-Print the version from December 2014.
I have reverted the Shared\DvB and Program4\DvOld to the customer's files submitted for (my escalation).
Login M, which is a security-level 1 (view only user) in the Shared\DvC, and a senior user in the Shared\DvB and Program4\DvOld files. When printing, the (print options) button is gray and trying to edit the options returns the "You must be a senior level user." This is expected behavior for the user login security level "view only".
Login D, which is a security-level in both files (to allow general user) is getting the inability to edit program options and the grayed out button. This is expected behavior for user login security level "general".
Logins from DvB all still senior level, when printing, function normally and are able to edit Sub-Print options. This is expected behavior for user login security level senior user.


Thus…
There's a bug.

It can go one of two ways.

Either:
The Program4-Sub-Print was always intended to pick secondary login permissions from Shared\DvB, which in prior versions it had not, and up until version released May 2016 no one ever noticed. Now that it does, it does not recognize user permissions that are not present in the Shared\DvB file and thus "drops" the permission, resulting in (my escalation's) error of "You must be a senior user".

Or:
The Program4-Sub-Print ignored the Shared\DvB table up until the 5/2016 release, and something was changed to force it to / "fixed" to cause it to now look at Shared\DvB. Now that it does, it does not recognize user permissions that are not present in the Shared\DvB file and thus "drops" the permission, resulting in (my escalation's) error of "You must be a senior user".


The overall fix:

Repair the bug.

Either: Put Program4-Sub-Print back to the way it was prior to this release
Or: Do something additional* to cause the Program4-Sub-Print to better differentiate the user login tables so that user login permissions carry over from Shared\DvC when they are also not listed in Shared\DvB.
*Do something additional = that will not modify, edit, or break the existing respective login tables.


About an hour and a half later I get

Dev: A bug ticket has been entered for this issue.

[Merrick ap'Milandra]

About an hour and a half later I get

Dev: A bug ticket has been entered for this issue.

Your QA skills are FUCKING IMPECCABLE.

I worked QA for years and I was good at it. You're approaching guru level.

Have you ever used (or do you have enough code knowledge to use) fiddler for tracing? https://www.telerik.com/download/fiddler/fiddler2

It's my ABSOLUTE last resort with an app gone wrong, and I usually go to Russinovich's "Process Explorer" first when it comes to finding dll hooks and whatnot, but if you have to pull out the massively spammy guns, fiddler can really nail down information you already suspect.

The ending does not surprise me, I fight with this shit every day.

I actually had an issue the other day where our security team, in anticipation of a PCI DSS v3.1 audit this quarter updated ALL of the security rules on ALL of the firewalls for the ENTIRE COMPANY, Including client firewalls (we're an ISP).

This resulted in 100% of the technicians, provisioners, onboarding folk, cutover team, the people who work in remediation for customer non-payment and disconnects, and pretty much anyone else that ever needs to access a customer device (switch, router, firewall, phone controller, load balancer, modem, you fucking name it) being completely unable to sign in because it broke all RADIUS/TACACS logins FOR THREE DAYS.

It's STILL not back up for all techs a week later.

Restoring from backups? No, we didn't have those for most devices. Configs take space *facepalm*.

No, we didn't follow ITIL processes, that would require paperwork and this was a firmware update for security reasons.

What's a rollback plan?

The worst part is, before anyone would EVEN send an announcement out, I was telling everyone "put it tickets. Each person. Every one of you. They're not going to believe it's widespread unless you ALL put in tickets. Fuck it, you're just standing around unable to work anyway. PUT. IN. TICKETS"

Shortly after that announcement from me, a notification went out.

OOPS! We're aware of an issue!

*sigh*

If I'd have just reported one through the normal channels, it would have gotten kicked back with no notes or commentary.

It took purposely fucking up the ticket metrics to get attention on this.

[Dawlin]

Mad props. Pinning down stuff like that is why you should NEVER interrupt someone debugging.

Quite a crystal castle that you had to construct and hold in your head until the pieces fitted together.

[Mileron]

Your QA skills are FUCKING IMPECCABLE.

I worked QA for years and I was good at it. You're approaching guru level.

I very much appreciate the kudos.
The sad thing is, this isn't the first time I've had to do something like this in the twelve years I've been in this position (I'm averaging 4-5 times a year perhaps), but it is probably the most extensive in the last couple years.

Also, the developers to whom such messages are addressed always seem to feel insulted by it, as if they're not doing their job. There are five now, but 6+ years ago there were fifteen.

And that's not the tone with which I'm trying to convey my message. I try to keep it matter-of-fact.

I'm trying to say, in succinct English (yes, it's their first language) "Hey look there's a problem" and it's up to them to read through it, but I guess it's the proprietary "it's my baby" thing that makes them think they cannot be wrong or there cannot be a problem.

Dr-Evil-small.jpg

[Eremius]

There is an old adage that if you want a mule to do what you want, you first need to get its attention with a 2x4.

In my experience I have not found developers to be significantly different. 2x4 is optional.

[Mileron]

There is an old adage that if you want a mule to do what you want, you first need to get its attention with a 2x4.

In my experience I have not found developers to be significantly different. 2x4 is optional.

I call them "clue-by-fours".

As in, beat them upside the head with.

[Eremius]

Stolen!

[Mileron]

About an hour and a half later I get

Dev: A bug ticket has been entered for this issue.

I've rescinded my dancing banana.

Three more Program4-Sub-Print executables are due to be released today and guess what?

All three of them have the same fucking bug.

[Ialnyien]

I've rescinded my dancing banana.

Three more Program4-Sub-Print executables are due to be released today and guess what?

All three of them have the same fucking bug.

At that point it's no longer a bug, it's a feature

Sent from my SM-G930V using Tapatalk

[MI Redeux]

Your QA skills are FUCKING IMPECCABLE.

I worked QA for years and I was good at it. You're approaching guru level.

Have you ever used (or do you have enough code knowledge to use) fiddler for tracing? https://www.telerik.com/download/fiddler/fiddler2

It's my ABSOLUTE last resort with an app gone wrong, and I usually go to Russinovich's "Process Explorer" first when it comes to finding dll hooks and whatnot, but if you have to pull out the massively spammy guns, fiddler can really nail down information you already suspect.

The ending does not surprise me, I fight with this shit every day.

I actually had an issue the other day where our security team, in anticipation of a PCI DSS v3.1 audit this quarter updated ALL of the security rules on ALL of the firewalls for the ENTIRE COMPANY, Including client firewalls (we're an ISP).

This resulted in 100% of the technicians, provisioners, onboarding folk, cutover team, the people who work in remediation for customer non-payment and disconnects, and pretty much anyone else that ever needs to access a customer device (switch, router, firewall, phone controller, load balancer, modem, you fucking name it) being completely unable to sign in because it broke all RADIUS/TACACS logins FOR THREE DAYS.

It's STILL not back up for all techs a week later.

Restoring from backups? No, we didn't have those for most devices. Configs take space *facepalm*.

No, we didn't follow ITIL processes, that would require paperwork and this was a firmware update for security reasons.

What's a rollback plan?

The worst part is, before anyone would EVEN send an announcement out, I was telling everyone "put it tickets. Each person. Every one of you. They're not going to believe it's widespread unless you ALL put in tickets. Fuck it, you're just standing around unable to work anyway. PUT. IN. TICKETS"

Shortly after that announcement from me, a notification went out.

OOPS! We're aware of an issue!

*sigh*

If I'd have just reported one through the normal channels, it would have gotten kicked back with no notes or commentary.

It took purposely fucking up the ticket metrics to get attention on this.

IDA Pro/Hex-Rays decompiler softwares are amazing for debugging shit like this. If it was developed on Linux you could utilize a recordable/replayable debugger. There are not any for OS X or Windows last time I checked, they are worth their weight in gold though.

[MI Redeux]

IDA Pro/Hex-Rays decompiler softwares are amazing for debugging shit like this. If it was developed on Linux you could utilize a recordable/replayable debugger. There are not any for OS X or Windows last time I checked, they are worth their weight in gold though.

Radare 2's also an option but it has no decompilers, an insanely arcane black magic computing domain with very little open publications available on it and every open source offering being unmaintained or had the team bought wholesale by Hex-Ray.

[Mileron]

Me: How can I help you today?
Tech: A couple weeks ago you sent me instructions on installing your fonts. So I ran through the steps but the customer's still having bizarre wingding-like symbols on his tax forms.
Me: Okay. What OS is his computer?
Tech: Windows 7
Me: And you've logged in as an administrator of the computer?
Tech: Of course, I'm the admin of my computer
Me: ... Wait, I'm confused. You're the admin of your computer, sure, but what about the user's computer?
Tech: What about it?
Me: Is the user an admin of his computer?
Tech: How should I know?
Me: Well, the font install needs to be done on his computer, at his computer, by an administrator of his computer, in order for the tax forms to display correctly. You can't do the install on your own computer and hope the fonts on his eventually match.
Tech: Why not?

[FilanFyretracker]

Me: How can I help you today?
Tech: A couple weeks ago you sent me instructions on installing your fonts. So I ran through the steps but the customer's still having bizarre wingding-like symbols on his tax forms.
Me: Okay. What OS is his computer?
Tech: Windows 7
Me: And you've logged in as an administrator of the computer?
Tech: Of course, I'm the admin of my computer
Me: ... Wait, I'm confused. You're the admin of your computer, sure, but what about the user's computer?
Tech: What about it?
Me: Is the user an admin of his computer?
Tech: How should I know?
Me: Well, the font install needs to be done on his computer, at his computer, by an administrator of his computer, in order for the tax forms to display correctly. You can't do the install on your own computer and hope the fonts on his eventually match.
Tech: Why not?

Oh good ole stupidity.

Of course this is likely why Ariel is so common as a font, Its on all OS.

[Grindel]

Me: How can I help you today?
Tech: A couple weeks ago you sent me instructions on installing your fonts. So I ran through the steps but the customer's still having bizarre wingding-like symbols on his tax forms.
Me: Okay. What OS is his computer?
Tech: Windows 7
Me: And you've logged in as an administrator of the computer?
Tech: Of course, I'm the admin of my computer
Me: ... Wait, I'm confused. You're the admin of your computer, sure, but what about the user's computer?
Tech: What about it?
Me: Is the user an admin of his computer?
Tech: How should I know?
Me: Well, the font install needs to be done on his computer, at his computer, by an administrator of his computer, in order for the tax forms to display correctly. You can't do the install on your own computer and hope the fonts on his eventually match.
Tech: Why not?

The last lines of conversation between the tech and the customer, just prior to your post:

Tech: It looks great when I look at it on my computer now, it's fixed.
Customer: Uhhh.... OK
/twoInterveningWeeksAnd30RebootsLater
Customer: I'm still having bizarre wingding-like symbols on my tax forms.

Software quality assurance is my profession these days. The best way to handle developers is to give them all diabetes by constantly throwing them candy. They'll still react like cavemen but they'll want their sugar supply so they'll at least slightly pay attention to you.

[Zumino Zufeilon]

Me: How can I help you today?
Tech: A couple weeks ago you sent me instructions on installing your fonts. So I ran through the steps but the customer's still having bizarre wingding-like symbols on his tax forms.
Me: Okay. What OS is his computer?
Tech: Windows 7
Me: And you've logged in as an administrator of the computer?
Tech: Of course, I'm the admin of my computer
Me: ... Wait, I'm confused. You're the admin of your computer, sure, but what about the user's computer?
Tech: What about it?
Me: Is the user an admin of his computer?
Tech: How should I know?
Me: Well, the font install needs to be done on his computer, at his computer, by an administrator of his computer, in order for the tax forms to display correctly. You can't do the install on your own computer and hope the fonts on his eventually match.
Tech: Why not?

Even better when the program in question is coming on 20 years old (Developed for Win95 originally), doesn't even begin to understand how anything "modern" system handles things like 'fonts' and hangs when trying to install a font already in use (like the one shared with Logitech Setpoint driver) /boggle.

And people are effing pissed that we're finally pulling the plug on this goddamn piece of shit. SQL based replacement's been around for 7 years, but good-god-damn people are assholes.

[FilanFyretracker]

People are sticklers for never upgrading unless forced. How long did MS keep supporting WinXP even though 7 is superior in every single way.

Or how much of a shit storm was caused when EQ started to require DX8.

Hell Overwatch got heat for requirements of a 64bit Windows. And its a new game.

[Eremius]

but good-god-damn people are assholes.

Quoted for emphasis.

[Mileron]

I post so often in here I can't remember if I mentioned the laptops. In December 2013, after we found out my office would be moving from South Jersey to Philadelphia, we had a whole host of laptops ordered to replace ailing hardware.
Lately, many of the Support and testing staffs have found that, when working with their daily workload of software (especially since our call tracking system went from hosted DB to web-based in November 2015) BSoD became common place.

Several very vocal people were emailing 3-5 managers at a shot, plus me (since I'm in charge of hardware, natch) complaining about said BSoD and how they're losing 10+ minutes productivity to get shut down and booted back up.

So once I found out the company policy of replacing hardware every three years got extended to every four, I talked the technology manager of my group into recognizing that our systems needed additional memory.

We ordered it - after almost three months of back-and-forth with the purchasing group who would not approve the PO request because:
1 - The item on order was computer hardware, and there's a special IT vendor group who handles that
2 - The item on order was not intended to be ordered from a corporate-approved vendor even though sourcing through the CAV was over 100% more expensive ($72 versus 26) and would take 5+ weeks to fulfill
3 - Since the PO wasn't processed through the separate ITV group, they could not begin to schedule IT resources to perform the installs

Fine. I'd done it three times already, and it takes me less than ten minutes.

So eventually the admin assistant just put it through on her corporate card. Ordered from Newegg. And saved over $700. And got the delivery two days later.

I managed to corral most of the users as they came into the office last week and spent 4-10 minutes doing the installs(*).
Four people required me to ship the chips to them. One of them was among the more vocal BSoD complainers.
But, every step of the way, he'd push back and say "I don't have time to do this install." Mother fucker, you work permanently from home and I've seen you in the office a grand total of TWICE since we moved in here eighteen months ago. It took me less than ten minutes the first time I did the install, and if you start with a shut-down PC, I can do it in two or less.
He'd also push back and say "why don't you have my address on record?" Because, asshole, you've moved twice since the office relocated.

Finally today (at my behest) the AA sent three of the "I don't want this delivery" people an email stating "If you don't give us your address you're going to have to suffer with your POS computer for another year and a half because I'm going to return the memory we bought FOR YOU".

She received responses with addresses inside of a half an hour.

Now I just need to get the three remaining chips installed...
One is a PITA because the manager for whom it's intended keeps not showing up in the office with the laptop. Not my fault.
The other two are going to require house calls I sent emails about them setting aside time for me to visit last week but have yet to get a response.

I'll hit them with the "we'll just return it if you don't have this installed by X date" thing if I don't hear anything by next Monday.

* - It still somehow shocks me how many times I need to tell this one woman that she CANNOT leave her system on, in sleep mode or not, and then stuff it into her backpack for her drive. I can't wait for when she fries it.

[Mileron]

We ordered it - after almost three months of back-and-forth with the purchasing group who would not approve the PO request because:
1 - The item on order was computer hardware, and there's a special IT vendor group who handles that

I forgot to mention - the special IT vendor group who handles hardware purchases declined something as basic as a memory upgrade

[Merrick ap'Milandra]

People are sticklers for never upgrading unless forced. How long did MS keep supporting WinXP even though 7 is superior in every single way.

ALMOST every single way.

The removal of hardware profiles SUCKS.

We would have FAR fewer connection issues if you could have a docked and an undocked profile for laptops again. (Docked? Wireless disables. Undocked? Enables. etc.)

Windows 7's insistence to look for all drivers on windows update first also sucks. I can have drivers downloaded, and point to them, and still have it use a driver from windows update EVEN AFTER I DIRECTLY SPECIFY THE MANUFACTURER'S DRIVER.

Even if you tell it not to use windows update for drivers, sometimes it still will.

Now, 90% of the time, this is fine because *most* of the generic drivers on windows update will function just fine, but there are some pieces on certain Lenovo models (T440 audio driver, T450 wireless, T460 USB 3.0 driver) that simply CANNOT function without the manufacturer driver.

The windows firewall is better than XP SP1, but worse than XP SP3.

In general, though, I agree with you.

The main problem with 7, though, is that they moved a TON of stuff around.

It's not as big of a change as xp to vista, or 8, or even 10, but it took my wife a good 6 weeks of using her win7 box to feel truly comfortable with it and she's no slouch in the brains department.