this post was submitted on 09 Jul 2023
21 points (100.0% liked)

Sysadmin

5625 readers
1 users here now

A community dedicated to the profession of IT Systems Administration

founded 5 years ago
MODERATORS
 

Hey all, in my company we've been having a lot of trouble with our first-line support team and I wanted to get some ideas how it works in other companies.

To give some context, I work in a Customer Team (L2-L3 Support) for a MSP, previously I belonged to the Internal Operations Team and they had a very negative view on the first-line team, with opinions like:

  • we don't need them
  • they lack knowledge
  • management can't create a good first-line team because they don't want to invest

But I didn't interact a lot with them before, but now, I have to interact with them on a daily basis, and I see some things that have started to make me worried about the team:

  • They ignore KB's
  • They say that they don't have access to certain servers, or that they don't find the correct credentials and just pass the ticket for us to solve
  • They have people that lack knowledge in some basic support, I have had tickets passed on with notes like "I don't know how to use Linux"

From my point of view and the team I belong now, we all think that management didn't really verify the required knowledge for some members of that team, but they really have a few that are trying really hard to improve their skills.

We have started to try to help them, so that our job can also become easier:

  • Improve the language in legacy KB's
  • Simplify the process in the monitoring platform with more directions
  • Automating some processes so that the first-line can execute fixes without having the required knowledge on the backend
  • Picking the best members of their team and promoting them to our team

That team also has some problems that I fully recognize:

  • Shit pay
  • Bad leadership, that team has had 6 different Team Leaders in a short time (I have been here for only 2 years)
  • Lacking interview and requirements for the position

Sorry for the long text, would love to have some feedback from your sides, or is this normal in a lot of companies?

top 12 comments
sorted by: hot top controversial new old
[–] Wxfisch 10 points 1 year ago (1 children)

In most companies I’ve worked for, T1 is there to put in tickets from calls, and handle the simplest of tasks (password resets, account lockouts, “have you tried turning it off and on again” tasks). Anything beyond that is generally sent to T2 (usually the desktop team who then force other teams to accept tickets as needed) and T3 for anything that more systemic or needs deeper troubleshooting and system knowledge.

In many places it’s a combination of piss poor pay creating little motivation and high turnover (and thus lack of training) and management prioritizing the wrong metrics (generally looking for short call times and short call queues). If you want to try and improve things I’d suggest learning about the KPIs that team is expected to meet, and then ask management why they chose those metrics. Generally I’ve found prioritizing first call resolution over call times to be a huge improvement to motivation of the team and user satisfaction scores (we all like solving problems and users tend to be way nicer when you fix the issue vice kick the can).

I would say, at least to your point about them not having access to systems, that’s it’s very common for T1 to have pretty limited admin access to systems. Partly to protect against inexperience, but also as a social engineering protection. If they need to ask for access to pass a ticket for elevated rights, it gets another set of eyes on the call to ensure it’s all kosher.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

Thanks a lot for that insight, I have asked current and previous members of the Team about the KPI's and unfortunately they are clearly not alligned with ours.

They have stuff like:

  • Don't keep Incidents with your, only solve if it's fast (minus 5min), else pass to the expert team
  • Can't keep a Change for more than 5 days, even if it's related to waiting for a client response

They do have credentials to access servers, even with root privileges on some servers (not that we think that's safe), but there's a lot of infighting between their management and other Customer Teams, they're management want them to be able to touch everything but at the same time, doesn't invest in training and want's the tickets to be gone fast.

There's also a lack of leadership and training, currently they have a week of onboarding and it's just watching some Udemy videos.

[–] reddig33 7 points 1 year ago (1 children)

In a lot of companies, the front line support personnel are one step up from a chat bot. They are there to either look up a pre-formed answer for a common problem, or route the caller to a specialist who can help them. The jobs are usually low pay, and outsourced, so the employees don’t really have a stake in the success of the company.

Sometimes these jobs can be an entry level into applying for a position with the company. Employees who realize this often try to go a bit further in their efforts.

[–] [email protected] 3 points 1 year ago (1 children)

I really wished they could be a bit better than a simple human "chat bot".

I try sometimes to teach them what I can, they're all in house and local.

I would hope the company could see them as a "first step" to the Customer Support.

Maybe I'm just trying to make them something they're not.

[–] TornadoRex 8 points 1 year ago (1 children)

You also have to think about what metrics your company is using for them. A lot of times for front line phone support it’s length of call. They’re not going to dig in they’re going to push it to the first relevant department they can.

[–] ElectroNeutrino 2 points 1 year ago

This reminds me of the common saying, "if your metrics become targets, then they are no longer metrics."

Worked at one place where they looked at first call resolution for the front line techs, and they would often just mark a ticket as resolved because they told the caller to restart and call the support line back afterwards, or something similar.

[–] [email protected] 2 points 1 year ago (1 children)

@StuffToWrite Our org is on the smaller side and we outsource our front line to another company in the same city. It’s… challenging sometimes but they’re acceptable for what we hired then to do. The org has a resource they can speak to immediately since all the on-prem staff are slammed. Even if they cannot resolve it and have to make a ticket anyway.

Although if I get one more ticket from the T1 techs saying they tried to reset an O365 password (we don’t have) I’m might scream.

[–] [email protected] 1 points 1 year ago

In our company it seems like there's different mentalities on what they should be.

We want them to act like juniors for the Customer Team, but their Team management want them to be just like those old phone operators that pass on to the right person.

[–] [email protected] 2 points 1 year ago (1 children)

We recruit people with no IT background sometimes and we are doing great, consistently beating every other branch since a massive corporation bought us out. We have a great team that helps eachother out, but more importantly we work closely together with all the senior people within the company, most starting out in first line themselves at some point.

If there's something a first liner is interested in they can schedule an appointment with someone who is our main player in that field and have a personal little course in that platform/whatever.

Seems your guys are very "us and them", which helps no one.

[–] [email protected] 1 points 1 year ago

Yeah, what you're saying from your company is what our Customer Team wants to do with the first-line support team, but they belong to a Manager that reports to a different stakeholder that we do, unfortunately they see the first-line support team differently.

[–] [email protected] 2 points 1 year ago (1 children)

It is pretty typical in my experience for tier 1 to take a ticket and escalate.

The first step in improving it with most organizations I'm at is helping get better information from tier 1.

Once you put a form and policy in place you can go from the first example to the second which is very helpful. If the ticket templates can be pre-populated depending on the type of ticket for those tier 1 workers, you can also bake in troubleshooting for some of the common issues. If the issue isn't well understood and requires out of the box troubleshooting, I don't expect them to be able to deal with it.

Original ticket: Ticket#23232 User states they cannot access Z drive. Reboot no avail.

New ticket once we have a policy and form: Ticket#23232 Desktops --> Drive Access --> Existing access issue

File path: User is attempting to access //company/hr/forms Username: user.name Hostname: hr-3423423-lt Can they access other shared drives?: Yes Are they in the proper group? (See KB2332 to confirm) Error message: error.png Confirm file path is being entered or mapped correctly(y/n): Y Rebooted system(y/n): Y Account locked(y/n): N Account expired(y/n): N Confirmed VPN connection(y/n) etc etc

[–] [email protected] 1 points 1 year ago

Yeah I think the problem with my company is that they want both things.

That they can be able to solve easy issues following a KB and to just escalate tickets if they take a bit more time or knowledge to solve.

We're trying to solve the easier issues with Rundeck jobs that they can execute, but I think that will bring a new bag of issues:

  • If the Rundeck job fails, they will have to not only include the issue but also the error in the Rundeck Job
  • How can they check (or even have the knowledge) to know if the Rundeck Job worked like it should, to avoid situations where they said the issue is resolved, but then the client returns the ticket saying that nothing changed
  • If a Rundeck job needs some parameters, that may add a bigger level of knowledge for them to have, to avoid running the job on the wrong machine for example
load more comments
view more: next ›