this post was submitted on 13 Sep 2023
72 points (97.4% liked)

Programming

17313 readers
555 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 1 year ago
MODERATORS
 

cross-posted from: https://programming.dev/post/2933587

  1. How much extra do you get paid for being on an call rotation?
  2. Is the salary/benefits the same for inconvenience of being on call and working on an incident?
  3. What other rules do you have? Eg. max time working on an incident, rota for highly unsociable hours?
  4. How many people are on the same schedule with you?
  5. Where are you based, EU/US/UK/Canada?
top 34 comments
sorted by: hot top controversial new old
[–] [email protected] 15 points 1 year ago* (last edited 1 year ago) (1 children)

Policy is 7 day rotation, 24h a day. Must be available to respond within 30m.

  1. ~$800 US a week. More if there are holidays. Get paid even if no incidents occur.
  2. I get phone and Internet reimbursement that normal devs don't get.
  3. There's supposed to be a policy where if I get paged between 10pm and 6am, I don't have to show up to work for 11h. It's not strictly followed in my team, but I always try and get my value from it.
  4. 7 others, so I'm on-call for 1 week every 2 months.
  5. Job is US based, but company is EU owned.

I'm an SRE though, so our on-call is different from on-call for our product devs.

[–] [email protected] 6 points 1 year ago (1 children)

In Germany you must be able to have an 11h break between shifts by law.

[–] [email protected] 3 points 1 year ago

Same in France

[–] [email protected] 14 points 1 year ago* (last edited 1 year ago)
  1. We get paid $70 per weekday and $105 per weekend. I think it's $140 for public holidays.
  2. Eh, it can be a bit annoying at times. It's pretty easy to swap with people as needed. I believe we're allowed to opt out of it too, some of the other devs have. Since we've started it we've tuned our monitoring scripts that false alarms are pretty rare.
  3. Any time spent on incidents is rounded up to 15m. Which can make it feel quite unworth it if you get an alert in the middle of the night. Unsurprisingly since they reduced down from an hour it's taken at least 16m to investigate any alert.
  4. We've got a decent number of people on rotation that I'm only on call about three weeks a year.
  5. Australia
[–] [email protected] 13 points 1 year ago
  1. $150 per day, call out to office is $275, and the start billing hours as per normal. (AU)

  2. Mostly worth it, call outs are rare, but when it rains it pours, so can completely ruin a weekend.

  3. Have to be within 1hr of the office, which implies staying sober.

  4. 3 people, 1 week rotations.

  5. Aus

[–] [email protected] 12 points 1 year ago (3 children)

$150 per week of on-call.

It’s almost never worth it to me (we almost always get at least 1 call during the week, if not 4-5).

[–] MajorHavoc 6 points 1 year ago

I worked at a place like that and I ended up saying no to on-call. It depends on your leverage, of course. I was able to tell them "I'm here to fix your mess, not to roll in it."

[–] [email protected] 6 points 1 year ago

Dang, that's a big difference from the top post.

[–] [email protected] 5 points 1 year ago* (last edited 1 year ago)

Seems like you have some organizational and technical debt in the company that would be worth addressing before agreeing to be on-call

[–] [email protected] 9 points 1 year ago

To work all night and still be expected to be in the office at the same time with no compensation was benefits you got at my last job.

[–] [email protected] 9 points 1 year ago (1 children)
[–] [email protected] 5 points 1 year ago

Same boat here haha

[–] AzPsycho 8 points 1 year ago

I am the lead on a team of 5 and technically we have an on call rotation but it is rarely used. In the last 3 years I have been called after hours a total of 4 times and TBH it was usually due to an AWS outage which resulted in us saying well let's see how long it takes for them to o restore services and then someone run a safety check to make sure shit is normal.

For that I make $120k, have unlimited PTO, and work from home.

[–] [email protected] 7 points 1 year ago* (last edited 1 year ago) (1 children)

We don't have our devs on call at all. Infra / platform ops are and I think they get 750€ per on-call week (not more than one week out of four) which includes two calls or two hours of call duration whichever is reached first.

After that it's another 70€ per call or started hour and it's the same if an expert who is not on call is asked to help out with an issue reported to on-call (but they may not answer / decline as there's never an expectation to be "soft on-call")

Overall that's an okay deal and some sorely needed extra money for the ops guys and gals. But all the same I'm happy that my devs don't need to plan their lives around an on-call schedule.


Edit: Ah sorry, didn't even answer all the questions in OP...

We're in Germany and there is a cooldown time after you fielded an emergency on-call report (which is outside of regular working hours by definition) which is either 8 or 10 hours (not entirely sure since my team doesn't do on-call as previously stated) before you are allowed to start your regular work time for the following day.
Not sure how they tally up working hours for payroll but if you wake up to a call at 3am then certainly no one expects you to be online again at 8am. If you get a call at 10pm however then you get to start working normally the next day. (unless that issue took forever to troubleshoot ofc)

On-call rotations are one entire week per person who participates (which is not mandatory) and the participants per pool must be at least four - which is why they are pooling web admins, DBAs and other ops folk together.
That seems to work okay even though every so often more specialized know-how is required than the current on-call tech possesses for the topic at hand and then they request extraordinary assistance as described above.

[–] [email protected] 1 points 1 year ago

thats a really good deal, thanks for answering

[–] Styxia 7 points 1 year ago (1 children)

My employer is suuuper generous. I get a “shout out” on Slack, and if it’s a big incident my slack profile photo appears on a slide at the company all-hands and the CEO graciously extends his thanks. Sometimes he might even say my name!

I’m on call every 3rd week, no cap on time, usually 3 to 5 people (cross teams). base salary $175k US, no RSUs or 401(k).

I want a new job but not getting many resume bites at the moment.

[–] Potatos_are_not_friends 3 points 1 year ago (1 children)

$175k isn't bad at all. In my company, that's senior level salary.

Also depends on what it entails. No amount of money will ever make me do IT work and support a CEO directly.

[–] Styxia 1 points 1 year ago* (last edited 1 year ago)

Yeah, it’s not bad at all. For context, I’m Senior Staff with > 25yrs experience living in a M/HCOL area, so it’s on the low side. Honestly I’m fine with the base, it’s the casual indifference to the inconvenience, and there’s something about the “cheapness” of the way it’s rewarded that niggles me. Not terrible, not great!

[–] [email protected] 7 points 1 year ago

We’re a team of 7 devs and the on-call schedule is 1 dev per week.

We bill 15% of the hourly rate for being on standby and 150% when responding to an incident. Incidents are fairly infrequent, roughly 1 per month.

The company’s based in the EU.

[–] [email protected] 6 points 1 year ago

I get nothing. So after a while I told my bosses I would simply stop doing it, since the work to compensate us was still "in progress". It helped the rest of the team get a free day per on call week, which I guess is something, but still not enough for me personally.

I told them I wasn't even sure it was legal in my country (Spain) which I guess they didn't even discuss with legal, or legal didn't even blink.

[–] [email protected] 5 points 1 year ago

It's been all over the map during my career. Currently there is no extra for your on call week as was an expectation of employment, but I am only on call one week out of eleven.

[–] [email protected] 5 points 1 year ago

My salary, I guess.

Everybody on my team is required to do on-call once they have enough experience (except for the low budget offshore contractors who I wouldn’t trust to do it anyhow…)

We have 2 people on call at a time, 1 primary and one backup. You do a week on backup, then the next week you’re primary.

There’s no set time limits etc, but if you get sucked into some fire, people are reasonable about letting you take some time off the next day or whatever.

All in all, there are very rarely fires that happen inside or outside of normal working hours. Making the whole team be on call helps incentivize everyone to write more stable code since it’s your own ass on the line.

[–] [email protected] 4 points 1 year ago (1 children)

Would love to see some base salaries posted along with the responses. If you're getting paid shit base maybe this is how they make up for it?

I'm in SRE. No on call benefits at all. Base salary is 175k USD plus 20% annual bonus.

[–] MajorHavoc 2 points 1 year ago

Anecdotally, I've seen the opposite. Places with shit pay tend to not attract the ops talent needed to keep developers out of the on-call rotation. Those orgs tend to fail to recruit both the Ops talent needed to field interesting problems, and the DevOps talent needed to build resilience into the systems allowing Ops to do less on weekends.

[–] InformalTrifle 4 points 1 year ago

Used to be on $140k with on-call paid at $1050 a week (whether called or not). It was later reduced to $550 a week.

In a new job on-call is unpaid but on $230k (plus RSUs)

[–] [email protected] 3 points 1 year ago (1 children)

There are a lot of different flavors of being on call.

I'm on call all the time. That just means I might get a phone call at any time and have to help one of our techs on a site. I don't have to be near the office for that - it's the knowledge in my head they want.

I charge regular time for it.

[–] [email protected] 2 points 1 year ago (1 children)

Even for an outage starting at 2am lasting 6 hours?

[–] [email protected] 0 points 1 year ago

I'm not sure I understand. I'm working, so I'm charging for it.

[–] [email protected] 3 points 1 year ago

We have a team of 6 and rotate on call regularly. I'm in the US and receive no benefit for on call specifically, but other regions do. My salary more than covers the inconvenience though.

[–] [email protected] 2 points 1 year ago

I don't do it and we have no expectation of it. A good portion of our infrastructure is self healing and spread across multiple zones which has been enough for us for the past 10 years. The parts that aren't can wait until business hours and clients are aware no work is done outside of them so any fixes or changes wait until the following business day.

You would have to more than double my salary to get me onboard with structuring my personal life around "but what if there's an outage", and even then I probably wouldn't do it lol

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

I've been on-call in 3 of my past jobs in Germany, most of it was pretty similar, 1 week per person, 600-800€ per week and some extra pay on incidents. Current job:

  1. ~500€ for a week and I get an extra day off after each week on-call, but no extra pay for incidents
  2. With the extra day off it's really nice. Our team is light on incidents too.
  3. Not much past that. The standard German worker protection laws would apply, but pushing those would not help me much and it might not be the best career move.
  4. 5 others
  5. EU
[–] [email protected] 1 points 1 year ago

I'm on a team of 5 and we don't have an on call rotation since developers are not prod ops. But in a sense we are all on call all the time. The NOC has our phone numbers and if we are needed for something urgent we will get a call or a text for things like helping prod ops troubleshoot an issue if they get stuck. My boss has texted me while I was on vacation before. Usually it's a quick question for something obscure. Once it was an escalation from a senior executive. I don't have to respond if I'm on vacation, but if I'm getting a call they really need help with something. It also is a good opportunity to lay a guilt trip on your boss that results in a few reward points. Never had to actually log into anything though.

We also have BCP, business continuity plan, events. I work for a company that provides a lot of critical infrastructure. If the BCP event is really nasty, like a natural disaster, and our team needs 24/7 representation on the bridge, we take turns and will relieve each other. You won't be expected to help out on a BCP event while on vacation.

Besides BCP we usually have to be available for certain production changes. Like a few months ago I had a DNS and load balancer change done. I wasn't doing the work, but the team making the change wanted me available between 3 and 5 am to validate the change.

If I were paid hourly things would be more formal. I would get overtime(1.5 x hourly rate) + comp time. Since I'm salaried I just sleep in the next day. Our schedules are really flexible. We basically need to be mostly available for meetings for around 4 hours a weekday from late morning to late afternoon, and complete our projects on time. It was like this in the before times. Back then I would go into the office around 11 am for our daily standup. Get lunch with some team mates. Do some afternoon meetings then go home, and do my more focused work at home after dinner time. Most of my team mates did something similar.

Rest of the compensation is your typical American senior software engineer salary with a 10% to 20% bonus, 7 weeks pto, health insurance, life insurance, short term and long term disability insurance, 401k with 6% match, pension, retirement health insurance, pet health insurance, can use the corporate travel agent for personal travel. I actually like this perk a lot. You still pay for personal travel but it means a lot of discounts and upgrades. We also get to keep our various travel points.

[–] [email protected] 1 points 1 year ago

Like many others here, at the company I work for you get nothing.

I do one on-call shift as primary per week and one as secondary. I then also cover a week every six weeks or so.

If shit really hits the fan, them work is pretty cool about taking some time back, but we're far from micromanaged as it is, so we can just kind of make it work.

I'd say an incidency probably occurs on around half of my primary shifts (and I've yet to ever do anything as secondary), and nearly always it was something I could resolve within one hour.

Every dev at the company is on the rota once they've got a few month's experience.

Based in the UK.

[–] MajorHavoc -1 points 1 year ago

My developers are only "on call" in the sense that their code supports responders who save lives.

In that context, I maintain the expectation that I might call any of my developers at any time.

That said, I have made that call once ever in the years since I built this team, and it happened to be during business hours anyway.

Officially, I require my team members to take off time to make up for any overtime worked. (Only officially, I guess, since it has not been tested.)

Our flexible work schedule already sets an expectation that someone too tired to work should let their manager know and sleep in and come in later. That has been tested, for various personal reasons. So that would still apply for a late night or early morning incident.

I'm not able to pay an overtime rate, at this time.

I credit my prioritization of graceful rollback for some of the lack of actual on-call calls. But lots of credit is also due to our fantasticly skilled first line responders.