My take on how a decade (or more) of using cloud services for everything has seemingly deskilled the workforce.
Just recently I found myself interviewing senior security engineers just to realize that in many cases they had absolutely no idea about how the stuff they supposedly worked with, actually worked.
This all made me wonder, is it possible that over-reliance on cloud services for everything has massively deskilled the engineering workforce? And if it is so, who is going to be the European clouds, so necessary for EU’s digital sovereignty?
I did not copy-paste the post in here because of the different writing style, but I get no benefit whatsoever from website visits.
I think its actually that most people generally don’t really understand most things beyond the minimal level necessary to get by. Now that the tech industry isn’t just a bunch of nerds you’re increasingly more likely to encounter people who are temperamentally disinclined to seek understanding of those details.
That and also - humans not knowing something can man up and learn it. When they need, they’ll learn.
And OP’s question about European clouds - it depends really. A lot of what this endeavor needs is just advanced use of OpenStack. I’m confident there are plenty of people with such skills in the EU countries.
As for the post content - I dunno, my experience with Kubernetes consists of using it, but not trying to understand or touch it too closely, because it stinks. Maybe those engineers were like that too.
When they need, they’ll learn.
100% agree. But. If you are a principal engineer claiming to have experience hardening the thing, you would expect that learning to have already happened. Also, I would be absolutely fine with “I never had a chance to dig into this specifically, I just know it at a high level” answer. Why coming up with bs?
Maybe those engineers were like that too.
I mean, we are talking about people whose whole career was around Kubernetes, so I don’t think so?
Ah. OK. Yep, people lie in their CV’s.
That is technically correct in a way, but I’ll argue very wrong in a meaningful way.
Cloud services are meant to let you focus less on the plumbing, so naturally many skills in that will not be developed, and skills adjacent to it will be less developed.
Buttttt you must assume effort remains constant!
So you get to focus more on other things now. E.g. functional programming, product thinking, rapid prototyping, API stuff, breadth of languages, etc. I bet the seniors you are missing X and Y in have bigger Zs and also some Qs that you may not be used to consider, or have the experience to spot and evaluate.
Mind you that my take and experience is specifically in the context of security.
I struggle to make the parallel that you suggest (which might work for some areas) with a security engineer.
Say, a person learned to brainlessly parrot that pods need to have setting x or z. If they don’t understand them, they can’t offer meaningful insight in cases where that’s not possibile (which might be specific), they can’t provide a solid risk analysis etc.
What is the counterpart to this gap? Because I struggle to see it. Breadth of areas where this superficial knowledge is available is useless, IMHO.
Because a security engineer focused on cloud would rightfully say “pod security is not my issue, I’m focused on protecting the rest of our world from each pod itself.”. With AWS as example: If they then analyze the IAM role structures and to deep into where the pod runs (e.g. shared ec2 vs eks) etc. then it would just be a matter of different focus.
Cloud security is focused on the infrastructure - looks like you’re looking for a security engineer focused on the dev side.
If they bring neither to the table then I’m with you - but I don’t see how “the cloud” is at fault here… especially for security the world as full of “following the script” people long before cloud was a thing.
I mean, the person in question had “hardening EKS” on their CV. EKS still means that the whole data plane is your responsibility. How can you harden a cluster without understanding the foundation of container security (isolation primitives, capabilities, etc.)? Workload security is very much part of the job.
I mean the moment some pod will need to run with some privilege (say, a log forwarder which gets host logs), and you need to “harden” the cluster, what do you do if you don’t understand the concept of capabilities? I will tell you what, because I asked this very question, and the answer was “copy the logs elsewhere”, which is the “make it work with the hammer solution” that again shows the damage of not understanding.
I am with you about different scopes, skillsets etc. But here we were interviewing people with a completely matching skillset on paper.
Oh yeah I see…
As some old philosopher once said: “shit’s fucked, yo”.
Seems to be appropriate here.
Yeah I can see that.
However, you are now arguing a different point than I am getting from your original post. Maybe my fault in interpretation ofc, but the main difference (in my view) is:
You say “incompetent” and “less skilled” as general statements on senior engineers. Those statements are false.
You also say “missing the skills you are looking for” which is obviously true.
And the implication that before cloud, people developed the specific skills you need more naturally - because they had to. This makes sense and I believe it.
You say “incompetent” and “less skilled” as general statements on senior engineers. Those statements are false.
I am saying that the competencies of people who grew up (professionally) with outsourced services are more superficial and give them way less understanding (and agency) on the systems they oversee. I make the opinionated argument that knowing which service to use in a cloud provider is not just a different skill from implementing that functionality “manually”, but is hierarchical inferior, easier to acquire and less useful in general.
A weird parallel would be someone who hikes 100% of the time with a guide who takes care of orientation, camp setting etc., and someone who goes alone. If I am simply comparing the pictures they are showing me, I might not appreciate the difference, but if you asked me who I would trust to come hiking with me, I wouldn’t have doubt, because I consider the skill “finding, choosing and listening to the guide” to be hierarchial inferior to “orient, set camp etc. by yourself”.
So it’s not just a matter of matching the skills I need, is actually a much broader argument about deskilling engineers.
I understand.
Obviously, “knowing which cloud services to enable” is a lesser skill than knowing how those services work. That is not a parallel or equal skill in any way.
But do you assume people are just going drrrrr brain off when they don’t learn that one skillset you are accustomed to spotting?
Well, for the relatively small sample of Kubernetes experts I interviewed, basically any topic beyond “you use this tool” was a disaster, including Kubernetes knowledge. I am not selective, it’s not like I expect a specific skillset, but what would you think if someone with a decade of platform security doesn’t understand cryptography and supply chain, Linux permissions, Kubernetes foundational concepts, container isolation or networking? At some point the question is legitimate, what are you expert in? The answer I have been able to give myself so far is “stitching together services that do stuff” and “recommend what the documentation/standard recommends”. I consider myself satisfied to have somewhat decent knowledge in some of those areas, I am not expecting someone understanding all of that, but none of them? Maybe from someone who just joined the industry.
That being said, I am genuinely frustrated by how little people know or care about the plumbing these days. :D
I am so fucking tired of seeing someone spin up 3 cloud databases for what could be a 40k in-memory hashtable.
I disagree. On paper that sounds good, but I firmly believe good engineers are curious, so they’ll learn a lot more than necessary to do the job.
For example, when I worked at a company that designed antennas as a software engineer (built something tangentially related), I didn’t need to know anything about electrical engineering, but I was curious so I asked a ton of questions and now I know a fair amount about EE. These days I work in a very different domain and still ask a ton of questions to our domain experts. In my own field, I look into all kinds of random things tangentially related to the tools I use. In each case, that curiosity has come in handy at some point or another.
In each role, I can tell who’s there to clock in and clock out vs who is genuinely curious and looking to improve, and it’s the latter group who tend to produce the best work and go on to great roles after leaving our company, while the 9-5 warriors who just focus on the requirements tend to do pretty mediocre when it comes to advancement.
When I hire, I look for that curiosity because you never know what you’ll need to know to fix a prod issue quickly. My esoteric knowledge about SSH helped keep my team productive for a few days when IT was being slow revolving our issue, and likewise we’ve had quick resolution to prod bugs because someone on the team knew something random that ended up being relevant. That’s what I mean when I say I look for a diverse team, I want people with different strengths who all actively seek to improve so we’ll have a good shot at handling whatever comes down the pipe (and we get a lot of random stuff, from urgently needing to embed 3D modeling tools into our reporting app to needing to embed complex C++ simulation code or rewrite Fortran code into our largely CRUD Python app).
Most of these cases of “focus on one niche” are often symptoms of lacking curiosity and just wanting to tick boxes to quality for a role. I’d much rather someone miss a few important boxes but tick a lot of random ones because they’re curious; they’ll take longer to on-board, but they’ll likely be more useful long term.
I don’t work in the security space, but I think the same applies to most technical fields. Breadth of knowledge in an individual provides depth of knowledge in a team.
Yeah I don’t think we actually disagree much here. :)
I think my angle is just slightly different? I see that ease of access (eg cloud) make it possible for a lot more uncurious and clock-out people to enter the field and pass as competent. To be honest, even the modest introduction of auto-formatting editors are easy to see as good and useful, but I also feel that they allowed shoddy work to look passable at first glance. AI will make this a lot worse.
But as for the actual people who have it in them to be competent, people that were always there and still are, cloud is not going to make them worse.
I guess my point is that it’s harder to suss out the actually competent people if they’re able to build a good portfolio using tools. AI makes this harder, since they can sound more competent than they are, and them a few months down the line we need to discuss them leaving the org.
The main factor, IMO, is that everyone wants good engineers but good engineers don’t change jobs that often.
Meaning most of the candidates you interview will suck in one way or another.
And everyone calls themselves “senior” nowadays.
Everyone calls themselves senior because that’s the only type of position recruiters look for.
I’m a mid level dev, but I’m encouraged by recruiters to apply for senior positions because their clients are actually looking for a range of levels
Yeah, that’s true, everyone thinks they want a senior where usually someone who’s not a straight up junior is more than enough. And a fast learning and motivated junior is the best you can get, IMO, though those are pretty rare as well.
Exactly. We don’t hire “junior” positions, because all the midlevels are juniors, all seniors are mid-level, and seniors don’t apply. I’m a senior and a recruiter found me, I didn’t apply (at least not to this company).
That has been my experience with security people, too. They are button pushers and copy pasters. But I don’t think it’s cloud computing causing it. They were like that before clouds.
Yeah, they are frequently just parroting things like CVE notices as highlighted by a fairly stupid scanning tool.
The security ecosystem has been long diluted because no one wants to doubt a “security” person and be wrong, and over time that has made a pretty soft context for people to get credibility as a security person.
I’m a very good engineer, but so much of my time is consumed fighting with Tekton pipelines and migrating testing frameworks and versions I barely have time to write code. But that’s because I can figure that stuff out when I have to. All the code is written by the people who can’t figure that stuff out.
Why this isn’t two separate jobs I can’t understand. Let me do some stuff I’m good at rather than constantly fighting with things I’m not?
This hits the nail right on the head. The point of cloud services is to take away all the overheads of building and delivering software solutions that have nothing to do with the actual business problem I’m trying to solve.
If I want to get a new product to market, I want to spend most of my time making my core product better, more marketable, more efficient. I don’t want to divert time and resources to just keep the lights on, like having to hire a whole bunch of people whose only jobs is to provision and manage servers and IT infrastructure (or nurse a Kubernetes cluster for that matter). Managing Kubernetes or physical tin servers is not what my business is about. All this tech infrastructure is a means to an end, not the end itself.
That’s why cloud services is such a cost efficient proposition for 98% businesses. Hell, if I could run everything using a serverless model (not always possible or cost effective) I’d do it gladly.
This is quite a trite argument from my point of view. Also, this is from the perspective of the business, which I don’t particularly care about, and I tend to look from the perspective of the worker.
Additionally, the cloud allows to scale quickly, but the fact that it allows to delegate everything is a myth. It’s so much a myth that you see companies running fully on cloud with an army on people in platform teams and additionally you get finops teams, entire teams whose job is optimizing the spend of cloud. Sure, when you start out it’s 100% reasonable to use cloud services, but in the medium-long term, it’s an incredibly poor investment, because you still need people to administer the cloud plus, you need to pay a huge premium for the services you buy, which your workforce now can’t manage or build anymore. This means you still pay people to do work which is not your core business, but now they babysit cloud services instead of the actual infra, and you are paying twice.
Cloud exploded during the times of easy money at no interest, where startups had to build some stuff, IPO and then explode without ever turning a single dollar of profit. It’s a model that fits perfect in that context.
I get you that it’s easy to over-provision in the cloud, but you can’t return an on-prem server. A cloud VM, just shut it down and you’re done.
AWS talks about minimizing undifferentiated heavy lifting as a reason to adopt managed services and I find that largely to be true. The majority of companies aren’t differentiating their services via some low-level technology advantage that allows them to cost less. It’s a different purchasing model, a smoother workflow, or a unique insight into data. The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.
A cloud VM, just shut it down and you’re done.
If this flexibility is needed, and it’s an “if”, a dedicated server does the same. But even a cloudVM is already lower level compared to other services (which are even more abstract) - like EKS, SQS, etc.
The value an organization provides to customers should be the primary focus of the business, the rest is a means to sharpen that focus.
In my experience this often translates in values that flows to AWS, while the company giving value to customers is stuck with millions of cloud bills each month, and a large engineering footprint that eventually needs to cut, leaving fewer and fewer people working on the product.
That said, I acknowledge that cloud has business reasons to exist, I wrote an entire other post about my hate for it, but I still acknowledge that. However there are some myths that finally are getting dispelled (outsource infra and focus on your product).
I’d like to understand how self managing all the lower level components abstracted by the cloud is saving on headcount. Care to math that out for us?
It depends. An EKS cluster can cost easily 20x what an equivalent cluster costs with same resources. The amount of people necessary to manage it is very close compared to a bare cluster, which depending on the scale can save hundreds of thousands or millions per year, therefore allowing extra headcount.
For example, a company I worked for had a team of 6 managing all their kubernetes cluster on rented dediservers. The infra costed around 50k/year. The same clusters on EKS could be managed by 4 people (maybe?), but would have costed easily 5-600k, especially since they were beefy machines, possibly even more. That amount of money would pay for 7-8 additional headcount in local hires.
Considering that in those clusters there were 40-50 postgres clusters, if moving those to RDS they would have probably looked at millions in cloud bills per year, and the effort to run those dB’s once the manifests were developed was negligible (same team was managing them). This was a tiny startup, with limited resources for internal tools and automation development.
So it’s not like managing everything can save headcount, it’s that not outsourcing everything can save so much money that largely compensates for more headcount, plus you are giving money to real people, who spend local and pay taxes.
I somewhat disagree here, but also somewhat agree.
In my org, we get a lot of requirements that require very different skillsets. For the first 2-3 years, our task list was mostly CRUD stuff with some domain specific logic, but otherwise a boring web app. In the last 1-2 years, we have:
- ported a Fortran simulation to Python
- embedded a C++ simulation in Python
- created a 3D UX for our previously 2D only app (lots of 3D logic on both FE and BE)
- implemented a machine learning algorithm to train our simulations
If I hired only for the work I’d seen in the past, we’d be completely unfit to handle this workload since we’d mostly have people who are really good at building CRUD apps (so DB optimization and quick UX building).
On the flipside, we cut off huge swaths of work so people don’t need to wear too many hats. We have:
- dedicated devOPs - handles everything from trst pipelines to prod deployments
- dedicated QA - manual and automated app-level testing - devs still do unit testing
- dedicated product teams who handle feature requirements and documentation
- dedicated UX team to produce designs for FE engineers to implement
So our devs only need to worry about development, but they also need a broad skillset in that domain, from everything from local tooling to working in different domains. We hire a diverse set of candidates, some with a heavy math background, some with design experience, and some with low level programming experience, because we never know what projects we’ll get or who will suddenly leave the org.
If I understand the gist, I’ll just say I’d like my job to be some stuff I’m good and some stuff that challenges me. When I do nothing but challenge myself, imposter syndrome sets in. When I do nothing but the stuff that I’m good at, it gets really boring. I need to find a better mix than I have been.
Nah brah, knah waddahma? Running my own Nextcloud instance is basically what drove me to become a linux novice.
I used to be a windows gamer. Now I run my own home-LLM server for the self hosted cloud assistant.
People should try, it’s fun!
Juat as a reality check:
What you and me consider fun isnt fun for most outside of the lemmy techie bubble.Haber most people reconsidered what they consider fun though?
They’re missing out!
I’m not in any way, shape, or form an engineer so I don’t really understand the exact details of your post.
However, you post reminded me of a really good episode of a podcast called Hidden Brain. In it the host, discusses the topic of knowledge with a cognitive scientist.
At one point, they talk about how sophisticated technology has gotten that people don’t know how to solve problems if that technology brakes, especially since technology is getting so good that it makes fewer mistakes. They use an airplane as an example in which an experienced pilot forgot how to get out of a nosedive and crashed the plane. On a smaller scale, the host mentioned that he has a hard time navigating if his phone’s GPS doesn’t work.
Its a really interesting listen if you have the chance.
Thanks, indeed I think there are many parallels with other areas. I will check it out.
Or maybe it’s just a different skill set
Not when the skillset is essentially outsourced and you are left consuming the product of that skillset.
Understanding is nonnegotiable in security, IMHO.
You can’t fail to understand how signature attestation works, if you are implementing it, to make one example I made in the post. Otherwise you end up verifying the signature in the CI (like that person claimed it should be done) and waste the whole effort. You can definitely still outsource the whole infra and scripting to Github, but you still need to understand. The problem is that when you can outsource everything, at some point understanding becomes an extra step.
interviewing senior security engineers
Or maybe senior security engineers from 10 years ago were somewhat different from (wannabee senior) security engineers today?
Did you ask them to write 0xD6 in decimal? 😃
That’s the thing! I think it wouldn’t be conceivable that your “principal engineer” (real position for one of the people) doesn’t understand the basic theory of the stuff they are implementing. Now it feels you can instead work years and years just shuffling configuration and pressing buttons, leading to “senior” people who didn’t gather actual years of experience.
I don’t want to pretend I am outside this logic. I am very much part of this problem myself, having started my career 10 years ago. I do despise cloud services though (if anything, they are super boring), so I tend to work with other stuff. But I could 100% just click buttons and parrot standard and keep accruing empty years of experience…
I agree with your lack of affection for cloud services, but I think your view might be a little skewed here. Does a senior mechanic need to understand the physics of piston design to be a great mechanic, or just gather years of experience fixing problems with the whole system that makes up the car?
I’m a Senior Systems engineer. I know very little about kernel programming or OS design, but i know how the packages and applications work together and where problems might arise in how they interact. Software Engineers might not know how or don’t want to spend time to set up the infrastructure to host their applications, so they rely on me to do it for them, or outsource my job to someone else’s computer.
Does a senior mechanic need to understand the physics of piston design to be a great mechanic
I would argue that if senior mechanic doesn’t understand the physics of piston design at least on some degree he’s not a great mechanic. Obviously mechanic doesn’t need understanding on metallurgy, CAD models and a ton of other deeper level stuff just like an IT engineer doesn’t need to know on a deep level how circuit boards are designed or how CPU die manufacturing process works. But both benefit greatly when they understand why something is built the way it is.
I’m also an systems engineer of sorts and have worked with software engineers. And I’ve had requests like “Can’t you just set 'bind-address = 0.0.0.0 on mysql-server and disable firewall” on a directly internet-facing machine and then received complaints when I’m “making things more difficult” from “senior software” -titles. Sure, I can’t write the code they’re doing, or at least it would take me a crapload of more time to do that but on the other hand there’s guys who have so very narrow understanding on anything they work with that it makes me wonder how they can do their work at all in the first place.
Of course no one can master everything in any field but I find it concerning that a lot of guys just press the buttons more or less randomly until their thing works without any clue on what they actually did and how it might affect on different parts of the house of cards they’re building.
I 100% agree.
The best mechanics can track down an issue by reasoning about what could be causing it, and understanding how pistons work can help deduce whether that knocking is actually the engine or something else entirely. They probably didn’t learn that from their official training, but instead worked with some guy who used to work at a car manufacturer or something and picked their brain.
The best engineers are curious and jump on opportunities to learn more.
The best mechanics can track down an issue by reasoning about what could be causing it
Same principle works with IT. I do and have done sysadmin stuff for quite a while and there’s always some random software or whatever I’ve never heard of and someone comes and asks me to fix it. Then you start to ask questions, “what exactly doesn’t work”, “can you show me what you’re doing”, “what should happen when you press that button”, “can you show settings on that thing” and so on. Then you can start to dig down, does the server they’re using respond to ping, does DNS resolve (it’s always DNS after all), does that thing work on the next workstation, when did the problem appear and was there some other maintenance or changes going on at that time and so on.
Same principle, just start to reason the whole thing from bottom up, check everything you come across untill you find something which doesn’t work and then do what’s needed to fix that, rinse and repeat until the problem goes away and make sure that what you’re doing won’t cause new problems. Just the tools are different, the mindset is more or less the same.
Exactly! If you know enough foundational principles, you can quickly rule things out and develop ways to narrow down what remains. If you rely too much on diagnostic tools, you’ll miss out when the tools fail to catch something odd.
I’m a software engineer and we had a problem where our corporate laptop wouldn’t allow us to install our dev tools (needed to debug a windows specific integration and we dev on macos). Instead of waiting a week for IT to come fix it, I realized we just needed it to look like a service was running locally, and we had ssh through the git bash shell, so I set up an SSH tunnel between the windows system and the dev machine and they were able to keep working while waiting for IT to get time to help us. We rarely use SSH at work, but I understand enough about how networks and sockets work so I was able to quickly help them solve the problem.
You don’t get that type of intuition if you don’t understand how the underlying tech works, and that’s true regardless of your field.
But you know what the kernel is. You know that syscalls are a thing, you know what role the kernel performs, you know that different filesystems have different properties (and pros and cons), etc…
You don’t need to know the details, perhaps, but you can’t ignore the fundamental theoretical concepts of kernel and OS. You might not know the whole detail of the boot procedure, but if your machines are stuck on boot, you know at least what to look for.
Here I was talking about equally foundational topics. There is nothing “above” - say - producing attestations and then verifying them. That’s literally all there is to it, but if you don’t understand the theory behind it, what exactly are you doing? As as I said, I don’t care about the details, I didn’t expect someone mentioning ciphers or timestamp authorities, transparency logs etc. All I would expect is “we produce a signature with a bunch of metadata and we verify it where we consume the artifact, so that we are sure that the artifact has the properties attested by the signature”.
Not knowing this is like someone claiming that they administer Linux machines but can’t explain what network interfaces are or how routing is determined. This is not a question of being expert on different layers, this is just being oblivious to those other layers completely.
You want to hire the “guru”, not the “principal”. You want to actually ask him to write 0xD6 in decimal, and if he dares to answer “Seriously? Come on now, that’s boring”, then you hire him on the spot.
But you can’t hire only gurus. You need normal seniors, too. Build a normal team around one guru. Maybe build one ultra advanced team around 2-3 gurus, if you really need to invent new and hardcore difficult stuff.
Instead of hiring gurus, I think you want a diverse set of curious “regular” people. Maybe one person is really good with working in different number bases (and 0xD6 in decimal is something they know off the top of their head), another is really good w/ databases, etc. None of those would know everything, but they’re all curious and picked up random stuff from their career because they asked a lot of questions.
Hiring the right guru is hard, having the equivalent of a guru across a diverse time is a lot more tractible, and maybe one will become that guru you need after cross pollinating with the team.
deleted by creator
