Episode 16, One Admin, One Server
04/03/2007 10:38 AM Filed in: Podcast
Here are
the show notes for episode 16.
Make sure to send us feedback so we can make the show even better.
Make sure to send us feedback so we can make the show even better.
Seg: "One Admin, One Server" by Seg.
I've been a sysadmin for a number of years now. I've seen a few different data centers. I've seen a lot of customers, and I've seen a heck of a lot of servers.
In my experience, I've noticed a growing trend - something that I deem a problem - which is a small group of people, sometimes just one person, managing a large group of servers. Typically, with some of the bigger clients I've seen, it can be as simple as five people for ten serves, five people for 50 servers, five people for their entire environment of 100 servers.
Sometimes, it's not so bad. Sometimes it's one person for four servers. One server or maybe five servers, four web heads and SQL database server.
But it's not always set up the same, and what I want to convey to you the importance of one admin and one server. On my own, I run ONE server, exactly one server. And I have my home systems, but I don't consider those servers. I have one web head, SQL database out on the Internet. That's one actual server.
This one server does a lot of things. It actually hosts four different websites, it hosts mail for a couple of them, and it hosts a SQL database which serves a couple of them. Under our best case scenario, I would have separate servers for mail, separate servers for web, separate servers for SQL databases, and then additional servers for anything else I would have run, such as monitoring or any other custom applications that needed to be on other boxes.
Now, I'm one admin for this server, this one server. But this one server performs a lot of functions. If I were to split this one server up into several servers -- one server for each function -- then the idea I'm trying to convey still holds true. Because it's one solution, it's one environment, it's one part of an environment.
If I was one admin administrating four servers, and each server had its own function, and it was for only one website, that would be the best possible scenario. As it is right now, honestly, I'm spread too thin. I have two or three small websites that don't get a lot of traffic, the SQL databases don't get a lot of hits, and the application that I run are pretty lightweight. I'm not in charge of anything massive for my personal server.
However, I am stretched too thin on this. To be a good systems administrator, I need to be able to focus all of my attention to one instance, one solution, or one project. And that one project can include a web head, a SQL database, an application server, a monitoring server, and a mail server. But they're all geared toward the one project, maybe it's the one website, in this case.
If I was only hosting a mail server, then that one mail server would be the project. If I was just responsible for a SQL database, that that would be the project. But as it is, hosting four all on one server, it's spread too thin. I can't put enough attention to all of them at the same time.
Now, having them all on the same server has its ups and downs. It's a single point of failure, that's a down. But an up is it's a little bit easier to maintain, in that having all of the programs run on the same server makes it easy to cross-reference log files.
I don't have to log into a lot of different boxes to do different things. It's one box, and I'm checking out logs on Apache and I need to cross-reference information in MySQL then I just do it. I'm on the box that can do it. But as I mentioned earlier, the best case scenario, it's best to spread them out. But it's still one project per website in this case.
The customers where I see it's one or five or ten people hosting five or ten or 1,000 servers, it's ridiculous. I don't know how these people make money doing this. I really don't. I see multi-billion dollar industries every day, selling their products and serving their customers with just the lowest possible head count. And it's horrible. It's horrible.
Now, I know that there are not a lot of sysadmins in this world. We are a dying breed, it seems. And we are very scarce, hard to find, and if you want one with a security clearance, well, hey, it's almost impossible. In fact, if you're a sysadmin with security clearance in the Virginia-D.C.-Maryland area, you can name your price. That's how scarce they are.
But even sysadmins without security clearance are very hard to find. And good ones, double so. In fact, I think it's harder to find a good sysadmin than it is to find any sysadmin with security clearance. But I'm getting off track here.
The problem that I'm seeing when we have... Let's give one example I know of, off the top of my head. Let's say we have five admins working on 20 servers. And these 20 servers make up different functions. Some of them are mail servers, some of them are web heads, some of them are SQL database servers, some of them are NSF servers, some of them are for custom applications, some of them are for monitoring. But they're all within the same one project.
I know you'll tell me, "Seg, I thought you said one admin for one project." No, no, no. I said one admin, one server. It's good when you can have one person devoted to one project, or a small group of people devoted to one project.
But if you've got 50 servers, you've crossed a line when it comes to the one admin, one server rule. You get to the point, with 50 servers, that you really need more than one DBA, you really need more than one application manager; you really need more than one to deal with the web heads.
And the example that I'm thinking of, where it's five people for 50 servers, it's terrible. We have five people who are butting heads all the time. Only one of them is a DBA, and the rest of them sort of work on the applications. One of them is a manager, so he really doesn't do anything. And they just barely get by.
But I'll tell you what: I can hack their website in a second. I don't know anything about hacking or cracking or phishing or breaking into anything. I just know that their systems are sub-par. I know this because they don't manage them. All they do is they worry about the application, or they worry that the database is up.
They don't care that Apache hasn't been patched in two years. They're running the ancient stuff. They don't care that their OS hasn't been patched in two years. They don't care that what they're doing is sub-standard, it's flawed, and it's open to exploitation. They just want to make their money.
They're not concerned about the systems administration. So, in that sense, they don't have any system admins. They just have web developers, application developers, and a DBA and a manager. And that's horrible. You have zero sysadmins for 50 servers. And this is rampant. This is rampant all across the Internet. This happens all the time.
I can't give you any names. I can't give you any company names. I'm never going to do anything like that. But I'm telling you, they're out there. If you ask anybody who runs a botnet, if you ask anybody who's trying to crack Apache or make a SQL injection and been successful, they know. The people who are out there doing it every day, they know that this is going on, that people are not taking care of their OS. That people are not taking care of the programs that they use to server their information. And it's horrible.
One example that I talked about a second ago, which was one person for, I guess in this company it's four web heads and one SQL database, a Microsoft SQL database. Now with this company, I don't know, they've had a hard time keeping people on staff. But whenever they do have someone on staff, it's just one person. And most of the time it's just been a web developer.
Last time I checked, they had one sysadmin and they contracted out for the web developer, who also doubled as a DBA. And this situation was just horrible. This guy had a really hard time just with keeping up with the Microsoft updates, keeping ISS in line, making sure that - even though they were fully patched - that they were on top of the latest security trends, that they were blocking suspicious activity and things like that. So much so, that at one point he just stopped doing it.
When he started with that company, he came to us. He came to my company, who were hosting his application. He came to me and he said, "I'm new. I don't really know how their stuff is set up. What I want to do is just get everything up to date. I just want to get the OS up to date. I want to get all the applications up to date. I want to get the SQL database up to date."
I said, "That's fine. Let's work with you, let's make this happen. Let's be good systems administrators."
At the end of it, it was too much for him. And he said, "You know what? F' it. It's as up to date as it's going to be. I'm not going to worry about it anymore. I'm just going to go write the website that they want." And left it. And I hope that they haven't been exploited. I don't think that they have, but it's just a problem waiting to happen.
One of the data centers that I worked for recently was just atrocious. It was just this horrible, horrible data center. It's like a fake data center. It wasn't a real data center. They had walls and they had servers and they had pretty lights and a lot of cables and they built themselves a data center, but they weren't really serious about it. They just didn't know what they were doing. The problem was that they were F'ing huge; they're worldwide.
But everyday with this company, there was a ticket of a POP customer, of a customer who got hacked. A POP customer who got cracked. And every single one of their customers that was in this predicament didn't know what they were doing.
Most times, it was one person with a handful of servers. And, again, we have the web builder/manager problem, where they read enough Google articles to get it up and running and then said, "Screw it. It was two years ago I got it up and running. If something breaks, I'll deal with it. But in the meantime, I'll just work on my website."
Every day, new exploits are found. Every single day, new exploits. Every hour, new exploits are found. Every minute, new viruses and worms are created. And even the most up-to-date program is not un-hackable. Nothing is. Nothing is without its flaws. And it's important to understand that.
And when you really understand how vulnerable everything is on the Internet, how flawed everything is, you realize that if you just have one person managing five servers, or five people managing 20 or 50 servers, then it's mind-boggling. The workload is enough to choke a horse ten times over. It is just massive.
Because you have to worry about these types of things all the time. That's the life of a real sysadmin, is worrying about what's going on and fixing the problems. And always investigating the new things, looking at the patches. Does it really patch it the way it needs to be patched? Is it going to cause a problem shortly down the road? What are the new bugs? What are the new worms? What are the new exploits? Always, always, always keeping on top of what's going on. That's what a good sysadmin does.
So, because the workload is so huge, for a systems administrator, you need to minimize the number of systems that a sysadmin work on. When you minimize the number of systems that a sysadmin works on, you increase their efficiency. It's pretty simple logic, I think.
Not everyone agrees with me, and of course there are plenty of managers and suits and salespeople out there who are going to tell me, "Well, we really just can't afford to have a hundred sysadmins for our hundred servers." And I can understand where they're coming from. I don't agree with it, but I can understand where they're coming from. They're looking at a completely different system than a system that is serving their information.
The system that they're looking at is financial. It's money: dollars and cents. I understand that system, but it's not what will get you productive in the long run. It will not be a solution of any kind.
Now, I've said that it's really hard to find good sysadmins. I've said that it's really hard to find any sysadmins, even bad ones. So I know that I'm making this plea on tied hands. I know that I'm saying, "Well, this is the way that it needs to be,' even though that's the way it can't be. Because there just aren't enough resources to do it. Even with all the money in the world, you can't just force people to be sysadmins. And you can't force them to be good sysadmins.
People are going to do whatever they're going to do. Some people want to be fry cooks, and some people want to be taxi drivers. Some people want to build skyscrapers. And some people are dumb enough to want to by sysadmins, myself included.
Those scarce few people on the planet who've chosen the role of systems administrator need to be valued to a high extent. Because, in the very specific sense of a systems administrator, their job is to make sure that the project works.
If you're running a website, something small, you need to make sure it works. If you've got a web developer who doesn't know anything about systems administration, and he's the only person in charge of the server, it's not going to work that long.
Now, in the case of one of my favorite data centers that I worked for, we had one web developers working on a project that was spread across five different servers, six different servers. And their solution worked. But, it worked because the data center that hosted them also supported them in a systems administrator sense.
And they knew this. They paid for this. They paid through the nose for it. They said, "Hey, I've got one web developer, I'm the president and CEO of the company. It's just us, we're running this website, and it needs to work. Please take care of our server." And so we patched it. We maintained it. And, when need be, we investigated it.
But, you know, that's a group of sysadmins working on one project. And this group of sysadmins is responsible for tons and tons of other projects for other customers. And so even though this one customer was paying through the nose for their support, when it came down to it, it still wasn't going to be enough. Because the sysadmins that were supporting that couldn't spend everyday on that server. They had to spend time on other servers.
When you host through a hosting provider, a data center, and you pay for management, you're not paying for full management. You're not paying for sysadmin. You're paying for an on-call sysadmin. You know, someone that when it's on fire, they'll come and put it out. But they're not going to be there on the day-to-day to make sure you don't start fires. They show up after the emergency has already happened.
And if you need patching, if you need upgrading, that's fine. You let them know and they come in and they do it. But you've got to make that step first. And it's not enough.
Now, I make my living in shared managed and co-located data centers. That's how I make my living. And I know in some sense, that I'm sort of denouncing the thing that makes my living. But these things have to be said.
I work on hundreds, sometimes thousands, of servers within a data center. I can go across 50 different customers in one day, 50 different projects in one day. And each project can be one server, it can be five servers. It can include storage arrays. It can just be a completely complex solution comprising of hundreds of servers. But I've got to spread my load that way.
And I always know I can't give everyone my full attention as much as I want to. And when I'm working on a ticket, it's got my full attention; I'm giving it my best. But there's a point where I cut and run, because I've got to get to the next one. This is done, but it's never going to be fully done. The problem that they asked about is solved, the fire is out, but I'm not going to be there to prevent the next one. I'm going to have to wait until they open up a new ticket and tell me that something else is on fire, and then I'll react to it.
But when you hire sysadmins directly for your company, for your project, it's different. Because you're getting closer to the "one admin, one server" model. If you're running a website and you've got maybe one or two servers out of a data center, maybe you're even paying for a little bit of management, maybe you're co-location and you're not paying for any management at all.
But you hire a sysadmin. You hire one person who's just in charge of the OS, making sure the base applications run, making sure that the programs run, such as the web server programs and the applications. Everything that I mentioned previously. Making sure that they're taking care of that, because that's what systems administrators are about.
Systems administration is not about writing websites. It's not. And it's not about writing application code. It's not. I know that some people think it is, and some data centers will tell you it is. But it's not. Systems administration is about taking care of the operating system. Systems administration is about taking care of the main programs that are used for serving data, be it a SQL program, a mail program, or a web pad.
If you're writing an application in PHP, then making sure PHP is up to date. If you're doing it in Perl, Perl is up to date, Python, Python is up to date whatever. Making sure that the core systems are up to date, that they're solid, that they're functioning normally. That's what systems administration is all about.
If you need somebody to work on your database, you need a DBA, a database administrator. If you need somebody to work on the application, you need a developer. If you need somebody to write the website, you need a web developer. Web developers are not application developers. Application developers are not database administrators.
You might know somebody who can do a couple of different roles, but each role is, in fact, separate. So, when you're looking for a systems administrator, don't hire a web developer, hire a web developer for web development. If you need a systems administrator, hire a systems administrator.
If you can't find one, and chances are you can't, then go with a managed data center. It's not perfect, it really isn't. Managed data centers are not perfect and they're not a total solution. But they're pretty close. They're as close as you're going to get if you're not going to spend the money, and if you're not lucky enough to find a good systems administrator.
In my own life, I try to keep it to one admin, one server. Obviously, I haven't. I've got multiple projects running on one server. And, as I've already said, it's hard keeping up with all that. But I know my goal, and I try to work towards that goal. And I understand that you can't find an admin for every server. You can't find an admin for every project. There just aren't enough heads out there for all the different projects that exist.
But you've got to try. You set the goal. You know it's unattainable, but you've always got to head towards that goal, the goal of one admin and one server.
Thank you.