StormForge Office Hours

Noah: Thank you everyone for coming out. This is the first in our series of StormForge Office Hours. This will be a bi-weekly event where we will be having people come to ask us questions about all the many things that we can provide them help with.

This week, however, we do not have anyone here to ask us questions probably because we did this on very short notice with very little buildup, but we’ll hope to fix that for next week. So, as is tradition, that I am starting right now, the topics of discussion for this week will be picked out of this hat that you can’t see with my background. This hat contains a numerous set of topics. This is not “Whose Line Is It Anyway”, but it’s pretty close.

Chris: Whose hat is it anyways?

Noah: It’s my daughter’s hat.

Chris: Okay, glad she could help.

Noah: So it’s you know, very very small. It would not fit on my gargantuan melon.

Chris: So you’re saying we have a small list of topics?

Noah: We do have a small list of topics, that’s correct. Okay, are you ready for our first topic and we’ll see how many we get through until whether or not somebody joins us. Everybody ready?

Chris: Go for it.

Noah: Okay, our first topic is: When to not use Kubernetes?

Chris: All the time? See Brad and I differ on this one. I’ll bring this one up just because this is gonna be a good topic. I still think databases on Kubernetes are a bad idea.

Brad Ascar: I have to disagree.

Chris: I think that databases on Kubernetes are a bad idea because of the risk factor. It’s not whether you can or can’t do it okay? You can do it. You can shove anything into Kubernetes. You can run Linux applications on Kubernetes that go to thin clients. Absolutely it’s doable, but just because you can doesn’t mean you should and I think that in that instance, I don’t think that databases on Kubernetes are a good idea. Because while you can do it, yes, there’s a lot of things that still aren’t flushed out and the practical means of it for somebody to run a database on Kubernetes properly is a higher skill set than I think a lot of people typically have with Kubernetes.

Because you can’t do it. Brad you’re totally muted.

Brad Ascar: No I’m not muted. No, I was gonna say, really depends on the kind of data fabric that you actually put underneath your node, so there are folks out there that are doing things like Open EBS, folks at Portworx, disclaimer used to be a Portworx guy so I know firsthand how many people are actually able to run really large databases. For database, it’s all about your connectivity to your storage and it’s true whether it’s in a containerized environment or not, storage is key because if you can’t reach your storage you can’t write down from memory down, right? That’s the problem – that’s the whole nature of persistence you have to have disc. But a proper fabric will give you that and same kind of thing that sands give you, right? Is they give you a method of non-locally attached storage as a way to do that the proper data fabric that’s done the right way can do it. You need some very specialized software for doing. It’s not something you’re going to be able to pull off on your own, but there are solutions out there that’ll let you do that.

Chris: Right, but the two key things in your statement there was you need specialized software and not everybody can do it. While I agree that you can run databases on Kubernetes, if I have a team or a customer or a client or somebody who is not in their third year with Kubernetes, running databases on Kubernetes is going to be a significant challenge, and to recommend running databases on Kubernetes to them, if they do it wrong, they could experience significant data loss. The risk is really high and the barrier entry of knowledge is very to get it done right is very high, but the ease of use in which you can just throw a database in a container and put it on Kubernetes and say hey, I’ve got a database and a container on Kubernetes and it’s working is really low. So this is one of those things where we can throw it on in a container, we can throw it on Kubernetes really easily, they can run it and think it works just fine until it doesn’t.

Brad Ascar: Okay and there are people there that want the performance and they want the capabilities and the other advantages that they get from Kubernetes in a containerized environment, right? So it’s a difference in whether you want to drive a Toyota and you turn the key and it works or you’re trying to do a Ferrari because you’re going the highest speed possible to perform it with what you’re doing. So it depends on who you are and what your appetite for what you’re doing is, but I wouldn’t say that that’s probably the best thing for a small customer or somebody that’s not well-versed, but for advanced users I think that that’s no longer the case. If that’s really what you want to do, then you get all the rest of the advantages that that kind of deployment gives you and you can still have your persistence later.

Chris: I mean yeah, if you’re like the Apple’s or the Google’s or if you’re any medium to large enterprise that’s all in on Kubernetes, sure. But if you’re like an education platform that’s just trying to move off ECS to EKS, right, and you’re like hey I’m gonna spin up my MongoDB databases on EKS, you’re probably gonna have a bad time.

Brad’s just over there smiling. I feel like he’s just like…

Brad Ascar: I would say I spent a year and a half settling against that mentality and seeing actual success with people, so I think that a few years ago that was definitely much more the case. I think now that area has gotten addressed in a good way and in a way that’s actually pretty easy to install into those environments and you don’t have to be a complete expert, but you do need to be a moderate to advanced user. You can’t be a new user and understand the implications of that.

Chris: Right. You would have to be moderate to advance because the risk of data loss is high. You have to be able to do migrations and so forth on the containers properly and run the containers you know in a stable manner in which it’s not just the data layer, right? It’s the networking layer too, and I know that Brad Beam brought this up in another conversation that “hey, you know we can reduce latency and get the containers to communicate very well by implementing Calico and BGP tables and stuff, but how many moderately sized teams do you know of that spin up Kubernetes with BGP tables in their own data fabric?” I can think of a very large one…

Brad Beam: …trying to pull me in very very yeah…

Chris: You’re just smiling! We got to get some engagement going…

Brad Beam: So I agree with your statement and I felt like I came up with this at the end of the day too. Just because you can put something in a container, you can run in Kubernetes, doesn’t necessarily mean you should. Part of what built up to that is when we had done our Kubernetes deployment, we really took that to the extreme. We were able to get free IP running the container, having that going, so we had our off running on there. We had a need for a desktop interface for users to connect to for some application or whatever they needed, so we were able to run x2go hitting off a jump box that was running on Kubernetes. There’s some very creative and neat things you can do.That being said, we definitely had a lot of issues trying to run free ipa and in a container. That being said I feel like as more and more technologies start to embrace the cloud native way of doing things, they start decoupling a lot of the assumptions that were made when the technology was built up on a VM, when I had a single server, I had a guarantee that systemd was going to be available, I had a guarantee for you know x, y, z. So as software stacks start embracing a cloud native model for things, it becomes a lot easier to make that transition to running on Kubernetes. From a database perspective, I find it really interesting because with the gains you get from persistent storage on Kubernetes, you can have that same storage follow you wherever you go. So at that point you have some network latency for whatever the storage back end is, but let’s assume that’s reasonable. What becomes interesting with that is if you run a single instance database, on the bright side you get all the the failure recovery things that you got from Kubernetes, right? I get my readiness check, my liveness probe, something happens it gets moved my node dies i realize i have not enough replicas that gets rescheduled and started up. So I have a bit better failure recovery proposition from that on Kubernetes, but with that, I also have a much much different way to handle failures tuning investigation debugging than I would have on a vm. So there’s trade-offs with it. I don’t think it’s necessarily a bad idea given how a lot of the technologies evolved around the storage back end and things they have available for it. I think it’s still reasonable, but where I see the biggest challenge for it is the scalability factor. If I want to run multiple replicas, if I want to run hot standby, if I want to run different topology configurations for my database, there’s challenges in that getting the orchestration right and that’s something that just having a typical pod I think, pod deployment staple set whatever that that model, I don’t think is handled the greatest with Kubernetes because at the end of the day, you have an init container that runs some shell script that does who knows what. That’s, I feel like, very fragile. I think where it starts getting more interesting again is when you have an operator come into play. That starts extracting the configuration it has that additional control loop running to clean that extra information about the state of things what the configuration should be and knowing what I want to get at so I think controllers and operators are a good fit for that, but then it does change the operational profile of that of how do I maintain it? Now I have some other thing that’s trying to do the same thing I’m trying to do, and you have a lot of friction with trying to manage that. I think we’ve seen it with the LCD operator and other stuff which there are challenges with that. I think that kind of goes with what Brad has mentioned is, you do start developing the more advanced users and the stronger skill sets with that. It’s worthwhile, but it does take stronger skill sets to get there.

Chris: I’m just looking at it from the practical perspective. I agree with you – everything both of you have said. But when you consider everything that goes into let’s say you’re an AWS customer customer, right? When you consider everything that goes into standing up a Postgres database or Postgres cluster on EKS or on Kubernetes and getting all of those things squared away and figured out versus just deploying Aurora, right? Setting up an Aurora cluster, like the amount of responsibility that you have by running it on Kubernetes versus Aurora. The practical aspect is to just use Aurora. Amazon handles all of those replicas, all that scale and everything for you. Now if you’re, I mean, I suppose I started this off a little bit wrong. I don’t mean necessarily that it’s always a bad idea. I mean if your Google, or your Microsoft, or your Apple, or your even, and those are extreme examples, I fully understand that. I just can’t think of a mid-tier example in my mind, right? All of the telecom providers for example run databases on Kubernetes. They have the ability to, but they also don’t necessarily have access to the cloud resources like Aurora, for example, where they can easily spin up a database there. They actually have to have it inside their cluster because they have no other alternative because they’re running all Kubernetes bare metal clusters, so in those instances, yeah I mean it would definitely make sense to do that, I’m just saying, I think for the vast majority of customers, there’s easier alternatives where when you look at it from a practical perspective, it stops making so much sense to run a database on Kubernetes. But if you’re a big customer, sure yeah.

Brad Ascar: You want the portability, you want to be able to go from one provider to the other, you don’t want any kind of lock-in, right? There are a lot of other operations including advanced encryption scenarios, and things like that, where you don’t want anybody else to have any of the keys, right? So you can hold all of those yourself. So there are plenty, but again this isn’t the everyday customer that wants to go buy a Toyota. This is somebody that wants to race Ferraris and when you race Ferraris, you get a lot more performance and you get a lot a lot of benefits out of it, but you have to know how to handle that much more of a complex system.

Chris: And up until a couple years ago when you race Ferraris, you had to rebuild the engine every couple races, but I mean hey, you gotta go fast and you achieved its purpose. It had a different purpose, just to add to your point, you’re right. So can you tell us the topic again, oh great moderator Noah?

Noah: The topic itself was: When to not use Kubernetes. So let’s take it in a slightly different direction. So far we’ve been primarily focusing on databases. What other use cases do folks see are good use cases to avoid Kubernetes and or as you mentioned earlier, to potentially deploy primarily into cloud services as opposed to managing them yourselves.

Brad Ascar: See some people’s own development practices, trying to deploy local Kubernetes for doing things that they could just do in containers, they can just do in another way, adds a level of unnecessary complexity to what they’re doing and it doesn’t really give them a lot of benefit. So yes I’m doing the same thing as I’m doing someplace else in the same way, but there’s no benefit to doing it that way. So you know, it’s like anything else. It’s the appropriate tool for what you’re doing at the scale that you’re doing it and for the reason that you’re doing it, and if it’s not then why throw all this extra in where it could actually be a very different use case, right? So I’m running locally and then I check it into Git and then it gets pulled in someplace else. This side doesn’t have to necessarily look like that side to do that kind of work. So I see people trying to use a very large tool for something that’s very local. They’re editing a half a dozen files and it’s not the same complexity as the environment where it potentially runs.

Chris: So I’m going to take another edgy approach and kind of add on to what Mr. Ascar said. I think in general, if you’re not planning on running five Kubernetes notes, like three to five Kubernetes nodes minimum, you probably shouldn’t bother with it because there’s a cost-benefit analysis that comes into play, right? Where to get Kubernetes running and get it working and get educated on it and get people who know it and get you know a proper prop everything up, there’s a significant cost there, and once you pass that cost, you start really getting the benefits. And I think that if you’re not going to be saturating or using anything less than three to five nodes, you’re probably going to be expending a lot of energy, a lot of time and money to be able to get Kubernetes up and running and get it in your environment when you could probably just take advantage of any one of the cloud container services, or in that fact even like Docker Swarm or just running bare Docker containers on VMs… 

Brad Ascar: …or something like Nomad that’s a little lighter weight.

Chris: Yeah, well you if you’re not gonna do anything more than three to five nodes, you’re gonna be running into things. Now there are always exceptions to rules and a good example of this is you may run three nodes, but you may have the need to have your developers do like preview images as part of your CI/CD. So like let’s say you’re building a website or something and you need your developers to have preview images on merge requests, so every time a merge request comes through, your e-learning website or something spins up and you can check the changes and make sure the changes are good before you approve the merge request. In that case, maybe you would run a Kubernetes cluster with less than three to five notes because you’re getting more benefits as part of your CI/CD pipeline process, as opposed to your running, right? So you got to be honest with yourself and do a real good cost benefits analysis and I think that three to five nodes – if you’re not going to saturate five nodes, it may not be worth it.

I’ll add another thing. Mr. Beam here said some of the things that they were running like a free ipa on containers and you know database applications and containers and stuff,

I’m not 100% sure that that’s a bad idea, but I’m not 100% sure it’s a good idea either. A good example of this is you can run a Linux application in a container and forward the the X server connection to a thin client and run your Linux application on the thin client, but also in the container. That’s kind of a cool concept, but I don’t know to what extent that’s going to be great. So that’s more of a question than an open thought provoker I suppose.

Brad Beam: I think from my perspective one of the big advantages to Kubernetes is more on the soft side of things, but then translates into an actual benefit. So like coming from the Ops world, you hop on a server, you go do something – maybe you install additional tools, you create some temp files, you move stuff around, create backups – there’s so much drift and craft that happens on the servers that you just accrue so much debt from that. Then comes time for security audits you’re like, oh well why is this random package installed on two out of the ten servers? Was this intentional? Why did this happen? So one of the things that really started to be a mind shift for me when we were transitioning from VMs into containers and then into Kubernetes was the whole idea of immutability. And I think that’s one of the real real strong benefits you get with starting to embrace Kubernetes and cloud native principles is that idea of immutability. If I have my jump box that I allow people to log into, it’s got a potential where I can just delete that pod and have my fresh starting state with no additional crop left there. That is to me one of the big, big benefits that you get over a traditional config management system. Use a config management system and you basically say I want these packages installed on there versus I only want these packages available. If you try going to that detail of being very precise I only want these time packages on there, I feel like you end up writing a bunch of custom stuff and you run into a lot of overhead in managing all that dependency tree and everything that’s needed for it. You can go to the extreme then disallow any sort of package installs a repository has been configured and like, but when you go to that extreme, you hurt your operational people from going in there and troubleshooting. Maybe it’s they’re missing tcp dump, maybe they’re missing mtr, maybe they’re missing who knows what. Those are things that probably could be addressed through processes and other things going forward, but I feel like when you’re waking up at 5am to a call and trying to deal with stuff, you don’t want to have to deal with those types of issues blocking you from getting stuff up and running. So that’s one of the benefits that I see from having something like an ssh jump box, which traditionally I feel is a VM as being a container is that it’s very easy to reset that. Get me back to a clean state, clean up all the stuff that I was doing for my investigation purposes, and then move on. So that’s where I see this evolution of going from more of a traditional VM architecture to containers and Kubernetes really picking up.

And I think that’s what really kind of transitions my mindset for a lot of this where I think if you can run it in container, run it in Kubernetes, I fully agree that there are going to be instances that don’t make sense to run in Kubernetes, but I think those are starting to become very much the minority.

Chris: So you’re saying don’t use Kubernetes if you do want configuration drift and bad VMs?

Brad Beam: Of course, yeah I got that mutable infrastructure going.

Chris: I agree, I mean I think that the more everything with Kubernetes gets adopted and the more that things, get how should I say, worked out or more user-friendly and get better and better as time goes on, the bigger question is going to be why not use Kubernetes, you know? And that’s going to be the big question there, so.

All right, you have another topic, Noah the wise one?

Noah: I do. All right, one more from the hat since we have no one else joining us. You ready?

Chris: All right, let’s see. Let’s see what we got.

Noah: Proper ingress load balancer setup.

Chris: Oh wait, was that a topic I came up with it?

Noah: It was, which means you have to start us.

Chris: Man, where do I start with that. I was just kind of coming up with topics. You put me on the spot.

Brad Beam: …dns, it’ll sort itself out.

Chris: I mean, I feel like if there’s anything that could break, it’s going to be dns.

Noah: 85% of the time it’ll be dns.

Chris: On that note, why hasn’t somebody come up with like a tool or something to make bind configs easier. It’s just bad.

All right, so what was the topic again, Noah? Let me hear the topic. These folks got me all derailed.

Noah: Proper ingress / load balancer setup. I can throw it out and pick another topic.

Chris: Yeah, let’s do that.

Noah: That’s going back in the hat. Sorry so we can bite you later.

Brad Beam: I have one comment about that. That was kind of fun. So with having the freedom we did with the the doing Calico and BGP peering and getting our network set up like that, is we were actually able to start doing anycast with our containers and our network topology and that was something that was not quite in the same vein of, oh related to load balancing but a little different, and again I more just that was kind of fun, it was interesting, but I think one of the things that highlights is just there are a number of things that aren’t commonly talked about with Kubernetes that you can do and it’s really really fascinating to see how far it’s come and what you can actually do with it.

Chris: So that’s actually something I was looking at the other day too. I’m glad you brought that up because I was looking at anycast and multicast from containers and so forth and going through like down that giant rabbit hole of documentation, and there’s a lot of age cases where everybody’s like no don’t do it you can’t do it et cetera et cetera and then you know there were a couple articles where they’re like sure you can do it, it’s super technical, here’s how you do it, but you have to set up your networking layer, right. And that’s cool and I think that’s one of the biggest strengths about Kubernetes, right? Is the pluggability of everything. You can swap out the network layer, you can swap out the schedulers, you can swap out just individual components inside of Kubernetes, and the more that Kubernetes evolves it’s almost like it’s not just a product, it’s like a standard. Almost a standard way of setting something up in a standard number of components that you need to achieve a certain result.

Brad Beam: Yeah, and I mean I think what’s interesting with that is like I come from on-prem bare metal, that was my jam. And it’s very common to hear people just talk about AWS, Azure, GCP, Digital Ocean whatever has their managed Kubernetes offerings and that’s what they’re constrained to and they often, that’s all they got. So I just fit into their ecosystem and do that, but when you are running on-prem, when you do have control of the full stack from the hardware all the way up to the network layers through the application layer, there’s a lot of things you can do. I mean of course with the right talent, but if you have the right network guys, if you have the right way of thinking and you start working with your network team and actually understanding how you can do this better, I think you can definitely start to put together some topologies that rival a lot of the cloud providers. And I think that’s something that’s often overlooked is oh if you’re doing better metal Kubernetes you’re handling it on your own oh man good luck like that’s just the lost cause. And I don’t think that’s true at all. It does require additional management, you do have to have a stronger skill set, but you can do really really good stuff with it.

Chris: Agreed, I also want to add in there that sometimes it’s cheaper.

Brad Beam: Yeah absolutely.

Chris: Because one of the prevailing theories with cloud was is that hey you know at least with some of the executives I’m in, I’m not going to say it’s a prevailing theory across the whole industry, but some of the prevailing theories that I’ve met with some of the executives that I ran into was, hey we’ll do cloud because it gives us that flexibility, but also we’re gonna save money, right? We’re not gonna be able to have to have all those SREs, or all those people on staff, or something, right? And then they start going down that road, and they start going down the road of the cloud provider, and you start running into those instances where you expect something to work a certain way and it doesn’t work that way or you have an outage or something and your staffing just changes, right? You still have similar staffing levels it just changes, but now you also have this cloud bill that’s usually not insignificant. I think just myself with an application that wasn’t live, I ran up a $3,000 cloud bill on Microsoft Azure one day for a personal project. Those cloud bills can get expensive very quickly, very fast. And sometimes, honestly, it might just make sense to just buy a bunch of servers, dump them in the data center and throw Kubernetes on it. I mean a lot of the places that I’ve seen, I had a rack and a switch data center, right? And cost us about $500 a month for a full rack and full rack was 48 or 42u, right? You can fill that thing up with Dell servers and run Kubernetes pretty much all day long, compared to you know a cloud provider bill, which the same amount of CPU and compute could be very expensive. Kubernetes kind of, and the other argument for cloud was the flexibility, right? Oh I can go spin up as many virtual machines as I want. I can go spin up as much of this and the other thing as I want. You still get that with Kubernetes now and it’s even getting even better.

Brad Beam: Yeah.

Chris: So, yeah. This is not a paid promotion product placement for Kubernetes, just saying.

Brad Ascar: I think the other thing too is that some folks are more advanced in the thinking, don’t think it’s actually going to save money, they actually think it’s going to cost more. Opportunity cost, what else can they be doing to go after business functions rather than maintenance functions, right? And so they say it’s actually a calculated risk. I’m going to spend more, but if it allows me to go faster in the applications that I want to actually advance, then it’s actually worth it because opportunity cost allows me to leapfrog my competition who’s spending time worrying about racking and stacking in cables and they’re putting emphasis in those places. And I know that’s something that changes skill set changes, but actually that speed is worth it to me. It’s actually worth the cost for the speed because I can leapfrog my competition, so that depends on what your view is, but if you think it’s a cost saving measure, it’s not a cost saving measure.

Chris: Yeah no and that’s actually a really good point. One of the things that I always taught when I was teaching CI/CD classes is the best startups iterate quickly. You want to see how healthy a startup is, you want to know how healthy a startup is if you’re investing in it? Look at their release schedule. Look at their iteration cycles, right? If they’re doing releases every three months and iterating every three months, they’re probably not a startup that’s a really good idea to invest in outside of some minor exceptions. And that’s by going with Amazon and going with some of these cloud providers and utilizing some of their big credits, like when I formed my LLC a while back, we got $5000 dollars in cloud credits. For a small company that goes a long way. For me not having to worry about the infrastructure and be able to just build the app and focus on it, I mean that’s actually a very good point.

Noah: Okay, do folks want another topic?

Brad Ascar: One more topic.

Brad Beam: Ninja steal the old topic and talk about something else.

Chris: Yeah, we might as well just upload this video and title it like “When to use Kubernetes and when not to use Kubernetes,” because I feel like we’ve done pretty good on this subject so far.

Noah: Let’s see what the next one is.

Chris: This better not be one of mine.

Noah: It is. Testing as part of CI/CD.

Chris: Oh yeah, so this goes back to the whole thing about when to use Kubernetes if you have less than three to five nodes you know, being able to test your applications on Kubernetes in a CI/CD fashion…

Let me take that back. Being able to utilize Kubernetes allows you to run CI/CD pipelines where you can run almost an infinite number of tests compared to your cluster. And if you are big on iteration, if you’re big on testing, and your company has a very low tolerance for errors or failures or defects in your software like, I don’t know, if you’re NASA or like Tesla and a defect in your software means something really bad happens, you probably want to test a lot. And I think that’s probably, looping that back into our previous topic, a probably a good reason why you’d want to use Kubernetes, is because you can test and run multiple tests of software simultaneously in their own environments. I think that you know for our audience members and people watching this, I think that, I’ll be honest, people don’t test as much as they think they do.

Brad Ascar: Yeah, I think the other part of that is not just the unit test, not just the things that give you the green check boxes, but also performance testing. Understanding the performance of the application and if it’s drifted from previous configurations. That’s one of the areas we focus on, of course, but it’s really important because before you do your releases, you want to know that thing still behaves – it still behaves in the same way, it can still take traffic in the same way, that you haven’t introduced something that gives you some sort of admiration in the operations of your application because that’s true testing, right? It’s not just does it pass its own unit test. Does it actually pass real test taking? Real data, real load, simulated load, right? For the use cases that you have and has that changed because you’re changing the application. It could be the application change for a legitimate reason – you added a new feature, it’s a new version, it behaves differently because it needs more CPU or memory – or something else got checked in, but you didn’t know some library changed its default from this to that or CDE now disabled mathematics functions on certain kinds of CPUs and your app now behaves very differently. You want to know that before you go because if you push it out and you’re using even using advanced processes using canaries using blue green deployments whatever, you may not see that scenario until you’re all the way rolled out, and then you take the proper load in production, and then you have a problem you have to roll back. It’s good to know that you’re testing properly and trying to avoid those scenarios, particularly when you are in a low risk tolerance kind of environment. But anytime you’re pushing something out like that that isn’t performance tested, doesn’t matter the deployment methodology, your users are now your testers and they’re going to get bad results because of that. And when they get bad results, it may not mean anything, it may mean that they go to competition, right? It could be a small percentage because that you find it immediately in the community it may be that it rolls all the way out and 100% of your users will have that problem at 9am when everybody logs in, right? So it really makes a difference that you have some comfort in the way that you’re testing and that it’s actually full testing, not just unit testing.

Noah: To go on that a little bit more, one of the things I’ve seen is people assume that testing of any kind is something that once you have it in place, you don’t really have to think about it, you don’t have to continue to evolve your tests with the application. But then they also think that it’s a replacement for some of the other standard test life cycle procedures, so they’ll say well I have some testing built in. Whether or not it’s proper full testing actually does end to end testing if it’s just unit testing they’ll say well we’ve got our tests in place we don’t need a staging environment anymore, we can just deploy straight to prod, and then things immediately go on fire. I’ve seen this live in the wild and dealt with it. It is not a replacement for staging, it is not a replacement for actually understanding what your application is going to do.

Brad Ascar: I can’t remember whose quote it is, but “Everybody has a QA environment, some people are lucky enough to have a production environment that’s separate.”

Noah: It’s probably charity.

Chris: We all need to show up to the next office hours with t-shirts that says I only test in production. So I’m going to throw a little shade here. I’m going to throw a little shade on three separate communities. If you’re part of the nodejs community, the Python community, or the Ruby community, you probably have no idea what’s in your dependencies.

You, I mean, companies legitimately will go through that and you know go through them and white list certain ones et cetera et cetera, but you always have one developer who thinks the rules don’t apply to them – usually it’s me – and then they’ll bring in a package or something and it’ll be bad.

Testing is really important to shake down your dependencies and especially in Kubernetes where you have self-contained containerized environments, where I don’t know, if those dependencies suddenly get some form of malware or something in them, because you’ve updated it, and you didn’t do a source code check on the version because let’s be honest, who checks all the source code for all their dependencies, I don’t think anybody does. So something happens. Key gets leaked, your dependency gets malware in it, you run your application in a container, and suddenly the container is entirely encrypted because of malware, right? Hey it didn’t happen in your production environment because you tested it in a container in Kubernetes, so you just delete the container, you remove the dependency, you spin back up and congratulations now you can go to lunch. You don’t have to spend your lunch hour trying to fix something, you didn’t lose your job, and you don’t have to tell your boss that you broke the production servers.

Brad Ascar: Well aside from dependency, how about just version locking. That saves you a lot of grief. Latest whoa, it doesn’t matter if it’s your container, it doesn’t matter if it’s a library, at each point a dependency chain, and your use of the dependency chain, they give you these tools to lock it at versions, but if you just always take latest, you’re just you’re just asking for a world to hurt. Somebody will change something that will hurt you.

Noah: I used to teach the Docker classes and one of the things we started with whenever we talked about tagging was, please for the love of all that is good in the universe, never use Latest in production. Always use at least a shaw hash or at least a tag that you’ve set yourself, but tags are mutable! I could take an ubuntu image and tag it as a Debian image or tag it as a Centos image, it does not matter if the names are mutable. You need to know what’s actually in it and use that sha hash or you’re not getting what you think you’re getting.

Chris: That’s a key thing with Kubernetes too that I think a lot of people that are either new to Kubernetes or don’t really understand how it works. Sometimes they’ll build deployment configs and stuff with the name in it and they’ll use the tag name instead of a hash. Thankfully, Kubernetes system protections built in to kind of prevent that and turns it into a shell, but some people still overcome that and use the tag name, so that’s a very good topic. If you were able to teach Kubernetes class, something to keep in mind.

But I mean Kubernetes and CI/CD pipelines are really, if there were a reason to have Kubernetes, CI/CD is probably one of the top ones or should be.

Noah: Any other comments? Questions? Pontifications? We’re coming up on our hour.

Chris: Noah, do you want to draw another topic?

Noah: I can.

Brad Ascar: I think what we learned from the previous conversation is don’t have a Chris on your staff that’s actually doing those things, right?

Chris: That’s terrible!

Brad Ascar: You said it yourself, I’m just repeating what you said.

Chris: Not all Chris’s are bad. I’ve learned my lessons. Listen I’ve had to write a shareholder statement as to why we lost millions of dollars in an afternoon, I’ve learned my lesson.

Noah: Okay, I picked one more topic from the hat.

Chris: Is it one of mine?

Noah: No, and it’s gonna be an uncomfortable one…

Chris: Okay, let’s hear it.

Noah: …and I’m saying this intentionally. Are you guys ready for the importance of diversity in tech aka why are there so many white guys here?

Brad Beam: Oh that’s a heavy heavy one.

Chris: I value my job, I’m not sure when it touched that one.

Brad Ascar: That is a challenge. It’s actually something I think that needs to be directly addressed. I think that it’s something that we have as a core value here – I’m actually part of the D&I team here at StormForge. I think it’s really important and sussing out why that is and historically why it’s been and making changes is really really important. I think you have to be adamant, you have to make sure that it’s supported from the top down, which we have support from the top down, and it’s critical because it gives you diversity of thought. It doesn’t give you microculture. We’re a lot in agreement on a lot of things because some of our experiences are the same, other people’s experiences aren’t the same and that diversity and how you operate, how you interact, how you communicate, is important. I think it’s a huge piece to making a company well-rounded and critical for our success and why we’re putting so much effort into it.

Chris: So I’ll say this, I think that diversity is exceptionally important not just because it’s the right thing to do, but also because I think that it makes a company successful. When you have something of diversity and you start working on diversity, you start getting a couple different things. The first thing is that with true diversity, and not just the kind of diversity where you put it on your website and say hey we’re diverse, but with true diversity that’s hard and difficult and has difficult truths to it and is uncomfortable, the teams that you work with and the people that you work with start having better communication, better transparency, they’re able to express their emotions and feelings better, so your team members don’t leave a meeting feeling like hey i feel unheard or something like that. With true diversity you start getting those kinds of passive effects by being inclusive of everybody and understanding people’s feelings and emotions and also understanding that everybody doesn’t come from a background that you come from. The second aspect of that is that a byproduct of that is the fact that through that openness and transparency and through those shared values and being able to communicate everybody’s backgrounds and upbringings and different looks and approaches at things, you’re able to problem solve things better because there’s nothing that makes problem solving worse than a bunch of like-minded people in the same room looking at something trying to solve a problem. They’re going to come up with the same answers and same solutions. It really requires diversive thinking, diversity of mindsets, diverse backgrounds and that’s something that just just comes with different people of different cultures and different walks of life. The second aspect of that is I think that it’s really the right thing to do for a lot of reasons. I mean not just being respectful to people and treating people like human beings and understanding that people are different etc etc, but I’ll kind of divulge a little bit to a story… When working at GitLab and GitHub when they started the renaming of the master slave or like master secondary branches and and server names and so forth to like main and secondary, right? and they went from master slave and so forth, a lot of people that I know were a little bit upset by the change. They couldn’t really understand it. They’re like hey it’s been the terminology in computers forever et cetera et cetera, and part of that comes from the fact that they were not around people necessarily routinely who that terminology had hurt, right? I had team members that came from other cultures and team members who came from a lot of cultures where those terms had a significant meaning to them and it wasn’t something that was like a historical meaning or something. It was something of their upbringing, their childhood, something very dear to them, something very recent to them, something that affected them negatively and hurt them because of the culture that they grew up in, right? And so having those terms and so forth is not inclusive. It drives those people to not feel accepted, and it drives those people to not feel involved, and not want to be involved, and if we’re going to strive to be better about things, everybody kind of needs to be involved. Everybody needs to feel that they can contribute. Everybody needs to feel like they can be part of the group, right? There’s no individual part, it’s us – everybody involved. And I think that diversity and inclusion really drives that. So I guess a little bit of rambling there, but to kind of recap, I guess the point of it is that diversity and inclusion is not just the right thing to do, but it’s also extremely helpful to bring people in and make sure that they feel heard, make sure that they feel listened to, and make sure that they can contribute, so that it helps everybody. It helps the whole project. Rising tide raises all boats, right? The other aspect of that is that we need to understand that people come from different backgrounds and different cultures and sometimes things that we do will hurt them and we’ll pain them and while not intentional, we should do what we can to avoid that because frankly, it’s not nice to go around and hurt people. Whether inadvertently or not, it’s important that we’re mindful of that.

Noah: We’ve got a guest on the line!

Chris: All right, you kind of dropped in in the middle of a tough question, Guest. Sorry.

Noah: I think that might be intentional.

Rin: It totally was. I read Twitter.

Chris: Oh perfect.

Brad Beam: I think part of what I’ve come to realize more is just historically how much is just messed up. There’s really no other way to put it, and a lot of it you can’t help but, you feel guilty about coming to a certain point in your life, and you’re like oh yeah everything’s fine, it’s happy. I’m cool. And then all of a sudden you get this groundbreaking revelation of just how messed up things have been in the past, and it’s difficult to swallow. There have been a few talks I’ve watched recently about the other side of history that we typically aren’t exposed to from the American side of things, and I watched it maybe a week or two ago and at least for a couple days afterwards you’re just really trying to come to terms with like, this is just messed up with what’s happened. And you know it’s important to learn from what happened and things that have gone on, and really make a conscious effort to try to change for the better with that because like a lot of the themes that you brought up, Chris, about respect and treating each other appropriately and that stuff, like that just wasn’t the case. And I think it’s so important to just embrace that people are humans, we’re in this together, like we got to get over that. So and the specific question, why are there so many white guys right now? Historically it’s because we’ve set up this environment where we’re here. I’m very hopeful and optimistic that as things progress over time, that will change.

Noah: Rin, would you like to jump in? Would you like to contribute to this conversation, or are you just here to listen to us rant about it?

Rin: I absolutely would. Oh gosh, this is being recorded I just noticed that so I’m gonna think for a second before I answer the question.

Brad Beam: Oh for sure.

Noah: I’ll reread the question as it was stated when I pulled it from the hat, the importance of diversity in tech aka why are there so many white guys here? Which was incredibly germane when we had this Zoom and there were only white guys on the call. I’m not sure that’s a, I gotta be honest, because the question was directed, I answered it, but I’m not sure I’m the exact right person to answer that question. I’m not sure the four of us in here really are, right? I mean we can grasp it with what we can, but I would really like to hear your opinion, Rin, when you can formulate it.

Rin: There’s a lot to this. I’m coming out from a perspective that I am Jewish and I’m disabled. I come from a non-traditional tech background, and I’m non-binary, I’m trans, and I have ADD, and all those other things and I didn’t have a CS background, and tech was not really something that was offered at my school to people like me, so I didn’t get into this industry until 2015, and I was already in my 30s at that point. So I wasn’t jumping into it right out of high school. It wasn’t introduced. I wasn’t able to do networking classes or have anything like that in high school. It was you get one path and it’s probably something home skills related. You got that path, and that was the way you were offered, and you had to deal with it because you didn’t get any of the fun options. I feel like there’s a lot that can be done and I think that it is the responsibility of people that have more privilege to say whatever, not necessarily what are we doing in the future, but what are we doing right now to address things like the fact that there aren’t… it’s not even a pipeline problem because I think that’s a cop out saying, oh it’s just a pipeline problem. No, there’s people, you just need to change your hiring policies so that you can hire the people without the gatekeeping. It’s not the pipeline, there’s people with the talent out there, you’re just not hiring them. When it comes down to it, it’s like it’s a matter of making the hire and even if they don’t look like you or they don’t have the same background, hire the person that has the skills that you need that might not have the background that air quotes “looks good on paper”. Hire the person that can do the job, even if their background doesn’t match the ideal candidate. There’s people that are skilled out there that might not have matched that entirely, and it’s just a matter of breaking down those gates and saying we can do better, and that’s really what it’s about — paving the way and saying it’s not a pipeline problem because that pushes the responsibility off of you. By saying oh it’s a pipeline problem, it makes it so it’s abstract. It’s not your problem anymore. It’s somebody else’s problem. It’s the problem of the mystical pipeline, but these people are out there already making content, already doing the work, and they’re just not respected because they don’t look like you. So I think that’s what you’ve got to do is go back and say what are we doing to not only hire people, but retain them. What do you do, what supports are in place for these people when they get to your company because it’s not enough to say we want to hire them. You’ve got to go further than that and say how are we keeping them there? What are we doing to support them? To make sure they’re psychologically safe? To make sure that they are valued and that they are respected and they are paid equally to their peers and if not more so? I think that’s very important is that you have to understand that you can’t just level the playing field, you have to forget there’s a field to begin with. So yeah, that’s my two cents.

Brad Ascar: Well thank you, thank you for that. I think it’s really important. I think that as you described, it’s not just a pipeline problem, but also initiatives within our own company and within companies to make sure that hiring practices are actually something that aren’t inadvertently harming folks and and eliminating people, that the vendors that we work with also adhere to the same kinds of practices that that we adhere to, right? And hold them accountable for that as well. And when we see that there’s not diversity in what we’re seeing from applicants, we have to ask the question why. Are you not being inclusive enough? Is there something going on with the hiring process that doesn’t bring us candidates? Because we know the candidates are out there.

Chris: I want to kind of zero in on something that you said, Rin, and I hope I’m pronouncing that properly. You pointed out something about how people you know, being paid in retention, retaining people, and how making sure that people are paid according to value and position and whatnot equally, I think that’s something that we as individuals need to handle and understand, right? Because as we work in workplaces with other people…

…I can’t speak for other countries, but America in every company I’ve worked for seems to have this thing where nobody’s allowed to talk about pay, right? Like your paycheck’s your own private thing, don’t tell anybody, right? I’ve worked for a couple of companies where somebody who was equal in talent to me was paid less than me, and to me that was deeply upsetting, and I think that as individuals, if we know that, if we know that one of our peers is being significantly treated less because of that, that’s something that needs to be brought up, and that we shouldn’t be afraid to bring up to management and to people above us, that hey this person’s my peer. They’re putting in equal, as much work as I am if not more work than I am, but they’re paid significantly less. What’s the deal? Why is that? and that’s something that we need to police each other. Yes yes, I did.

Rin: Did they do anything about it?

Chris: No, of course not. It was not a great hiring, it was not a great place to work in general so I left shortly after that, but I guess my point to that is we need to keep each other honest about it. We need to make sure that our leadership, our management, and the people around us are honest about it as well.

But yeah no in that particular case, I did bring that up. That was not necessarily a great company to work for, I’m obviously not going to say the name or anything, but that was not the first time that something like that happened and it wasn’t the last time and they’re still in business and the same things are still happening and you know you stand up in a business as much as you can with that, but in some cases there’s only so much you can do other than you know leaving and finding a better better opportunities for the people around you.

I mean I think that that’s important. That we keep each other honest, and we keep our management and our leadership and our communities and so forth honest, and when we see something like that we speak out about it, and we say hey this isn’t right, this isn’t fair, this isn’t being nice, this isn’t the right way to treat people et cetera et cetera, so that’s my thought I suppose.

Noah: My thought is that this is a topic that I don’t think we can give enough attention to and as a quippy statement I’m going to take this topic and I’m putting it back in the hat, so they can potentially come up later again.

But we pulled this topic out of the hat with 10-15 minutes to go in this presentation, in this session, and A. that’s not nearly enough time and B. it deserves a lot more conversation than that, but I also think it is important to have these conversations, the question of why are there so many white guys here in rooms full of white guys so that everybody can take a step back and look and say what has got me to this point and I think having these conversations in the uncomfortable ways are something that needs to keep happening and we need to keep having it and pushing it so that when you see you go to a tech conference and you realize there’s you know one person in the room that’s not a white guy… time to have that conversation publicly in front of everyone and make everybody uncomfortable so that they can think about it.

Chris: To add to that, it needs to not just be bringing people in for the sake of diversity. You need to put people in leadership positions. If you’re at a conference and all the speakers are all a bunch of white guys, that’s not good. But just as bad as that is a conference in which all of the important exciting topics are white guys, and all the quote unquote boring topics whatever those may be, are given to the non-white guys. There was just… that I think that’s equally as bad people. If you’re going to do diversity and equality, which you absolutely should do and you need to do, and if you don’t do it then

I don’t know, I don’t want to wish you bad will or anything, but it’s not the right thing to do. It’s not good, but if you’re going to fully commit to equality and diversity, you need to go all in and make sure that everybody has those opportunities, right? You shouldn’t just have a conference full of all the white guys getting all the exciting topics and somebody else you know non-white people guys having this is very awkward for me to talk about so just just bear with me here because I’m probably not using terminologies right now, I don’t mean to offend anybody. Please don’t assume negative intent, but my point is that if you’re going to give all of the non-white guys topics like how to debug a container and how to handle support requests, and you’re gonna give you know all the white guys the exciting topics like how to drive a car without artificial intelligence, that’s not equality, that’s not diversity, and that’s just not okay.

Brad Ascar: Which is also why we need to make sure that systems are in place as well, so that things like the review and calls for papers are anonymous, are based on the topics, right? That if you don’t have any of those biases just because you know somebody and you’re on that committee, you put in the place the systems that make it so that everybody has a fair chance and a fair playing field, and so the systems that we put in place to support things like conferences also need to be addressed and need to be done in such a way that there’s 1. Requirement for more diversity and inclusion and 2. It’s in ways that you can affect the system, you try and take the bias out of the systems and a lot of the systems have bias built into them. So you’re intentional about that.

Noah: I would actually argue against some parts of that. In the idea of keeping everything intentionally blind, you are inadvertently introducing bias because now you’re looking for topics that appeal to you as a particular person, so without being able to identify yes this is a speaker that should be given a chance, you will have those same biases manifest.

Brad Ascar: Well depends on who you have reviewing, right? So that’s a broken part of the system. If the choices that you pull, then you’ve got a problem in your committees in the way that you’re actually addressing that as well, so… There’s no perfect answer here, right? But you can have both as well. You can have an intention to bring in more folks and not take that, but for everything else you can also decide to take it out.

Noah: I like a comment from Rin in the chat saying instead of saying blind, we should propose to say keep identifiable information obscured because there’s another you know ableism issue that when we’re talking about diversity initiatives we should be doing our best to try and avoid. I’m 100% on board with that. I know we have a lot more that we could potentially say on this topic, do you folks want to keep going or should we call it here and save it for next time?

Chris: I think we should stop honestly. Aside from Rin, we’re four white guys in the Zoom channel trying to talk about diversity. I feel like we’re not the best people to get on this road, right? If we have this topic, and I think that we should because this is a topic that’s important, if we have this topic on Office Hours, we should definitely ensure that we invite and bring people who are not white guys.

Noah: Straight up, agree.

Chris: I mean we’re not going to solve this and we’re not going to solve this just because we don’t have those viewpoints. We need those other viewpoints.

Noah: Okay well, I think that brings us to the end then. I don’t…

Chris: …and thank you for coming, Rin. I see in the chat that’s why you came, but thank you absolutely for coming.

Noah: This concludes our first office hours, and as I stated in another channel, this got deep really quick, but it is absolutely not fair to have technical conversations without a human element to try and say let’s have only these tech conversations that obscures the reality of what it is to be living in tech and to be asking questions. So thanks to our panelists for giving their viewpoints, thank you very much Rin for showing up and for spurring on that conversation and preventing us from being four white guys in a Zoom.

Chris: You picked an excellent last topic that’s why I was laughing a second ago. It’s just realizing the irony of like five minutes left and you pick one of the topics that we could discuss quite a bit…

Noah: …for hours with lots of people. I’m gonna stop recording now and I hope that folks come back and join us in two weeks for our next office hours! Goodbye!