Ignored By Dinosaurs 🦕

My wife is a Glennon Doyle podcast listener, and on a recent trip she put her on for a little while. Plenty of good stuff, but the phrase that's stuck in my head right now is

The intention doesn't matter. What matters is the impact.

One expression of micromanagement is the inability to sit and let your teammates (possibly) make a mistake on their own. The way I caught myself doing this just now was an innocent question being asked between two teammates for which I knew the answer. I am also on vacation, so I shouldn't be there. They were talking it out and I jumped in and answered it.

Implicit in this action was that I didn't trust them to figure it out correctly for themselves. It's not that I didn't trust them, but it probably comes across that way and what matters is not my intention but the impact. I'm potentially stunting the ability of this team to think on their own and that is something I am totally grossed out by.

So peace out and do better next time, Self.

#business #management

Because I want to remember this for next time:

“Data modeling” is a kind of nebulous concept, but I define it loosely as

The empathetic act of thinking ahead about how somebody might actually use this data that you're creating or storing.

It turns out to be a skill that not everyone possess, and which a previous career stint as a web developer served me well. I think for the next technical interview I give, regardless of position, I will ask the candidate to explain how a simple blog data model works. A data model like this one...

#management

I caught this thought train last night and hopped off before writing it down. It has to do with the visible trappings of success. First the obvious bits.

True success isn't something that someone gives you. It's not something that you get. It's not a thing. It's a feeling. I am currently riding the wave of it for maybe the first time ever. I am doing my best work. This post, written over a decade ago – I'm finally there. It may not last, and I'm ok with that. I'm grateful for it now.

Thing is, for a long time up to sometime in the last handful of months I thought about the visible trappings of success – title, how many people report to me – they took up mental space. I think because I've always kind of had self-confidence issues and those things function as kind of a buoy or channel marker for me to know that yes, in fact, I'm headed in the right direction according to societal norms. They boosted my confidence in a fleeting but still meaningful way, because eventually the flywheel is spinning on its own and I realize I'm not thinking about any of that shit anymore. I mean, I need more reports because I need more brains and hands to execute on the job now, and not because I care how many reports I have.

This is one of those thoughts that's buried under a lot of muck and I don't really understand it much more clearly than I've attempted to lay out here. It's sort of like when a coworker recently asked me “did I just make a mistake by turning down the unexpected opportunity to have more folks reporting to me?” and my first reaction was something like “don't optimize the wrong metric” but that's a bit overly simplistic.

Whatever you need to help you feel successful is helpful to get you to the place of being successful, as long as you don't confuse it with success itself. It's like career therapy for middle aged white guys.

#life

Not sure if I mentioned this in the last post, but what I'm doing right now is essentially building a data warehouse for the company. It's from scratch, there was nothing here beforehand, so I get get to/have had to chose everything from the tech stack to the processes to my favorite part of late: naming things.

The name you give to a piece of software, or a command line flag, or a column in a database is an act of asynchronous communication with another human. You are asking them to care about the thing you've built. If they chose to work with your tool, you are asking them to understand the choices that you made in its design.

The most selfless thing you can do is think about how much effort it takes for the user who did not design the thing to understand how to use the thing. Name things according to what they are and what they do, make it intuitive. This is what design is. It's not the parts that most people will never see, it's the parts that most people will only see.

#generaldevelopment #data

Started here at this company about 5 years ago, a smallish technology startup (30 or so ppl) HQ'd in Paris founded by some really smart people that I wanted to work with.

I didn't view myself as having the engineering skillset to go out for the engineering team, so when I saw a post for a Solutions Architect that fit my resume perfectly I applied. It was a move out of the “back of the house” programming toward the front of house, talking with customers, working with the Sales team to bring in new customers. This was all I knew at the time about the role.

Turns out I loved it. I loved talking to customers and solving problems and making sure that they knew exactly how to be successful when they got started here. I love looking at the big picture and solving problems.

The company at that time didn't have much of a career ladder for that position and it got a bit boring after 2 years, so I asked for a role leading a small team whose manager had just been promoted. I was asking her to let me have her old role, basically. She said “yes” and we worked together fabulously for another 2 years, and built the team from 3 people up to 15 people and made all kinds of fun strategic moves that mostly worked out.

Last summer I started getting itchy again as the team had solidified to a point where I didn't feel that good at what the role needed now, so I decided to ask to start a new team based around data, broadly. It was my POV that the next turn of the wheel here at Platform.sh would be enabled by data, and folks higher up had the same thought at the same time so they said “yes” when I asked if I could build this team.

This was really interesting, because I'd taken over a small team and built it but this was a blank canvas. Luckily, it was a blank canvas that I'd bee staring at for years by that point, so I had a pretty clear picture of what belonged on it. The work then was getting the plan out of my head and into a format that others could read it, disagree with it, and eventually settle on what we're now doing.

It has been a long, 11 year road since I quit that band to start a tech career but here I am now. I'm a technical leader at a medium sized technology company (now 250 or so ppl), working on the for real technical problems and I think I'm pretty decent at it. I still need to work on organizing the pictures in my head into words that others can understand, and making sure that other know where those thoughts are, but I'm learning something new every single day.

thank you for reading <3

#life

This is one of the aspects I didn't consider. A bit of background:

I'll have been here at Platform.sh for 5 years this July, which is crazy because it's now the longest I've ever held a single job in my life. Last summer I was poised to propose that we start a new team here around Data and Analytics right as someone else, much higher up then I, proposed that we start the exact same team. I raised my hand to start this team.

Prior to this I was leading a team of 15 in charge of onboarding and the post-sales $stuff, account management or Customer Success if you want. It was a young-ish but maturing team in terms of processes and to be honest I feel like I'm better earlier when the magma isn't really cooled and the direction isn't set. I enjoy solving problems. Once a set of problems gets “solved” it turns into processes that need to be implemented and refined. This is the natural lifecycle of a business or a department, but I take less joy in those later operational refinements than I do the initial creative period where you're trying to figure it all out.

So I raised my hand to do it again, this time with my dream project – our Data and Analytics story.

It's also the natural lifecycle of a business that, if you're lucky, you get to a certain point where things are still going up and to the right, but you've gotten there largely on feel so far, and your investors are starting to ask questions about things like The Margin. Or the COGS. Or your customers show up in more and more places so it's tougher to keep track of all of them and make sure they're staying happy.

In short, you are much further from where you started and the navigation of the vehicle starts to become much more important, because all the landmarks you once knew have faded from view. This is where precise measurements about the direction and velocity of the vehicle start to take over from, or at least assist, your instincts. Having already identified about 500 questions I wanted to be able to answer with data (broadly) I still can't believe my luck, that I've wound up here.

It is not without its downsides, however...

---

One thing that I didn't take into consideration was the rather extreme sense of isolation that I feel in this new role. The team really has two missions, though from the outside they look like one. To borrow my own analogy, it's like catering a large event. The groceries have to be procured before they can be prepped and cooked and served to all the diners, but all the diner sees is the dish in front of the plate.

So, getting the data into The Warehouse is 1 job. Cleaning, prepping, cooking, and serving the data to the clientele is the second job. To do them sequentially would take a very long time, so while the procurement side has largely been handled thanks to my new data engineer teammate and Fivetran, it's a very lonely and rather stressful job to handle everything after that. I don't really have anyone to talk to about any of this part of the job and it's ... it sucks, frankly.

I'm not sure how this could've been different except if I'd considered that I was moving from an established team of 15 to a new team of 1, and that there would be little to no collaboration, which is a thing I really enjoy and thrive on. I'm feeling the burn for sure.

#business #life

Hello, me. This post is to jot down what I think I know about the AWS CUR bill at this point in time. There are entire companies built around helping other companies understand this bill, so this should be considered a primer.

There are several columns that matter in this giant bill, and many dozens that do not matter, and about a hundred or so that fall somewhere in between. The ones that I know about so far that really matter when you're trying to decode a bill AT SCALE are:

• line item unblended cost
• reservation effective cost
• public on demand cost
• savings plan effective cost

Line Item unblended cost

Basically everything that is not an EC2 instance that is covered by some sort of reservation or Saving Plan is going to show up in this column – storage, network costs, etc.

Confusingly, line items that have a Saving Plan applied to them will also show up in this column, making it necessary to check against the “Line Item Type” column to see if it falls under plain old “Usage”, in which case you're paying the on demand price. If the Line Item Type column says anything other than “Usage” then it's not what you're actually paying for that resource.

For resource usage line items that have a Reservation applied to them this column should be $0, but for resource usage line items that have a Savings Plan applied to them this column will contain the full, undiscounted price and you need to look elsewhere to find out what you're really paying.

Reservation Effective Cost

Any resource usage line items that have some sort of reservation applied to them will show up in this column. This column and the Unblended Cost column are, afaict, mutually exclusive. A line item can have a cost in one or the other, but not both. From the documentation:

The sum of both the upfront and hourly rate of your RI, averaged into an effective hourly rate.

These line items have a Line Item Type of “DiscountedUsage“

Public on demand cost

This column is mostly useful just for sanity checking as you do your math against the other columns. If you correctly sum up your Savings Plan Effective Costs, your Reservation Effective Costs, and your on demand Unblended Costs then you should have a number that is lower than the sum of this number.

Ideally it should be much lower, and I have not yet figured out if it is possible for the sum of all your discounted effective costs to be higher than this number.

If you incorrectly forecast your future usage and purchase discounts in excess of what you actually use, then you're potentially throwing away more money than what you'd be paying on demand. This is the nightmare scenario of course, and the Finops practitioner makes their money by avoiding this for their liege.

Savings Plan Effective Cost

The proportion of the Savings Plan monthly commitment amount (Upfront and recurring) that is allocated to each usage line.

This cost column is mutually exclusive with the Reservation Effective Cost, but not with the Unblended Cost. The Unblended column will contain the full, on-demand price for the resource in question but for those with a Line Item Type of “SavingsPlanCoveredUsage” the Saving Plan Effective Cost should be used.

I haven't tried this yet, but off the top of my head it would look something like

CASE 
	WHEN line_item_type = 'DiscountedUsage' THEN reservation_effective_cost
	WHEN line_item_type = 'SavingsPlanCoveredUsage' THEN savings_plan_effective_cost
	ELSE unblended_cost
END AS actual_cost,

CASE 
	WHEN line_item_type = 'DiscountedUsage' THEN 'Reservation Effective Cost'
	WHEN line_item_type = 'SavingsPlanCoveredUsage' THEN 'Savings Plan Effective Cost'
	WHEN line_item_type = 'Usage' then 'Unblended Cost'
	ELSE CONCAT('Unblended Cost: ', line_item_type)
END AS actual_cost_source

Good luck!

#business #finops

I never would've dreamed when I was 24 and hitting the road full time with a band that I would ultimately find career happiness not playing music but actually with a full time job working w a bunch of people I almost never see in person.

#life

So I'm back in the writing code game after 4 years off. I started a new position about 2 months ago as “director of data engineering” at Platform.sh. It only took ten years longer than originally planned but I have finally unlocked the “some kind of engineering mgmt at an actual tech company” badge.

I'll get into what I'm doing maybe later, but right now I want to leave a breadcrumb for my past self regarding looping over tasks in Prefect.

I'm just kind of tinkering with pulling info from APIs at this point, so the workflow looks basically like this for a couple different vendors:

• Pull a list of accounts
• For each of those accounts, pull a list of services
• For each of those services, pull service metrics for a range of days

This ends up fanning out to potentially thousands of APIs calls ultimately, so I'm doing a lot of for ... in: looping on Python. Given that I'm using Prefect though, I'm bumbling up the ladder of how to do these things in a more idiomatic way. Prefect has a native concept of mapping that reduces the amount of work you need to do in each of these steps as well as makes each iteration of each of these steps more atomic.

By example – my first, naive approach which was something like this:

• Pull a list of accounts() returns –> list of accounts, then
• Pull a list of services for each of those accounts(list of accounts) loops over and then returns –> list of services
• Pull a list of metrics(list of services) loops over enormous list of services and then returns –> ginormous list of metrics that is the thing we're actually interested in.

Obviously a failure in any one of the loops in the process means the whole thing derails, so I was doing a lot of work trying to handle that. Prefect has a smarter way in their native mapping API. Basically, instead of making the downstream tasks loop over a list of things, you write the downstream task assuming that it will handle 1 of the things in the list. Then instead of calling do_this_over_and_over_in_a_loop(list) you call something like do_one_iteration.map(list). Prefect then handles spawning a “child task” for each of the items in the list. This makes is possible to run the loops in parallel, and even to create entire parallel pipelines out of the original “pull the list of accounts” job. It also handles failures in any of the iterations atomically, and doesn't kill the entire loop.

The whole flow then looks like:

• Pull a list of accounts() returns –> list of accounts
• Pull a list of services for 1 account.map(list of accounts) returns –> list of services for 1 account, then you call map again on the task downstream of that
• Pull a list of metrics for 1 service.map(list of services) returns –> the stuff you're really interested in.

the .map() bit unrolls each of the lists and creates a child task for each element of the list, then calls the task on that 1 element.

I'm struggling right now with getting this all running on Platform.sh with Dask, but I'm pretty excited with the potential given my relative rustiness and that I'm working basically alone so far.

More later, <3 love you!!

#business #databases

I really enjoyed my time as a Solutions Architect, and wouldn't mind at all doing it again someday. The job of a Solutions Architect, or Sales Engineer, or “Presales” is be the technical person in the sales discussion on the part of the vendor who's trying to sell the technical $thing. The sales rep's main responsibility is to manage the sales cycle (a separate post entirely) and depending on how technical the product that your company sells, it's possible that knowing how to answer questions or sell the customer the right solution is entirely outside of the sales rep's capability.

This is where the Solutions Architect (SA) comes in.

The job of the SA is to have a very thorough conversation with the prospect about what they're trying to accomplish and why they decided to contact your company. It is your job to

• figure out what exactly the prospect needs from your company to solve their business problem and
• set their expectations for exactly how your solution will solve their problem

The second part is the most fundamental, because 100% of the time the prospect will have an idea of how your product solves their problem, but it will be from their perspective. In order to truly solve their problem they need to understand the mental model of how your company and its product works. This is the SA's real job.

All technologists are human and the products that they produce are therefore imperfect. It is impossible for the human prospect on the other side of the transaction not to view your solution as a magic bullet for their problem (I am doing this myself at this exact moment in my new job negotiating with a vendor). It is your job to bring the prospect down to earth and help them understand exactly what the experience will be when they come on board.

More later.

#business