Ignored By Dinosaurs main feed

DBT - incremental strategies for late arriving data

Mon, 10 Jul 2023 09:40:48 -0400

Hey there, just wanted to leave a signpost for you. My usecases lately have been something like -

Computing a BIG table, with a lot of math in it, over a LOT of rows of data, and then joining in other data to enrich the primary set. Specifically, this is container usage data, which I'm attempting to blend with our AWS bill to arrive at something like "cost per container" per time period.

I don't want to have to rebuild this table every day because most of the data is static once it shows...

Two different flavors of monthly metrics

Wed, 26 Apr 2023 04:47:02 -0400

Supposing a hypothetical organization that sold a product whose feature set and COGS closely followed a typical CSP like Amazon Web Services. That organization allows its customers to change products at will but must manually invoice a significant percentage of those sellables, therefore it needs a robust system to track changes to those sellables and ensure that they are properly charged at each turn of the billing cycle.

I’m picturing reporting format that reports on 2 different types...

FinOpsLand - Reservations

Sun, 19 Feb 2023 03:27:36 -0500

First day of my job at my current employer (almost 7 years ago now) I crack open the company handbook to start onboarding. I remember it saying something like “Reservations are the lifeblood of this company” and thinking “wow, what does that mean?”

I’d had a job prior to this one that had some Stuff on AWS so I was familiar with the concept - something something pay upfront get reduced prices - but it as far from the lifeblood of the company. It was something the IT head took...

FinOpsLand - Reservations

Sun, 19 Feb 2023 03:27:36 -0500

SQL is easy, Data is hard

Sat, 18 Feb 2023 02:27:55 -0500

I've been kicking around this thought for a year or so now - to the outsider a career in data looks like a technical path. The data practitioner learns SQL, how to query data stored in a database somewhere using SQL, and if you know enough SQL you can answer any question whose artifacts are stored in that database somewhere.

The reality is that SQL is the very last mile. SQL is code, and so it looks to the non-practitioner like the act of creation, like code written in any imperative...

Dispatch from FinOpsLand

Sat, 11 Feb 2023 03:55:03 -0500

I had a really organized map of things in my head I’d like to tell my younger self about FinOps last night. This morning it is gone. Let this be a lesson to me - jot some notes down. It was a primer course, from the point of view of a data person who was placed in charge of a FinOps practice - how to think about FinOps, what data are you going to need, what do the terms and costs mean, etc.

So what is FinOps?

Well, it’s the driest sounding topic that I’ve ever found incredibly...

Who’s hiring all these B players anyway?

Sat, 14 Jan 2023 04:10:55 -0500

I read an article earlier this week about lessons learned between $5MM and $100MM in ARR. To the layperson - this means growing a small company into a larger company, as measured by its yearly revenue.

One of the points in the article (maybe more, I don’t remember) was about hiring, and it referenced the old adage

A players hire other A players. B players hire C players…

While this sounds like one of those BS businessisms that some capitalist dude came up with, I absolutely...

Pensieve

Thu, 06 Oct 2022 10:20:34 -0400

Been reading the Harry Potter books for a few years with the family at night, and in the middle gets introduced this thing of Dumbledore's called the "Pensieve", which is like a bowl into which Dumbledore can put his memories so he doesn't have to keep them all in his head.

I just realized I've been doing this, sort of, for the last year or so. I'm full on manager now, all I do is phones calls for the first half of any given day. I started taking notes with pen and paper sometime last...

My kids and social media

Thu, 22 Sep 2022 03:34:22 -0400

I'm working through some thoughts in my head about social media, as I've been doing since founding this blog well over a decade ago. Back then I thought it was going to be a savior of democracy in oppressed societies around the world, and we see how that's turned out.

Lately it's an issue closer to home. My kids are creators. At some point years back they got inspired by Captain Underpants and started making their own comic books. We have bookshelves full of 8 inch sketch pads from AC...

What is a "data product"?

Wed, 21 Sep 2022 12:20:35 -0400

"Run your data team like it's a product team" was a common refrain at the DBT conference for the last two years. What does that mean? I am still figuring that out, but I have had an aha in the last few weeks about what a "data product" is, exactly.

Your standard software development process requires two main components to deliver the Things - a software developer and her wits. She takes those wits and expresses them into a text editor, and thereby makes something from truly...