497 Instructions for AI: Get the Data
Here’s a quick thought that’s been blowing my mind lately: the barrier to testing data patterns has almost completely disappeared.
The New Way to Build Data Pipelines
What you can do now is say: “Look, I’m going to give you the instructions AI of what I want you to do. Get the data from here, put it here.”
It’s going to ultimately land wherever you need it, right?
You can then tell the AI to build the same pattern in all the different tools and run them all together. See what works best for your scenario:
- Loading tables from SQL databases? What is the most efficient way to go?
- Pipeline to a data warehouse? Boom, write it.
- Pipeline to a lakehouse and then through notebooks? Done.
The concepts are the same across all these tools. You’re able to build five, six, seven different patterns of what works.
Testing Just Got Ridiculously Fast
And here’s the kicker: if I can build these things within a couple minutes, great. I’ll test them all.
The barrier to testing these things almost goes to zero. You’re not spending days setting up infrastructure just to see if an approach works. You tell the AI what you need, it builds it, you test it. Move on to the next pattern.
This is how we should be working now. Less time in setup, more time validating what actually performs best for our data.
Watch the Full Video
Want to dive deeper into this topic? Check out the full conversation: