Published in Towards Data Science·May 2, 2022Running dbt using Gitlab CI/CDThe easiest way to deploy, run and schedule dbt for free — My first implementation of dbt was while running the Data Team at Sirena and I immediately became a huge fan of it. We used it for running all of our transformations and creating our models in our data warehouse (Snowflake). It was literally a game-changer to be able to apply…Dbt6 min readDbt6 min read
Apr 30, 2022A Dockerized dbt WorkflowHow to setup dbt using docker containers with an RPC server for Dagster — Lately, I have been playing around quite a bit with Dagster, trying to understand how good of a replacement it would be for Airflow. I might write a more comprehensive post about it, but one interesting thing I came across in my testing was the integration with dbt. There are…Docker5 min readDocker5 min read
Published in Towards Data Science·Nov 30, 2021Abstracting Data Loading with Airflow DAG FactoriesCreating an abstraction layer for improved scalability and usability for loading Google Sheets data with Airflow — This topic is not new. I’ve seen quite a few articles about generating different levels of abstractions for Airflow DAGs. For example, this article explains how to generate DAGs programmatically, and this library allows you to generate DAGs using YAML. However, both of those abstractions are still for data practitioners…Airflow7 min readAirflow7 min read
Published in Towards Data Science·Oct 17, 2021Data lake in S3 from MongoDBUsing Python to upload MongoDB data to AWS S3 to build a data lake — I was extremely lucky to be able to write my bachelor’s thesis combining the topics I am most passionate about; economics (in the form of auction theory), software development, and data analysis. Even though my thesis was an empirical research about auctions and bidding strategies, it involved gathering data using…Data Lake8 min readData Lake8 min read
Published in Towards Data Science·Jul 12, 2021How we use Airflow for SQL alerts and Slack notificationsUsing Airflow to get notified over Slack when things don’t go as planned while loading data — At Sirena, we use Apache Airflow for running all of our data orchestration. Airflow is an open-source tool that was created by Airbnb to create and schedule data workflows. In essence, Airflow is an orchestrator that runs tasks on given frequencies while also handling backfilling, task dependencies and so much…Airflow5 min readAirflow5 min read
Published in Towards Data Science·Feb 21, 2021How to Build a Custom Estimator for scikit-learnImplementing a custom ensemble model with under-sampling for imbalanced data — This post will show you how to implement your own model and make it compliant with scikit-learn’s API. The final result will be a model that can not only be fitted and used for predictions but also be used in combination with other scikit-learn tools like grid search and pipelines. Introduction …Scikit Learn5 min readScikit Learn5 min read
Feb 12, 2021Bayesian Weight LossIn a previous post, I discussed the concept of linear regressions in the realm of Bayesian statistics. I will do something similar in this post but we’ll change the topic a little bit. We’ll go from finance to weight loss. Introduction I have recently decided to start being more healthy in…Bayesian Statistics5 min readBayesian Statistics5 min read
Published in Towards Data Science·Feb 9, 2021Crazy Simple Anomaly Detection for Customer SuccessHow to detect changes in usage, react in time and make your customers happy — The topic of anomaly detection is fascinating. There is a vast number of methods that can be used, from simple statistics to more complex unsupervised learning methods. Additionally, the impact of anomaly detection is huge. …Anomaly Detection6 min readAnomaly Detection6 min read
Published in The Startup·Feb 8, 2021Bayesian CAPM Beta EstimationContinuing with my (mostly) healthy obsession with Bayesian statistics (see my previous article), in this article, I’ll use a linear regression model from a Bayesian perspective. To demonstrate how linear regressions in the context of Bayesian statistics work, I will use the CAPM to estimate stock’s Betas. Introduction It’s not fundamentally…Statistics7 min readStatistics7 min read
Published in The Startup·Nov 17, 2020Bayesian A/B Testing WhatsApp MessagesIn this article, I will walk you through the process of performing an A/B testing analysis from the perspective of Bayesian statistics to determine the best WhatsApp message template to send to customers. To be fair, there is no difference if this is about a website landing page, a product…Statistics6 min readStatistics6 min read
