Testing is a critical activity in all software projects, but one that is often neglected in data pipelines. The complexities introduced by the inherent statefulness of the problem domain and the interdependencies between systems contribute to make pipeline testing difficult to manage. To make this endeavor more manageable Abe Gong and James Campbell have created Great Expectations. In this episode they discuss how you can use the project to create tests in the exploratory phase of building a pipeline and leverage those to monitor your systems in production. They also discussed how Great Expectations works, the difficulties associated with pipeline testing and managing associated technical debt, and their future plans for the project.
With libraries such as Tensorflow, PyTorch, scikit-learn, and MXNet being released it is easier than ever to start a deep learning project. Unfortunately, it is still difficult to manage scaling and reproduction of training for these projects. Mourad Mourafiq built Polyaxon on top of Kubernetes to address this shortcoming. In this episode he shares his reasons for starting the project, how it works, and how you can start using it today.
Whether it is intentional or accidental, every piece of software has an existing architecture. In this episode Neal Ford discusses the role of a software architect, methods for improving the design of your projects, pitfalls to avoid, and provides some resources for continuing to learn about how to design and build successful systems.
Your backups are running every day, right? Are you sure? What about that daily report job? We all have scripts that need to be run on a periodic basis and it is easy to forget about them, assuming that they are working properly. Sometimes they fail and in order to know when that happens you need a tool that will let you know so that you can find and fix the problem. Pēteris Caune wrote Healthchecks to be that tool and made it available both as an open source project and a hosted version. In this episode he discusses his motivation for starting the project, the lessons he has learned while managing the hosting for it, and how you can start using it today.
The importance of testing your software is widely talked about and well understood. What is not as often discussed is the different types of testing, and how end-to-end tests can benefit your team to ensure proper functioning of your application when it gets released to production. This week Luciano Puccio shares the work that he has done on Golem, a framework for building and executing an automation suite to exercise the entire system from the perspective of the user. He discusses his reasons for creating the project, how he things about testing, and where he plans on taking Golem in the future. Give it a listen and then take it for a test drive.
Do you know what is happening in your production systems right now? If you have a comprehensive metrics platform then the answer is yes. If your answer is no, then this episode is for you. Jason Dixon and Dan Cech, core maintainers of the Graphite project, talk about how graphite is architected to capture your time series data and give you the ability to use it for answering questions. They cover the challenges that have been faced in evolving the project, the strengths that have let it stand the tests of time, and the features that will be coming in future releases.
Email has long been the most commonly used means of communication on the internet. This week Antoine Nguyen talks about his work on the Modoboa project to make hosting your own mail server easier to manage. He discusses how the project got started, the tools that it ties together, and how he used Django to build a webmail and admin interface to make it more approachable.
Server administration is a complex endeavor, but there are some tools that can make life easier. If you are running your workload in a cloud environment then cloud-init is here to help. This week Scott Moser explains what cloud-init is, how it works, and how it became the de-facto tool for configuring your Linux servers at boot.
Server administration is an activity that often happens in an isolated context in a terminal. ChatOps is a way of bringing that work into a shared environment and unlocking more collaboration. This week Jacob Tomlinson talks about the work he has done on opsdroid, a new bot framework targeted at tying together the various services and environments that modern production systems rely on.
Routers and switches are the stitches in the invisible fabric of the internet which we all rely on. Managing that hardware has traditionally been a very manual process, but the NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is helping to change that. This week David Barroso and Mircea Ulinic explain how Python is being used to make sure that you can watch those cat videos.