Testing is a critical activity in all software projects, but one that is often neglected in data pipelines. The complexities introduced by the inherent statefulness of the problem domain and the interdependencies between systems contribute to make pipeline testing difficult to manage. To make this endeavor more manageable Abe Gong and James Campbell have created Great Expectations. In this episode they discuss how you can use the project to create tests in the exploratory phase of building a pipeline and leverage those to monitor your systems in production. They also discussed how Great Expectations works, the difficulties associated with pipeline testing and managing associated technical debt, and their future plans for the project.
We take it for granted every day, but creating and displaying vivid colors in our digital media is a complicated and often difficult process. There are different ways to represent color, the ways in which they are displayed can cause them to look different, and translating between systems can cause losses of information. To simplify the process of working with color information in code Thomas Mansencal wrote the Colour project. In this episode we discuss his motiviation for creating and sharing his library, how it works to translate and manage color representations, and how it can be used in your projects.
Many developers enter the market from backgrounds that don’t involve a computer science degree, which can lead to blind spots of how to approach certain types of problems. Gary Bernhardt produces screen casts and articles that aim to teach these principles with code to make them approachable and easy to understand. In this episode Gary discusses his views on the state of software education, both in academia and bootcamps, the theoretical concepts that he finds most useful in his work, and some thoughts on how to build better software.
With libraries such as Tensorflow, PyTorch, scikit-learn, and MXNet being released it is easier than ever to start a deep learning project. Unfortunately, it is still difficult to manage scaling and reproduction of training for these projects. Mourad Mourafiq built Polyaxon on top of Kubernetes to address this shortcoming. In this episode he shares his reasons for starting the project, how it works, and how you can start using it today.
One of the biggest issues facing us is the availability of sustainable energy sources. As individuals and energy consumers it is often difficult to understand how we can make informed choices about energy use to reduce our impact on the environment. Electricity Map is a project that provides up to date and historical information about the balance of how the energy we are using is being produced. In this episode Olivier Corradi discusses his motivation for creating Electricity Map, how it is built, and his goals for the project and his other work at Tomorrow Co.
Email is one of the oldest methods of communication that is still in use on the internet today. Despite many attempts at building a replacement and predictions of its demise we are sending more email now than ever. Recognizing that the venerable inbox is still an important repository of information, Christine Spang co-founded Nylas to integrate your mail with the rest of your tools, rather than just replacing it. In this episode Christine discusses how Nylas is built, how it is being used, and how she has helped to grow a successful business with a strong focus on diversity and inclusion.