Enforcing Style in a Python Project
A linter and a styler can help you to write cleaner and more consistent code. In this post we’ll look at how to set up both for a Python project.
Python-bloggers
Data science news and tutorials - contributed by Python bloggers
Articles by Python - datawookie
Historical Weather Data
I’m building a model which requires historical weather data from a selection of locations in South Africa. In this post I demonstrate the process of acquiring the data and doing some simple processing.
Persisting Data with Pickle & S3
I occasionally write scripts where I need to persist some information between runs. These scripts are often wrapped in a Docker image and deployed on Amazon ECS. This means that there is no persistent storage. I could use a database, but this would be overkill for the volume of data ...
Firing Up Firestore
I’ve just started collaborating on a new project, Votela, with Luke. We’re going to be using Firestore for stashing our data. I’ve never worked with Firestore before, so one of my first tasks was just figuring out how to get connected and how to shift some data ...
Scrapy with a Rotating Tor Proxy
This post shows an approach to using a rotating Tor proxy with Scrapy.
I’m using the scrapy-rotating-proxies download middleware package to rotate through a set of proxies, ensuring that my requests are originating from a selection of IP addresses. However, I need to have those IP addresses evolve over ... [...Read more...]
Selenium Crawler #3: Docker Compose
In two previous posts we’ve looked at how to set up a simple scraper which uses Selenium in Docker, communicating via the host network and bridge network. Both of those setups have involved launching separate containers for the scraper and Selenium. In this post we’ll see how to ...
Selenium Crawler #2: Docker Bridge Network
In the previous post we set up a scraper template which used Selenium on Docker via the host network. Now we’re going to do essentially the same thing but using a bridge network.
Default Network We’ll start by using Docker’s default bridge network.
docker network ls NETWORK ...
@pyconza (2018): Data Science and Bayes with Python
I’ve just returned from PyConZA (2018), held at the Birchwood Hotel in Boksburg North (Johannesburg) on 11-12 October. A great conference with a super selection of talks and great catering.
Obviously when the PyCon call for papers came out I was feeling ambitious because I submitted a Workshop and a ... [...Read more...]
Mastering Python Scientific Computing
I was asked to review “Mastering Python Scientific Computing” (authored by Hemant Kumar Mehta and published in 2015 by Packt Publishing). I was disappointed by the book. The title lead me to believe that it would help me to achiev...
Beautiful Data
I’ve just finished reading Beautiful Data (published by O’Reilly in 2009), a collection of essays edited by Toby Segaran and Jeff Hammerbacher. The 20 essays from 39 contributors address a diverse array of topics relating to data a...
Copyright © 2024 | MH Corporate basic by MH Themes