To create interesting data driven articles or landing pages with programmatic SEO, you need good datasets. Unfortunately, this part is too technical for most website owners due to the coding knowledge involved.
In this guide, we’ve covered all the interesting data sources that help you gather either paid or free datasets for your programmatic SEO projects without requiring any programming knowledge or coding scripts.
We also go one step further to walk you through how to plug quality data sets into your programmatic SEO tool. We will show you how it’s done with our software - SEOmatic to steer clear of low-quality articles.
Where to Find Programmatic SEO Datasets
Once you’ve settled on the keywords to target, you can acquire data for your project with any of these three methods:
- Data Scraping
- Data APIs or public datasets
- AI data (Open AI + Web Browsing)
1. Data Scraping
Imagine that you are building a relatively new site that provides a weather report for specific periods, say, three to six months. To create this website, you need to gather historical data of the weather conditions in each city over the stipulated time.
There are two ways to go about this:
- Manually study and pull out data from a weather website (time consuming) OR
- Automatically pull out the data by scraping it.
Data scraping is the method of pulling out relevant dataset or information from a website into a spreadsheet or other file formats.
You can get your own personal data by scraping websites (If they have historical data) or via a third-party website.
Regardless, there are two ways you can scrape data from any website:
- Scrape data yourself with web scraping software or an AI scraper.
- Outsource data scraping to freelancers on Upwork or Fiverr.
If you decide to save cost and spare few hours to gather data sets yourself, here are the paths you can take.
Scrape Data Yourself With Web Scraping Tools
These are web applications that help extract bulk information from a website. This information could vary from product data, comments, reviews, or in some cases, images.
After extraction, the data are structured and compiled into several exportable file formats: CSV, Excel, SQL database, Doc, or HTML.
In our opinion, Octoparse is the best no-code tool that allows marketers, SEOs, and E-commerce store owners to gather CSV datasets easily.
With Octoparse, you enjoy the benefit of scheduling extraction of quality data at regular intervals to spend less time on data collection and more on data analysis.
Here are some pros and cons of using Octoparse:
The Pros
- Easily extract and organize unstructured data with zero coding knowledge
- Scheduled scraping helps automate data collection
- Easily extract different type of data and export into multiple files.
- Knowledge hub contains resources and tutorials to get familiar with the software.
The Cons
- Limited crawl with the free version
- Difficult to scrape popular websites
- Has a long learning curve - Have to sacrifice 1-3 hours to learn how it works
- Runs on a local server, hence crawling is slow when extracting large datasets.
Scrape Data Yourself With AI-powered Scrapers
AI tools help you automate scraping without using any coding script: PHP, Python or JavaScript. Leveraging machine learning algorithms, these tools browse through a website to automatically scrape its data.
It requires little to no effort from your end, and you can download data in your preferred format.
Here are some of the best AI scrapers out there:
Kadoa - Uses generative AI to extract interesting data e.g text, images, and videos
Webscrape AI - For collecting public datasets from sites that don't require authentication or login.
ScrapeIt - Best for automated scraping at regular intervals - Monthly, daily or weekly. Perfect for niche sites: e-commerce, real estate, sales and marketing.
Import.io - Point and click data extraction software for enterprise companies.
But if you don’t have time to scrape the data yourself with any of these options, then you can outsource the scraping to professionals on Fiverr or Upwork.
Outsource Data Scraping to Freelancers on Upwork or Fiverr
You will find lots of data scraping experts on freelance sites such as Fiverr and Upwork. Most of these scrapers use Python, JavaScript, or PHP for data scraping.
The fee you will pay depends on the complexity of your task. You can browse through their profiles to decide the best pick for your project. For instance, on Fiverr, most data mining gigs from top-rated accounts cost between $50-$170.
Tips on How to Browse for The Best Data Scrapers on Fiverr
- Use top-rated sellers and avoid Level 1 or 2 sellers as the result could be a hit or miss.
- Read reviews to evaluate the quality of their service
- Request for samples of previous sites they’ve scraped - Might be lucky to find out they’ve previously scrapped the site you have in mind (especially if it’s a well-known website)
- Present your data scraper with a brief to help match your expectations.
- Explain what data you need: e.g I want the weather report for Paris in April.
- Outline what each row and column will include: Weather in Celcius, Fahrenheit, weather condition for each day - Sunny or Cloudy.
- Provide a sample to serve as an example
- Indicate preferred file format: CSV, SQL database or JSON.
Bonus tip: For custom scrape requests, ask for the code and the deployment of the code to automatically scrape data for you on a regular basis. Or a way to run the code at least with the detailed instructions.
Public Databases and Data APIs
Public Databases
There are platforms where you can find public datasets for free for any kind of niche. Here are a few online repositories to source for free data:
Data.gov: An open data source with 255,101 free datasets made available by the US government to internet users who intend to conduct research. You can find many Open Data from Governments for every country.
Datasets subreddit: r/datasets is a subreddit where data consumers find, share and discuss datasets for their projects. a community with 175k members. Post the dataset you need, and someone will reach out to you.
Socrata: Designed for developers and anyone who needs data to build a project. Their open data network includes data sets from ou will Find opend data from the federal government and NGO's around the world
Kaggle: Kaggle datasets available to data scientists and machine learning experts. Kaggle is the #1 source of reliable data for programmatic SEO projects.
Google dataset: You can find datasets on Google Sheets with this search term: site:docs.google.com/spreadsheets "subject". The problem with Google public data sets is that it’s difficult to find what you're looking for, and the output might be unreliable.
FiveThirtyEight: FiveThirtyEight datasets makes data collection easy. All datasets have undergone data cleaning, hence making data visualization and analysis easy.
Private APIs
These are online data platforms that grants access to their databases for a fee:
Rapid API: A marketplace with interesting datasets and APIs that cuts across our daily life: Sports, marketing, finance and others. Simply plug them and implement into your application.
Datahub: A SaaS platform that acts as a data storage house. Find the datasets you want with a quick search.
The Companies API: Used to find data points around 52 million companies and their employees. You can sign-up for free (no credit card required) and get 500 instant credits.
Quandl: A Nasdaq-owned data storage platform that stores data from the financial and economic world, and delivered in different formats; Python, Excel
Data.world: Access to 128,165 datasets. A marketplace that brings data producers and consumers together.
Instant Data With ChatGPT (Open AI + Web browsing)
The first two dataset sourcing methods have one thing in common: You have to import data from external sources and arrange target variables into rows and columns assuming you’re using CSV datasets.
But what if you can get the dataset within the programmatic SEO tool you’re using?
This will save you the stress of hiring someone, paying for tools or wasting hours going from one database to the other.
To help you find instant datasets for your programmatic SEO project, we have integrated useful datasets for programmatic SEO and Open AI’s web browsing feature into our tool - SEOmatic. With this integration, all you have to do is to log into our software, and search for the dataset you want (like searching for information on Google), and you will instantly gain access to recent data on your chosen topic.
You may choose to export the datasets or use them directly within your content.
To see how this method works, click here to sign up for SEOmatic’s free trial.
You’ve Found My Datasets! What Next?
How to Import Your Dataset Into SEOmatic
Once you’ve gathered your datasets, you will need to upload it into your programmatic SEO tool to use it for your project.
On SEOmatic, there are two ways you can do it:
- Manual upload: Input your dataset by uploading a .csv file
- Google Sheet Integration: Sync your Google Sheet with SEOmatic
And here’s a step-by-step guide for importing datasets:
Step 1: Create a Project, and add a description
Step 2: Connect your favorite CMS (WordPress, Webflow, etc.)
Step 3: On the next page, hover to “Data” and click “Import data”
Step 4: Select the “CSV import” option, and click “Import”
Good News! Now you have your datasets, and you start creating content.
Click here to see the other features SEOmatic offers, and see why it’s the best programmatic SEO tool right now!
Conclusion
When you're searching for datasets, remember that the quality of your datasets is just as crucial as the amount you've got. If you're getting your data scraped from elsewhere, don't forget to give it a good check-over and ask for tweaks if you need to.
But, guess what? With our tool, SEOmatic, you don't have to worry about all that. We've built a special feature right into SEOmatic that lets you hunt for datasets right from inside the tool. No need for outsourcing, no need for double-checking, it's all right there at your fingertips.
This isn't just about saving time (though it does that, too!). It's about saving you the headache of juggling between tools and services, and yes, it's about saving you some hard-earned cash, too.
Everything you need for programmatic SEO datasets is nestled right inside SEOmatic. So why not give it a try? We're offering a 7-day free trial right here.
