Google Search Console allows us to easily download the list of URLs that bring the most clicks to our page. Now, what do we do with this data? As we have been seeing for a few months now, at puntorojo we like to extract as many insights as possible from our data. An insight is any information that allows us to make a decision.

Let’s start!

1. Map the site to be analyzed: This time we are going to use a mix of tools to be able to see which theme is central to the blogs from Ahrefs, Semrush, and Moz. To achieve this, we have to start mapping the blogs with some crawler (in this example we are going to use ScreamingFrog).
2. Filter by URL type: Once we have the list of URLs, we are going to filter everything that is a page to make sure that no other type of content is lifted (e.g. image URLs).
3. Filter by subdirectory: The next step is to filter the URLs to show only those that are from the blog. A simple way to do this is with the top right filter.                                Advanced tip: you can filter by regular expressions and not only text strings.
4. Pull up URLs that return 200: Now we are going to sort the results by response code. Since we are going to work with URL slugs, we don’t want to pollute our analysis with old URLs.
   For example, there is a slug that changed from “how-to-build-links-with-guest-articles” to “guest-blogging”. This indicates an optimization due to a strategy (or change of strategy) since they went from focusing on the keyword guest articles to the keyword guest blogging. As we are interested in the current strategy, we are not going to look at the discarded URL but at the final URL after the redirect.
5. Save the URLs: In Google Sheets we are going to dump all URLs that return 200.
6. Filter by URLs that redirect: Since we also want to know the URLs that were modified, inside the Response Codes tab, we filter by the blog pages that redirect.
7. Save the redirection URLs: As in step 5, we are going to save the URLs that interest us in the same Google Sheets, but now we are going to look at the Redirect URL column since this is the one that indicates the page that is displayed after the redirection is executed.
8. Remove repeated URLs: Once in Google Sheets, we will remove the URLs that are repeated using the late functionality of Remove duplicates in the Data menu.

9. Remove irrelevant URLs: There are likely to be URLs that we don’t want in the listing such as blog home, paginations, author URLs, posts in other languages, and URLs that redirect and now point to external sites, etc. Clean these URLs manually or via a Sheets function if you find a repeating pattern.

Note: In this case, we are going to remove category URLs since we are interested in building our categorization and not necessarily sticking to the one selected by the blog creators. For example, the category “technical SEO” is too broad and we will want to segment it as “page speed, machine learning, IT, etc.).

10. Repeat with other benchmarks: Once our list is ready, we are going to do the same with all our competitors. In the Sheets spreadsheet, we are going to add a column that shows the name of the site we are benchmarking so we can then plot it in different series.

11. Write categorization UDF: To automatically categorize the information we extracted, we are going to use the Google Sheets script editor to create a UDF (user-defined function).

12. Select categorization keywords: To get the most out of each slug, we have to put together different lists (technically arrays) with keywords that determine without grayscale the content of a slug. Once the arrays are completed, run a loop through each of them checking if each word appears in the URLs being analyzed. If it appears, the name of the category is written in a cell.

Example script here:

categorize function (slug) {
  //Write an array for each category and fill it with keywords 
var keywords_tutorials = [
    "tutorial", "how-to", "guide", "step-by-step",
  ];
  
  var keywords_links = [
    "link", "anchor",
  ];
  
  var keywords_content = [
    "content", "text", "copywriting", "post", "blog",
  ];
  
  var keywords_technical = [
    "core", "rankbrain", "penguin", "panda", "hummingbird", "diversity", "unnamed", "medic",
    "maccabees", "fred", "possum", "mobilegeddon", "pigeon", "everflux", "pagerank", "page-rank",
    "sandbox", "penalty", "penalties", "google-bot", "googlebot", "mobile", "phone", "301", "200",
    "404", "403", "503", "302", "tag", "robots", "html", "javascript", "php", "keyword", "key-phrase",
    "domain", "directory", "htaccess", "redirect", "tool",
  ];
 
 
  var output = ""; 
  
  //Each array must have a loop that returns the category name
  for (i=0; i <= keywords_tutoriales.length-1; i++)
  {if (slug.toLowerCase().indexOf(keywords_tutorials[i]) >= 0){output += "tutorials" + ", ";}}
  
  for (i=0; i <= keywords_links.length-1; i++)
  {if (slug.toLowerCase().indexOf(keywords_links[i]) >= 0){output += "links" + ", ";}}
  
  for (i=0; i<= keywords_content.length-1;i++)
  {if (slug.toLowerCase().indexOf(keywords_content[i]) >= 0){output += "content" +", ";}}
  
  for (i=0; i<= keywords_tecnicas.length-1;i++)
  {if (slug.toLowerCase().indexOf(keywords_technical[i]) >= 0){output += "technical" + ", ";}}
  
  //If it does not find a category, it categorizes it as 'other'.
  if (!output){
    output = "other";
    } else {
      //remove final ', '  
      output = output.substring(0,output.length - 2);
      output = output.split(", ");
      
      //eliminate duplicates
      var output_unique = [];
      for (o=0; o <= output.length-1;o++){
        if(output_unique.indexOf(output[o]) == -1) {
          output_unique.push(output[o]);
        }
      }
      //sort the array alphabetically
      output = output_unique.sort();
      output = output.toString();
    }
  return output;
}

13. Writing the function: Now that we have everything categorized we have to write the function in the spreadsheet and see the output.

Pro Tips:

– Don’t copy/paste for your entire list of slugs because if you do a lot the Google server running our script may block your requests for a couple of minutes since running this function has a cost for Google.

– You should copy and paste about 500 rows, wait a few seconds and run 500 more rows… until you finish all your URLs.

– Be sure to copy and paste the result of the script as a value so that the function does not run again each time you load the spreadsheet.

– Do not copy and paste as a value until you have finished running the function on all the rows.

14. Sort outputs: Next step, we will separate the data into columns since it is likely that some entries have more than one category.

Note: In this example, we hide the column with the slugs to make it easier to work with the data.

15. List the benchmarked sites: Using the UNIQUE function, we list all the competing sites we are considering for the report.

16. Count total values per site: In order to automate the spreadsheet a bit, let’s count how many entries were contemplated for each site using the following formula:

=COUNTIF(A:A,G2)

17. Count unique values: Making use of the total number (and considering that the first row is the header), we put together the following formula to count how many times each term appears for each competing site:

=COUNTIF($C2:$F211,”=”&I$1)

Note: Don’t forget to adjust the range contemplated in the count for each client’s rows since rows 2 to 211 correspond only to Ahrefs.

18. Drawing percentages: We are almost there! But if we take into account that Ahrefs has 210 blog posts, we can’t really compare it with SEMrush which has 7 times more (and Moz even more). So let’s update our formula to show the percentage of the total instead of the total amount of each category. The formula would now look like this:

=COUNTIF($C2:$F211,”=”&I$1)/$H2

Note: Be sure to update only column I and then drag I2, I3 and I4 to M2, M3 and M4, respectively.

19. Format as a percentage: To do things neatly, let’s convert all the values in I2:M4 to percentages by selecting the range of cells and then clicking the percentage format button.

Note: Keep in mind that the percentages can add up to more than 100% since a note can have more than one subject (for example, contents and tutorials).

20. Set the data: To simplify things, we copy all the percentages and paste them as a value. Next, we delete the totals column (column H, in our case).

21. Graphing: The moment of truth has arrived! We select our entire dashboard and then generate the graph.

Final result

In conclusion, we draw the following insights…

• SEMrush has the most number of tutorials
• Ahrefs is the one that talks the most about links (obviously)
• SEMrush is the blog most oriented to content generation
• SEMrush is the blog with the most technical content
• Moz is the blog with the most varied topics among the three.

Note: If we want more insights, we should add more categories and filters inside our script to reduce the number of blog posts that fall into others.

Well, this was a bit longer tutorial, but we didn’t want to stop sharing it with you. We hope you can find SEO value in categorizing URLs (either your own or those of your competitors). As always, if you have any questions, leave us a comment below.

Until next time and good rankings!

La entrada How to categorize your URLs, and your competitors’ aparece primero en SEO Blog Punto Rojo.

Many algorithms sift through the information for you and return a complete report with insights into what was analyzed. To put it bluntly, Data Science is the use of technology to analyze data. In this post, we are going to see how you can start extracting data from your keywords using Python.

Before we start, let’s define some terms:

• Data: The information collected
• Analytics: The data analysis
• Insights: What you learn after analyzing the data

In order to bring some added value to the organic SEO industry, I’m going to stay away from Google Analytics and similar tools in this post as there are thousands of posts written about the topic by people more qualified than me.

We are going to use the following tools:

• Google Analytics (I know I said I won’t use it, but we are going to extract the data from here in this example)
• Google Chrome Console
• Notepad (or equivalent)
• Python

How to extract search term data from Google Analytics

1. Open Google Analytics in Google Chrome
2. In Google Analytics, go to Acquisition > Search Console > Queries
3. Filter view to show 5000 rows
4. Open Google Chrome inspector > “Elements” tab
5. Navigate to one of the rows that display Google Search Console information to make sure it is in the inspector cache. If this step is not done correctly, keywords may not appear when running the script.
6. Go to the “console” tab
7. Ctrl + L to clean the console
8. Paste the following script and press ENTER*.
   for(var kwds=document.getElementsByClassName(“_GApu”),clicks=document.getElementsByClassName(“_GAtjb”),total=kwds.length,content='<table align=”center”><tr><th>Keywords</th><th>Clicks</th><th>Impresiones</th><th>CTR</th><th>Ranking</th></tr>’,i=0;i<total;i++){var current_kwd=kwds[i].innerHTML,current_clicks=clicks[i].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_impresiones=clicks[i+1].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_ctr=clicks[i+2].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_posicion_media=clicks[i+3].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim();i+=3;var row=”<td>”+current_kwd+'</td><td align=”center”>’+current_clicks+'</td><td align=”center”>’+current_impresiones+'</td><td align=”center”>’+current_ctr+'</td><td align=”center”>’+current_posicion_media+”</td>”;content+=”<tr style=\”font-family: ‘Open Sans’\”>”+row+”</tr>”}with(output='<html><head><title>Punto Rojo Tools – Google Analytics Keyword Extractor</title><link rel=”stylesheet” type=”text/css” href=”https://fonts.googleapis.com/css?family=Open+Sans” /><link rel=”stylesheet” type=”text/css” href=”//fonts.googleapis.com/css?family=Open+Sans” /></head><body>’,output='<table align=”center” width=”50%”><tr><td align=”left”><img src=”https://puntorojo.com/blog/wp-content/themes/punto_rojo/img/logo_mini.png” width=”200px”></td><td align=”right”><h1 style=”font-family: \’Open Sans\'”>Google Analytics Keyword Extractor</h1></td></tr></table><br><br>’,output+=content,output+=”</body></html>”,window.open())document.write(output),document.close();
9. If you want to see directly the information without formatting to export it to a CSV, paste the following code:
   for(var kwds=document.getElementsByClassName(“_GApu”),clicks=document.getElementsByClassName(“_GAtjb”),total=kwds.length,content=”Keywords;Clicks;Impresiones;CTR;Ranking<br>”,i=0;i<total;i++){var current_kwd=kwds[i].innerHTML,current_clicks=clicks[i].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_impresiones=clicks[i+1].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_ctr=clicks[i+2].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim(),current_posicion_media=clicks[i+3].innerText.split(“(“)[0].replace(“%”,””).replace(“.”,””).replace(“,”,”.”).trim();i+=3;var row=current_kwd+”;”+current_clicks+”;”+current_impresiones+”;”+current_ctr+”;”+current_posicion_media+”<br>”;content+=row}var output=”<html>”;with(output+=content,output+=”</html>”,window.open())document.write(output),document.close();
10. Copy keywords, paste them into Notepad
11. Add ‘Keywords’ to the first row of the txt (without quotation marks and with capital K)
12. Save as ‘keywords.csv’
13. If you want to save the script for future reference, follow the steps below:
    1. In the Google Chrome inspector, go to the ‘Sources’ tab.
    2. In the menu on the left, click on the ‘Snippets’ tab (if it does not appear, click on the little arrow on the right).
    3. Click on ‘+ New Snippet’.
    4. Name the script ‘Google Analytics – Keyword Extractor’ or similar.
    5. Paste the code in the window next to it
    6. Ctrl + S to save (or its MAC equivalent)
    7. That’s it! Each time you want to run it, double click on the snippet name in the left window of Sources > Snippets

How to manipulate data with Python

1. Install Python3
2. Install modules:
   1. pandas
   2. matplotlib

Now we are going to see how to load data from a CSV in Python and how to manipulate it. For practicality, the code to read the CSV will be shown only once, but it must always be in the script.

Load the data with the following snippet:

import pandas as pd

import matplotlib.pyplot as plt

data = pd.read_csv(‘keywords.csv’, delimiter = ‘;’, encoding = ‘ISO-8859-1’)

Show the first 10 rows of the CSV:

print(data.head(10))

Count the number of rows:

print(len(data)

Count the number of rows 2:

print(«number of rows: %i» % len(data))

#The %i indicates that an integer variable goes there.

#When closing double quotes, the % indicates what should replace %i

How to do data splicing:

print(data[:10]) #displays the first 10 rows

print(data[5:]) #displays all but the first 5 rows

print(data[-3:]) #displays the last 3 rows

print(data[:-2]) #displays all but the last 2 rows

print(data[-5:-2]) #sample from the 5th from the end to 2nd from the end

Change all keywords to lowercase:

data = data[‘Keyword’].str.lower()

#Translated, it would be: the keywords are now the keywords but in lower case.

How to get the average CTR per number of words

As a demonstration, let’s take a look at some of what can be done to get SEO insights with Python.

Using the code above we are going to:

1. Load pandas and matplotlib.pyplot modules
2. Load the dataset with the search terms and dump them into a variable called “data”.

The data is now in what is called a Pandas DataFrame. Pandas is a Python module that allows you to manipulate data as if it were a spreadsheet, except that it does not render the content until you ask it to. This saves system resources: imagine loading millions of rows, clicking on a cell, rendering millions of rows again, and so on… it is not practical and that is why we use Pandas.

Pandas works with columns and rows, just like Excel or Google Sheets. In these examples, our DataFrame is going to be called “data”.

How to see the columns of a DataFrame in Pandas:

print(data.columns)

How to see the content of a column in Pandas:

print(data[‘CTR’]) #the column is called CTR

How to average the values of a column in Pandas:

print(data[‘CTR’].mean())

How to obtain the average of a column in Pandas grouped according to another column:

average = data.groupby([‘Ranking’])[‘CTR’].mean()

print(average)

The output should be similar to this (varies depending on the data)

How to plot the average of a column in Pandas with Matplotlib:

#continued from previous script

average.plot(kind=’bar’)

plt.title(‘Average CTR per ranking’)

plt.xlabel(‘Ranking’)

plt.ylabel(‘CTR’)

plt.show()

The output should look like this (varies according to data)

To plot only the positions from 1 to 10, it is necessary to reduce the DataFrame before drawing the averages.

df = df[df[‘Ranking’] <= 10]

average = df.groupby([‘Ranking’])[‘CTR’].mean()#.plot(kind=’bar’)#, cmap=’coolwarm’)

If we want to round the rankings and change the type of data displayed in the column so that they do not show decimals (rounding would treat, for example, a rank 1.9 as a rank 2):

df = df[df[‘Ranking’] <= 10]

df[‘Ranking’] = df[‘Ranking’].round().astype(int)

average = df.groupby([‘Ranking’])[‘CTR’].mean()

And if we plot that by adding the plotting lines shown above and adding these two lines to show the X-axis numbers horizontally and without decimals:

plt.xticks(rotation=’horizontal’)

And that’s it! We got to start getting into all that is Data Science for SEO. There are a lot of graphs that can be born from this data and even more after starting to generate data from the data with feature engineering. An example could be to look at positioning according to the number of words in a search term, some short, medium, and long-tail keywords, etc.

What programming language do you use to extract graphs and insights from your data? Let us know below so we can shape future posts. Until next time and good rankings!

La entrada Data Science: How to extract and process data for decision making in your SEO strategy aparece primero en SEO Blog Punto Rojo.