top of page

Wix’s bot log reports: How to troubleshoot technical issues for better SEO

Author: George Nguyen

an illustration of a bot log report in Wix

Your site’s technical health is a crucial part of your search visibility. If you don’t maintain it, search engines like Google may not properly crawl and index your pages.


This means that the content you’ve worked so hard to create might never make it to the search results, which can mean fewer potential customers for your business.


Server logs can help you avoid this by highlighting areas to troubleshoot and better ways to optimize your site. With the help of Wix’s log data reports and some familiarity with server logs, you’ll be able to remedy problems that could be affecting Google’s ability to efficiently crawl your site (i.e., crawl budget).


What are server logs?


How search engine bots work


Server logs are records of every request a server receives, including requests from bots (automated programs that sites like Google use to crawl your pages) and human visitors. These requests represent attempts to view a page on your site.


Each time a user or bot makes a request, the server records:

  • The page that was requested

  • The request date

  • The response the request got


Server responses are shown as status codes, which are grouped into five classes:

  • 1XX – Informational response. Request received and understood.

  • 2XX – Success. Request received, understood, and accepted.

  • 3XX – Redirection. An additional action was taken to complete the request.

  • 4XX – Client error. There was an error; the request could not be completed.

  • 5XX – Server error. The server could not complete the action.


What server logs can tell us

For SEO purposes, we’re going to focus on the log data from bots, which can reveal insights you can use to improve your organic visibility. These include:

  • Which pages are or aren’t getting crawled by search engines

  • Which pages may be getting crawled too often or not enough

  • Errors on your site that search engine bots are encountering


For example, if you’ve created a new product page but it’s not receiving traffic, your bot log reports could tell you whether search engine bots have crawled it. If they haven’t, that may explain why no users have visited the page.


Wix’s bot log reports


Wix users can access their bot log data by navigating to the Analytics & Reports section of their dashboard, and then selecting Reports. In the Marketing & SEO section, click the “show more” button to reveal the Bot Traffic over Time, Bot Traffic by Page, and Response Status over Time reports.

Bot log reports in the Wix dashboard
Access your bot log reports by going to the Analytics & Reports section of your dashboard, and then selecting Reports.

These reports only include requests from bots, which is what you need to troubleshoot technical SEO issues. In addition to viewing this information in the Wix dashboard, you can also download the reports as Excel, CSV or image files to use later.


Now, let’s take a closer look at each report to see how you can use them to improve your site’s SEO.


Bot Traffic over Time

You can use the Bot Traffic over Time report to view which bots have crawled your site or to get an overview of bot traffic trends over time. This information can help you:

  • See whether Google or another search engine crawled your site (or a specific page) recently

  • Estimate the time between when a search engine crawls a page and when the page gets indexed


This report is visualized as a split bar chart (shown below).

Bot traffic over time report in Wix
The Bot Traffic over Time report in Wix.

Highlighting a segment of a bar will show you how many requests a bot made on a given date. In the example above, we can see that Bingbot made 11 requests on February 11.


Filters can help you find the data that matters most

By default, this report shows request data from the most common bots, but you can add or remove bots using the Select bots option. You can also segment the data to find exactly what you’re looking for by using the following filters:

  • Entry page – This shows the report for a specific page (or set of pages) on your site

  • Response status classes – This shows just the data for pages that are returning 2XX, 3XX, 4XX, or 5XX status codes, which can be useful for troubleshooting.

  • Response status codes – This enables you to go into more detail and view the report for just the pages that are returning a specific status code (such as 404s, for example).


Bot Traffic by Page

Your Bot Traffic by Page report shows how frequently your pages are crawled, which can help you see:

  • Which pages on your site are getting crawled the most

  • Whether search engines are crawling your pages more or less often


You can view this report as a bar chart (shown below) or as a customizable table by selecting the Table option above the chart.

The Bot Traffic by Page report in Wix.
The Bot Traffic by Page report in Wix.

Use the table view for more detailed analysis

To get an easier, yet deeper understanding, switch over to the table view of your report.

Wix's Bot Traffic by Page report presented as a chart
The Bot Traffic by Page report presented as a chart.

In this view, you can add or remove columns, such as Date, Bot, Entry page, Unique pages, and so on, enabling you to create an overview with just the information you’re looking for.


Response Status over Time

Your Response Status over Time report is where you’ll go to learn about errors that may be preventing your content from reaching users. It can help you:

  • See whether redirects are working correctly

  • Identify missing pages (404 status code) that can be redirected to improve user experience

Wix's Response Status over Time report
The Response Status over Time report in Wix.

You can view this information in two ways:

  • Unique pages – This shows the number of different pages on your site that received a response status code.

  • Hits – This shows the number of times a specific page was visited by a bot.


You can also filter this data by bot, entry page or response status code. And, the table view for this report enables you to toggle columns to show more or less data.


Now that you’re familiar with bot log reports, the next step is to check in on them regularly so that you can continue to fix errors and find more ways to improve your SEO.

Wix's subscription options for its bot log reports.
The subscription options for bot log reports in Wix.

Consider scheduling each report so that they automatically get sent to your inbox as often as you like. You can do this by selecting the Subscribe button at the top-right of each report.


Monitor bot log reports to identify issues and opportunities


Log data is no different from your regular user analytics in that they both allow you to monitor crucial site functions and provide you with areas to troubleshoot or optimize. Now that you understand how to interpret this data, take a look at your reports to establish a baseline. Then, subscribe to your reports so that you’re always up to date with your site’s technical health and can address problems as they arise.

 

George Nguyen - Director of SEO Editorial, Wix

George Nguyen is the Director of SEO Editorial at Wix. He creates content to help users and marketers better understand how search works. He was formerly a search news journalist and is known to speak at the occasional industry event.

Twitter | Linkedin



bottom of page