A blog about digital marketing & website optimization

New Site Graph Visualization, Start URL Option & 4X Crawler Speed Boost

Austin in August means excruciating heat. So we’ve been keeping cool inside our air-conditioned office, working hard on making SiteCondor even more useful. A special thanks to all the customers and prospects who’ve been providing us with feedback, you rock!

Site Graph

Our deepest look yet into your site’s linking structure

This new visualization makes it easy to navigate through your entire site to understand how your pages are interlinked. While the picture below looks cool, the Site Graph is a highly interactive visualization that you should try for yourself in order to fully appreciate. We’ll be posting a Site Graph video in the next couple days, keep an eye out for that.
site-graph-screenshot

Start URL

This new Job setting lets you set the crawl starting point

start-url

This can be used in many different ways:

  • When combined with the URL filter, you can limit your crawl to a specific area of your site
  • Create a Scheduled Job that checks on one particular page
  • Start your crawl from pages that are otherwise not discoverable (e.g., landing, private, unpublished pages)
  • Visualize site architecture starting from a specific URL (useful on big sites)

Sign In or Sign Up to try the new setting.

Up to 4X crawl speed increase

SiteCondor can now crawl resource-dense sites up to 4X faster than before

speed

Additional Improvements

  • Added default option to disregard jsessionid from URLs
  • Improved error reporting – removed duplicate 404/Redirect errors that were showing up in the Other Errors section
  • And several other tweaks to make SiteCondor more robust and easier to use

Go ahead and try it out!

Welcome to the SiteCondor Blog

SiteCondor is a website analysis tool for digital marketing experts. You should check it out.

Welcome to our blog, where we share ideas about website optimization, web dev, product updates, and more.

Create your account
Recent Posts
Categories
Subscribe

Visualizations Galore & Crawler Improvements

We’ve been working hard on SiteCondor for the past couple weeks making several significant improvements. Here is a quick summary of the major updates.

Revamped Visualizations

Link Matrix – New!

This new visualization enables you to quickly see how the internal link juice gets distributed amongst interlinked pages. The Link Matrix categorizes all of your site’s pages by their top-level folder and summarizes the outgoing and incoming internal links in an extremely powerful and easy to consume way.

URL Structure

We have completely overhauled this visualization, making it easier to scale for larger sites and easier to extract insights from. It also loads much, much faster. In addition, we have also given you a couple controls over what gets visualized: by using the Max Levels and Max Children Per Node settings you can now choose how deep and wide you would like to visualize the site.

Site Map

Some of you have experienced sporadic timeouts or long load times for this visualization. We have sped this up significantly, usually rendering in under few seconds, even for very large jobs. As with the URL Structure visualization, we have also added Max Levels and Max Children Per Node settings to this visualization.

Crawler Upgrades

We have also made significant improvements to our crawler, enabling it to crawl even larger sites in a faster and more robust way. It just keeps getting better and better! (And we’re not stopping here either.)

Try Out the New Features!

SiteCondor API Ruby client released

It is with great pleasure that we’d like to announce the first (beta) release of the SiteCondor’s API Ruby client.

SiteCondor’s API enables digital agencies to integrate powerful on-page extraction and analysis functionality into their own custom workflows and processes. In-house marketing teams and third-party developers can use the SiteCondor API to enrich their products by adding accurate on-page element information and metrics, content analysis functionality, and more.

This thin wrapper makes it extremely easy for Ruby programmers to interact with various endpoints of our RESTful API. Also included within the github repository are multiple examples showcasing the currently available functionality. Also included are a few advanced examples that perform additional content analysis and data crunching on the client side.

While the client doesn’t yet fully support all of the API features, it does implement most of the job-related GET operations. Let’s look at an example.

Given an API key and a Job ID, the example above first gets a job result, and then gets all the H1 Heading results for that same job. API results are JSON-encoded so they are very easy to digest. The ruby client results are all encoded as Ruby Hash objects.

The job result includes fields that reflect the options selected at the time of job creation and fields that summarize job crawl results. Here’s an example:

SiteCondor aggregates H1 headings by unique values, counting the number of occurrences for each of them, and including the array of URLs where the heading was found. In addition, results are also reverse ordered by frequency (count) so that the most used unique titles are returned first. While the Get H1s API endpoint is actually paginated, the client takes care of this for you so you don’t have to iterate through the result pages.

Here is an example of how the Headings results look like:

For more info on the available functionality, please check out the examples included in the Github repo. If you’d like to gain access to our API, please consider signing up for SiteCondor, or let us know and we may just hook you up with an API trial account.

We plan on creating API clients for other languages, let us know what you’d like to see and we may just release a wrapper for it. ūüôā

In case you were wondering, the code is open source and released under the MIT license.

Crunching some numbers

Since we launched months ago SiteCondor has collected a bunch of data. We’ve been crunching some numbers and thought we’d share the results with you.

The bad

Resources

  • We found at least one redirect (301 or 302) in 80% of our jobs.
  • We found at least one missing resource (404 error) in 66% of our jobs.
  • We found at least one other error (time out, forbidden, etc) in 41% of our jobs.

When aggregated SiteCondor encountered one of the situations listed above in 89% of the jobs.

Pages

  • 88% of the jobs had at least 1 image with no alternate text in the markup.
  • 81% of the jobs had at least 1 page with missing meta description.
  • 77% of the jobs had at least 1 page with a URL too long, or containing at least 1 non-ASCII, underscore, or uppercase character.
  • 74% of the jobs had at least 1 page title being too long.
  • 72% of the jobs had at least 1 page with missing H1 headings.
  • 66% of the jobs had at least 1 page with meta descriptions being too long.
  • 57% of the jobs had at least 1 page with multiple H1 headings.
  • 31% of the jobs had at least 1 page with no title.

Resources & Pages

Overall, 98% of our jobs had at least one of the issues in the lists above.

The good

  • 38% of the sites used some sort of Structured Data (Open Graph, Twitter Cards, or Micro Formats)
  • 31% of the sites used Google Authorship.

Conclusion

While this is obviously limited to the universe of sites crawled by SiteCondor at the time they were crawled, we believe these results are pretty interesting. We can summarize the results with the following statement:

98% of the times, SiteCondor will find at least one issue worth fixing with your site.

Some of these are false positives (e.g.: pages that really don’t need a title, images that don’t need an alternate text, redirects that are meant and are there to stay, etc.)… so YMMV, but we still think that 98% is a pretty solid number. What do you think?

How to keep an eye on multiple sites

Hello there and welcome! We’d like to show you how to use SiteCondor to automate your site auditing tasks. SiteCondor makes it easy for you to schedule crawl jobs on multiple sites and stay aware of site errors and optimization opportunities.

Who is this tutorial for?

If you’re part of a digital marketing agency, custom web development shop, in-house marketing team, startup team, online retailer, or a search marketing freelancer working on client sites – this tutorial is for you.

Why Site Auditing?

To optimize the value your sites generate you have to constantly improve their quality. This ranges from improving the user experience, to providing engaging and relevant content, not to mention monitoring availability and performance – all in an effort to improve conversion rates and search result rankings. You’re always on a mission to get more and better quality traffic. Periodical technical Site Audits can be of enormous help with most of these tasks. There’s a lot to keep an eye on, and SiteCondor can help with many of these tasks.

SiteCondor can alarm you when there are 404 errors due to broken links (including broken links to external sites), broken images, missing resources, general availability issues, missing page titles, missing meta descriptions, missing headings, and images with missing alternate text. All of these basic errors can negatively affect your site’s user experience and search results rankings.

As sites grow and technology evolves, sites usually become more complex and increasingly dynamic, often with multiple teams working on them. Naturally this creates more potential for things to go wrong. Hence the need for periodically keeping an eye on them. [Note: check out 11 creative ways to use SiteCondor for more ideas on using SiteCondor ].

In addition to the custom email alerts triggered by scheduled audit jobs, SiteCondor also provides you with full access to more detailed reports and results which can help you discover many other minor errors and optimization opportunities. Feel free to check out our features page for more.

Alright, enough chit chat. Let’s get to it.

Step 1. Creating Jobs

Assuming you are Signed In to SiteCondor, select Jobs/Create from the navigation menu. Once on the Create Job form, enter the site’s domain name, and check the Images checkbox as in the screenshot below. For our example, we’ll use one of our favorite local coffee shops. If you would like a more rigorous crawl, feel free to check the other resource types and investigate the advanced options.

SiteCondor-CreateJob

Step 2. Viewing Job Results

To monitor your job progress, select Jobs/List from the navigation menu. Clicking on the Refresh button will refresh your job list with the latest status. Once your job has completed you’ll see a Finished status in the Crawl Status column as shown below.

SiteCondor-JobsList

In addition to this, SiteCondor will also send you an email notification every time a job is completed. To view the job results, click on the View Results button. You can view job results at any time, just keep in mind they may only be partial results if the job has not yet completed running. Clicking on View Results will take you to the Job Overview page below.

SiteCondor-JobOverview

You can click on the different sections to explore all the different job results, our interactive site visualizations, and the downloadable XML Sitemap.

Step 3. Creating Job Schedules

You are now ready to create a job schedule and its associated custom alarm. You can schedule jobs to run on a weekly or monthly basis. [ Note: If your site was not crawled properly, your job crawl settings may need tweaking. Feel free to contact us for help, or go back to Step 1 and give it another try. ].

Go back to the Jobs/List page using the navigation menu, and click on the Set Schedule button for the job you created on Step 1. Following the screenshot below, turn the Send Email Alerts switch to ON, and enter values for each of the conditions you would like to monitor. You can leave empty those conditions that you do not wish to monitor. SiteCondor will send you an email alert if any of these conditions are met, every time a scheduled job completes running. In our example, we scheduled the job to run on a weekly basis and alarm us if there broken links, missing images, missing titles, missing meta descriptions, missing headings, images with missing alternate text, and/or any other HTTP errors.

SiteCondor-ScheduleJob

Your job is now scheduled to run and you will receive an email every time it completes, on a weekly or monthly basis as per your schedule settings. The job completion email contains a convenient link to the view Job Results, but you can always see your results from the Job/Lists page.

If any of the threshold values are met you will also receive an email alert, and the Job Results dashboard will reflect the issue, highlighting counts that exceed any of the thresholds set as seen in the following screenshot.

SiteCondor-JobOverviewThresholds

Step 3. Rinse & Repeat

Repeat Step 1 & 2 for each site you would like to monitor. Keep in mind you can choose different job options and different schedule settings for each of them.

[ Tip: Use the Jobs/Scheduled list available from the navigation menu to see which jobs you have scheduled to run next and cancel any scheduled jobs if necessary. ]

Step 4. Keep Calm, Watch your inbox

That’s it, you can relax now :).

We hope you have enjoyed this tutorial, please feel free to leave a reply below if you have any questions or ideas for followup tutorials!

11 creative ways to use SiteCondor

alternative-useFrom a bird’s-eye-view, SiteCondor can help you find website errors and optimization opportunities, making it the perfect tool for:

  • Performing on-page technical Site Audits
  • Assessing and improving the quality of multiple sites over time
  • Quickly gaining insight about a site’s content and architecture

If you have used SiteCondor before, you probably already knew that (otherwise, what are you waiting for?). As we (and our clients) work on multiple sites, we often find ourselves using SiteCondor in more specific and¬†creative ways. We wanted to share some of these use cases with you, so here’s the list:

  • Competitive Analysis: use SiteCondor to keep an eye on your competitors as they publish new content. Research their optimization strategies and find their strong¬†and weak¬†areas. Depending on your industry and how active your competitor ‘s pages are, we recommend you setup a weekly or monthly job for each of your competitor’s sites. Don’t forget their micro sites, blogs, or other web properties they may own.
  • Earn New Business: If you work on client sites, use SiteCondor to quickly assess a prospect’s website. Leverage SiteCondor’s findings to build a quick report and show the skills and value you can bring to potential clients, or quickly evaluate a website’s dimensions, content, and quality to help you estimate effort before you send out that proposal.
  • New Sites: Building new sites can be a daunting task. Use SiteCondor to run technical audits until all errors are cleared, or as a way to check your vendor’s craftsmanship quality. In other words, use SiteCondor as an acceptance criteria for a new site being built and ensuring your new site crawlability.
  • Site or Infrastructure Migrations: Oh my! What are all my current URLs? Where can I see them after the current site is gone? Is the new site up? Does it have all the content is supposed to? Is it working correctly? Did I redirect all the old URLs to the new site URLs? Are there any errors? Use SiteCondor to run a crawl job before and after the migration, spot errors, missing redirects, compare results, and more. We recommend you also schedule a weekly job until the new site becomes stabilized. Use SiteCondor’s XML Sitemap feature to generate the file for your new site (or compare it against yours if your site already has one).
  • Security Vulnerabilities and Attacks: Lots of CMS and other web systems and applications suffer from vulnerabilities at different times. Depending on your version, configuration and setup, your site could be attacked and compromised by hackers or automated scripts trying to find those vulnerable sites. While SiteCondor is not a security tool per se, it can help monitor suspicious or malicious content. For example, you can run SiteCondor with Google’s user agent to check if your site has been subject to a cloaking attack.
  • E-commerce Sites:¬†with e-commerce¬†a broken page or image can mean the difference between a sale or not. SiteCondor can help detect them earlier, while also helping with content marketing and search engine optimization activities, a key area online retailers cannot afford the get wrong. Setup a Scheduled Job with custom Email Alerts to let you know of any broken links (instead of waiting for your customer or boss to tell you about them).
  • User Generated Content: groups and forums, Q&A sites, review sites, you name it. Search Engines love their content, but it can get messy too. You can use SiteCondor to help you optimize the content, but also to enforce contributing policies. For example, you can easily monitor all external follow links, potentially avoiding penalties from search engines (for more on this, check out our post on Find & Fix Your Low-Quality External Links).
  • High-value CMS powered, Corporate Marketing Sites: oftentimes these sites are being worked on by multiple teams. For example, the marketing team may add or refresh content, a development team add new features, a search team or agency optimize and distribute the content, and a dev ops or infrastructure team work on monitoring and improving performance and availability. In addition, each converted visit usually brings in a substantial amount of value, hence monitoring user experience while all these teams work on the site at the same time is a must. A quality site (particularly a site with better quality than your direct competitors) can help build awareness and ultimately capture that new business lead. Here we recommend using SiteCondor to help monitor new content, spot optimization opportunities, and keep an eye on overall quality. For example, compare crawl results over time to discover new content that may need to be optimized (e.g.: new images missing alt text attributes, or new pages with missing titles or headings), or unnecessary redirects (internal links that can be updated to avoid the redirect).
  • Blogs:¬†blogs often have new content, lots of writers, comments, and oh no, spam! SiteCondor is perfect to keep an eye on blogs, specially where multiple authors and contributors participate. We recommend you use the Explore/Pages report and URL Structure visualization to constantly monitor the different pages and URLs. This will give you a bird’s eye view of the blog that independent authors may not have, and you can use this to fix obvious errors (such as bad post URLs), but also to generate content ideas and opportunities.
  • Custom Development and Others: custom developed sites, web applications, integrations, custom CMS implementations, and other constantly evolving products are awesome, but keeping them bug-free can be a challenge. Sometimes they rely on other external resources that may indadvertedly move or otherwise become unavailable. Use SiteCondor to monitor your sites as upgrades, new deployments, or migrations take place. Setup a weekly scheduled Job or, even better, trigger a new crawl every time a release or build takes place by integrating your deployment process with our API.
  • Static Sites: Even static sites can benefit from SiteCondor. A typical example is having broken links to other pages in external sites that no longer exist. You can use SiteCondor to monitor those, just set a Schedule Job with email alerts for 404 errors.

Have you found a creative or alternative way of using SiteCondor? Leave a reply below and tell us about it!

Watch content, visual, and performance changes on any page

From time to time, we like to take short breaks from working on the main SiteCondor product and explore complementary ideas. Hence Page Watch was born.

Page Watch is a free tool that lets you track and understand page content, visual, and performance changes over time. You can use it to monitor your homepage, landing pages, competitor landing pages, product pages, terms and conditions or policies, and more.

Click here to try out Page Watch

To create a job, simply enter your email address and the URLs for the pages you’d like to watch (no signup needed):

SiteCondor-PageWatch-Create

Each Page Watch job can check up to 3 pages for changes over a 5 day span. Feel free to create as many jobs as you’d like. For each of the requested pages it will take a daily snapshot of content, visual, and performance changes and compares it to the previous day. Page Watch will send you an email letting you know there’s new results.

Here’s a quick screenshot of what results look like:

SiteCondor-PageWatch-Results

And here’s a screenshot on how content changes are displayed:

SiteCondor-PageWatch-Diff

We are really excited to release this free tool and sincerely hope you’ll find it useful. If you do, please help us spread the word and share it with your friends and colleagues. If you have a minute, we would love for you to check it out and let us know what you think!

SiteCondor now checks external links

We’ve added external link status checking as a follow up to our recent follow/nofollow update to SiteCondor’s External Links section.

This means SiteCondor now follows all external links found during a crawl, checks their status codes, and reports them appropriately under the Resources section. If the external resource is successfully fetched (2xx HTTP status code), it will be listed under the Resources » All tab, as shown in the screenshot below.

Resources

Otherwise, depending on the nature of the error, the Resource will be listed under the 404, Redirects, or Other Errors tabs.

You can use this improvement to check (and monitor) your site for broken links, but also for other purposes such as broken link building.

As usual, we hope you’ll like this improvement – let us know what you think!

Find & Fix Your Low-Quality External Links

As you may know, in late January Matt Cutts published a much talked about post entitled The decay and fall of guest blogging for SEO. In it he directly discourages guest-blogging for links, and cites a July 2013 Search Engine Land article wherein his colleague at Google, John Mueller, explicitly encourages no following links in guest posts.

(If you’re not clear on follow/nofollow, WordStream has an excellent post on the subject.)

For the last couple months there has been a sharp uptick in the number of people reporting manual link penalties showing up in Google Webmaster Tools.

At PubCon New Orleans it became public that industry stalwart Ann Smarty’s MyBlogGuest had been hit by a penalty for passing link value in guest posts. Google was sending a clear message – clean up your guest blogging links.

[Edit: As per Ann’s comment below, the reason for penalty expressed above is just our assumption, there was no clear reason given by Google]

Savvy internet marketing veterans like Eric Enge of Stone Temple Consulting have urged taking proactive approach in cleaning up your links (See recommendation #6 in Is link-building dead?). The question is how?

Using SiteCondor to Analyze Links to External Sites

The latest SiteCondor release includes an improved External Links section containing a breakdown of your follow/nofollow links.

Let’s take a look at a practical example, shall we?

I’ve run a crawl job on a made-up sample blog post. (Note: you can click the images to enlarge them.)

sample-blog-post

You’ll notice that there are several spammy links, a legit link, and a legit link with overly-optimized anchor text. There is one no followed link.

Pro tip: For sites with hundreds of links you can use the Search function to find particular links.

before-follow

before-nofollow

Given this scenario, you could clean things up by:

1. nofollowing the spammy links (if not getting rid of them altogether)
2. adjusting the anchor text for the real estate link

Here’s the cleaned up result:¬†after-follow

after-nofollow

If you’d like to see the follow/nofollow¬†links on any site, check out the External Links tab in your account or sign up for a free trial. Best of luck!

Scheduled Jobs and Customizable Email Alerts

Since early on, many of you expressed the desire to have SiteCondor periodically audit your site according to a schedule, sending alerts via email if certain thresholds were met.

As of today you can schedule your SiteCondor jobs to run on a weekly or monthly basis. You can also configure your scheduled jobs send an email if certain conditions are met. For instance, you can set up a scheduled job to email you if there are more than X number of 404 pages.

It’s difficult to keep a website healthy as new content is added, technology and infrastructure evolves, and people move around. SiteCondor’s Job Scheduling helps you more consistently spot issues and uncover optimization opportunities.

Let’s take a look, shall we?

Scheduled Jobs

As shown in the screenshots below, once a job has been completed you can now create a Schedule, which will trigger the same job to run on a weekly or monthly basis.

jobs_list

schedule_a_job

Customizable Email Alerts

While Scheduling a Job you also have the option to configure email alerts. Switching the Send Email Alerts option to the On position lets you specify thresholds for six different conditions:

  • Number of 404 Errors
  • Other Errors
  • Missing Titles
  • Missing Image Alt
  • Missing Meta Descriptions
  • Missing Headings

SiteCondor will then send you an email alert if one or more threshold value(s) is met or exceeded upon a scheduled job completion. In addition to the email alert, you will also receive our usual job completion email.

We plan on adding more monitoring capabilities as we get more feedback. Keep it coming!

email_alert

You can also look at all your Scheduled Jobs and when they are scheduled to run next, as well as cancel previously scheduled jobs.

Job Results Enhancements 

We have also improved our Job Results page. This page now has an overall job summary, enabling you to very quickly grasp the results and counts for each of the Explore sections as well as deep links to all the Explore & Visualize sections for quicker navigation. In addition, scheduled jobs will show any counts meeting or exceeding thresholds in red, and counts within acceptable ranges in green.

job_results

Go ahead and give SiteCondor’s new scheduling and revamped reporting interface a try and let us know what you think!