Greg Reinacker tells us how often NewsGator updates feeds

Posted in News
Fri, Feb 15 - 5:30 pm EDT | 6 years ago by
Comments: 0
Share This Post:
  • Facebook
  • StumbleUpon
  • Tumblr
  • Reddit
  • Twitter

Louis Gray commented a week ago on how poorly Google Reader updates feeds, in direct contrast is this post from Greg Reinacker on how NewsGator works:

Category A: these are feeds that are needed by certain commercial syndication services customers with extremely tight SLAs – some of these SLAs guarantee content available within 2 minutes of publication in a feed. Feeds in this category are retrieved every 60 seconds. Exception – if a feed reliably pings our system with updates, the poll-retrieval interval may be dropped to a lower category; however, if the feed does not appear to ping us with every update, the 60 second interval remains in effect.

Category B: these are feeds with over 20 subscribers, or occasional feeds that for whatever reason are deemed “important” enough to keep in this category. Retrieval interval is 15 minutes.

Category C: these are feeds with 2-19 subscribers, and any feed that requires credentials to access. These feeds are retrieved every 1-2 hours depending on system load.

Category D: these are feeds with only 1 subscriber, which do not require credentials. If that subscriber is an “active user”, interval is 1-2 hours. If that subscriber is not very active, interval is 4-8 hours depending on load. The definition of “active” changes, but think of it as people who use the system daily-ish.

Category E: this is what we affectionately call the “penalty box.” These are feeds which have returned some kind of error, and they are “penalized” for it. For example – if a feed 404’s, it is immediately penalized for 24 hours. A 500 server error? 4 hours. Other kinds of errors (including parsing problems) cause penalties of varying lengths, taking into account how many consecutive errors we see. If a feed continues to have errors for 90 days, it will be blacklisted and no longer retrieved at all…and the only way for a feed to get off the blacklist is for it to a) fix the error(s) and then b) ping us. [I should add that 410 (gone) is not considered an error; feeds that return a 410 are immediately removed and all subscribers are unsubscribed.]

Category F: this is somewhat of a grab bag of other cases. The most visible type of feed in this category is craigslist feeds – we retrieve them on a 48-hour interval. This sucks – for you, for me, for everyone – but the problem is craigslist will throttle and blacklist us, and they seem not to be interested in solving this problem with us (we’re also not the only ones with this problem). So 48 hours is roughly the minimum interval we can get away with and minimize the chances of getting blacklisted (which takes days to undo). Source: NewsGator feed retrieval intervals – Greg Reinacker’s Weblog – Musings on just about everything.

You can’t make it any clearer than that can you?  Greg also added that if you ping NG when you post, you are indexed nearly immediately.  Now why can’t Google to the same?  When I set up FeedBurner for myself or anyone else I have it ping NG and Google (and others).  Google, you have tremendous server resources at hand–step up to the plate guys!

I’m still a big fan of FeedDemon and discovered today (I know I’m late to the party on this one) the NewsGator toolbar for easy RSS subscriptions and one click sharing to my shared items/link blog all from Firefox!  I was wondering about this because often I want to share a post with you all, but if it’s a partial feed I’m not going to until I read it.  Once I’ve read it, well I forget to go back and clip it.  Now … well you get stuff as I browse!

Speaking of finding things, Marshall talked about how to find blogs in your niche and his conclusion isn’t glowing.  There isn’t a great way anymore.  I’m trying Ask.com’s blog search now, but Technorati just isn’t what it used to be and I think both StumbleUpon and del.icio.us are too hit and miss.  I think I find most stuff from other posts and shared items most of the time.

I’ll keep sharing things, though I wish I could have a single RSS feed for all the shared items that come into Google Reader from my contacts.

That’s a gripe for another day.

Share This Post:
  • Facebook
  • StumbleUpon
  • Tumblr
  • Reddit
  • Twitter