Drop in replacement for Snoopy

by Arnold Daniels on 05/18/2007

I’ve just finished a so called drop in replacement for the snoopy library. I grabbed a RSS aggregater from the planet-php site. The aggregater was using magpierss, which in its place was using snoopy to get the RSS feeds. Unfortunately the whole snoopy library is a bit outdated (the magpierss as well). This shouldn’t be a problem, but I ran in to a few anyway.

The first problem I had was that snoopy always got a time out when fetching content from a server with keep-alive enabled. It should look at the ‘Content-Length’ header, but it doesn’t. Second was that fetching data which was send chunked, header ‘Transfer-Encoding: Chunked’, didn’t arrive well at all. The sizes are send before each chunk, which made them end up in the result. That of course, made the whole XML invalid making magpie fail.

The new snoopy library uses Curl instead of the file handlers (fopen, fgets, etc). Basically all the functionality of snoopy is already in the curl extension, so I just had to write a wrapper for it. It should be a drop in replacement, but I’ve only tested it with magpie, so there might be some bugs still.

I’m happy again. If you are sad about snoopy and want to be happy as well, you can download the lib from this site. Should there be any bugs, please let me know.

For the record… If you are not using snoopy, don’t start now and just learn how to use curl.

Arnold Daniels

I've spend a big part of my life behind a computer, learning about databases (MySQL), programming (PHP) and system administration (Linux). Currently I playing with HTML5, jquery and node.js.

More Posts

Follow Me:
TwitterLinkedIn

There are 12 comments in this article:

  1. 10 June 2007Sined says:

    Thanks, that works for me !

    ReplyReply
  2. 10 July 2007Rom Pereira says:

    Cool man, thank you! RomRP

    ReplyReply
  3. 4 August 2007Pascal says:

    Works beautifully. Thank you, Arnold! (Once I installed the curl module for php, that is).

    ReplyReply
  4. 7 June 2008Chris says:

    I saw your comment regarding Snoopy on the infotech blog, and I couldn’t agree more. I started using snoopy as the base for a web spider, and it just didn’t cut it. I actually wound up switching over to the Zend HTTP packages.
    http://framework.zend.com/manual/en/zend.http.html

    ReplyReply
  5. 20 January 2009Mariano says:

    Thanks ! I\\\’ve spent a couple of hours trying to figure out why magpierss was failing on some feeds while doing ok with others and then finally spotted the problem (the failing feeds were sent as http chunked data). Luckily found this site, replaced snoopy with your code and now it works like a charm :)

    ReplyReply
  6. 18 March 2009Marlin says:

    The HTTP extension provided by PHP does everything snoopy does and much much more. It uses Curl if compiled in with it. It’s tested extensively, why re-invent the wheel?

    ReplyReply
  7. 18 March 2009Arnold Daniels says:

    Marlin,

    The HTTP extension is not a ‘drop in replacement’ for snoopy. You would need to do some rewriting for a project. This lib is here to replace snoopy in projects that already uses snoopy. For new projects, I do not suggest to use this lib (as stated in the last line of the article).

    ReplyReply
  8. 19 March 2009Marlin says:

    Yeah, I understand that you’re creating a drop-in replacement for snoopy, I guess I wasn’t explicit enough – I meant snoopy itself.

    ReplyReply
  9. 5 December 2009Barry says:

    fixes for submit:

    //Tell curl to use the specified request method and version
    if (!empty($this->httpmethod)) curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $this->httpmethod);
    if ($this->httpmethod == “POST”) curl_setopt($ch, CURLOPT_POSTFIELDS, $this->formvars);

    function submit($uri, $formvars=null, $formfiles=null)
    {
    settype($formvars, “array”);
    settype($formfiles, “array”);
    $postdata = ”;
    while(list($key,$val) = each($formvars)) {
    if (is_array($val) || is_object($val)) {
    while (list($cur_key, $cur_val) = each($val)) {
    $postdata .= urlencode($key).”[]=”.urlencode($cur_val).”&”;
    }
    } else
    $postdata .= urlencode($key).”=”.urlencode($val).”&”;
    }

    $this->formvars = $postdata;
    //$this->formfiles = $formfiles;
    $this->httpmethod = “POST”;

    return $this->fetch($uri);
    }

    ReplyReply
  10. 6 January 2010Stefan says:

    I followed the recommendation from Chris and I also only can recommend the Zend_Http compenent. It’s by far superior to Snoopy.

    ReplyReply
  11. 21 November 2011Holiday Homes to Rent in Cornwall says:

    Incredibly interesting details you’ve got remarked, many thanks for placing up.

    ReplyReply
  12. 4 December 2013Suresh says:

    it is working fine, thanks

    ReplyReply

Write a comment:


+ 4 = six