Ana içeriğe atla

Python mechanize, beautiful soup and html scraping

I used to have bash,sed,awk,curl,beautifiul  soup  when I was doing html parsing,scraping and automating tasks. I was aware of mechanize but was not using it. Today I decided to give it a try for an automation task and I didnt regret. It was a real fun(!) learning(!) it. Actually I learned a bit of it. It helped me a lot on automating browser requests. Although I didnt yet use it for form handling, I know it has some magical power there too. I also realised that one must use Beautiful Soup with Mechanize. These make an awesome combo worth trying and using. Thanks to all the folks behind them.
here is a snippet
br = mechanize.Browser()

br.open("http://www.site.com")
all_links=[l for l in br.links(url_regex="pattern")]
for i in all_links[5:]:


br.follow_link(i)
temp=br.response().read()
soup=BeautifulSoup(temp)                                                                                                  
link=soup.find('a', href=re.compile("mp3"))   #title\/tt[0-9]*\/"))
if hasattr(link, "href"):
    lin=link['href']
    file=lin.split("/")[-1]
    print file+"----"+lin
    br.retrieve(lin,file)

Here are some helper links
http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
http://stockrt.github.com/p/handling-html-forms-with-python-mechanize-and-BeautifulSoup/

Yorumlar

Adsız dedi ki…
Having read this I believed it was very enlightening.
I appreciate you taking the time and effort to put this information together.
I once again find myself personally spending
way too much time both reading and commenting. But so what,
it was still worth it!

My web page Boutique Air Jordan
Adsız dedi ki…
Hi i am kavin, its my first occasion to commenting anywhere, when i
read this piece of writing i thought i could
also make comment due to this sensible article.

Bu blogdaki popüler yayınlar

An Awk feature

I was trying to use sub (or gsub) function in awk to delete an \r from the end of line. For that purpose I used dum=sub(/\r/,"",$0) and thought that the variable "dum" will be filled with the changed value of $0. But I realised that sub function only make the change and returns 0,1 accordingly. So I wrote like this sub(/r/,"") ; x=$0;

Centos 6 Postgresql 9.1 Install

Today I have changed my mind and decided to begin learning and using Postgresql. On a Centos 6 host there are some ways of installing Postgresql. I have chosen the easy one: use repos. Here http://yum.pgrpms.org/repopackages.php is the official Postgresql repos for Red Hat related distros. I have chosen the x86_64 rpm for Centos 6 and installed it. Later using yum install postgresql91-server I have installed three packages postgresql91,postgresql91-server and postgresql91-libs which is enough for now to start Postgresql 9.1. Using the instructions at /usr/share/doc/postgresql91-9.1.1/README.rpm-dist , I first did service postgresql-9.1 initdb and then service postgresql-9.1 start.Voila! Now let the journey begin! Edit:  Here is another place to look instructions http://www.postgresonline.com/journal/archives/203-postgresql90-yum.html

OpenBSD Nfsen install

I was  trying to find a solution for network monitoring as a friend's request. OpenBSD is the firewall. I saw the Nfsen packages and installed to a 5.2 server. After some effort trying to understand what is going on behind, I realised that Netflow is a protocol by Cisco designed to export router traffic to outside. And OpenBSD has a support of it with pflow interface. First I made a new pflow0 interface like ifconfig pflow0 flowdst 127.0.0.1:9998 flowsrc 127.0.0.1 up Then made some configuration changesto /etc/nfsen.conf and /var/www/htdocs/nfsen/conf.php. Because Apache in OpenBSD runs in chrooted by default, if you dont make the necessary changes in these pages you got an error stating  ERROR: nfsend connect() error: Permission denied!  ERROR: nfsend - connection failed!!  ERROR: Can not initialize globals! which is because apache cant reach the communication socket which is initially  located at $PIDDIR/nfsen.comm namely /var/db/nfsen/run direc...