从网页提取所有链接

Question

我想写一个功能，需要一个网页的URL，下载网页，并返回该网页的URL列表（使用urllib模块） 任何帮助，将不胜感激

Answer 1

在这里你去：

import sys 
import urllib2 
import lxml.html 

try: 
    url = sys.argv[1] 
except IndexError: 
    print "Specify a url to scrape" 
    sys.exit(1) 

if not url.startswith("http://"): 
    print "Please include the http:// at the beginning of the url" 
    sys.exit(1) 

html = urllib2.urlopen(url).read() 
etree = lxml.html.fromstring(html) 

for href in etree.xpath("//a/@href"): 
    print href 

---

 
C:\Programming>getlinks.py http://example.com 
/
/domains/ 
/numbers/ 
/protocols/ 
/about/ 
/go/rfc2606 
/about/ 
/about/presentations/ 
/about/performance/ 
/reports/ 
/domains/ 
/domains/root/ 
/domains/int/ 
/domains/arpa/ 
/domains/idn-tables/ 
/protocols/ 
/numbers/ 
/abuse/ 
http://www.icann.org/ 
mailto:iana@iana.org?subject=General%20website%20feedback