刮板无法打印所有结果

Question

我已经在Python中编写了一个脚本，用于从craigslist中删除五个项目的“名称”和“电话”。我面临的问题是，当我运行我的脚本时，它只给出三个结果而不是五个结果。更具体地说，由于前两个链接在他们的页面中没有附加链接（联系信息），所以他们不需要再打开任何附加页面的请求。然而，没有（联系信息）链接的这两个链接无法通过我的第二个函数中的“if ano_page_link：”语句渗透并且从不打印。我该如何解决这个缺陷，以便它是否有电话号码，刮板将打印所有五个结果。

我，试图脚本：

import re ; import requests ; from lxml import html 

base = "http://bangalore.craigslist.co.in" 

url_list = [ 
'http://bangalore.craigslist.co.in/reb/d/flat-is-for-sale-at-cooke-town/6266183606.html', 
'http://bangalore.craigslist.co.in/reb/d/prestige-sunnyside/6259128505.html', 
'http://bangalore.craigslist.co.in/reb/d/jayanagar-2nd-block-4000-sft/6221720477.html', 
'http://bangalore.craigslist.co.in/reb/d/prestige-ozone-type-3-r-villa/6259928614.html', 
'http://bangalore.craigslist.co.in/reb/d/zed-homes-3-bedroom-flat-for/6257075793.html' 
] 

def get_link(medium_link): 
    response = requests.get(medium_link).text 
    tree = html.fromstring(response) 
    try: 
     name = tree.cssselect('span#titletextonly')[0].text 
    except IndexError: 
     name = "" 
    try: 
     link = base + tree.cssselect('a.showcontact')[0].attrib['href'] 
    except IndexError: 
     link = "" 
    parse_doc(name, link) 

def parse_doc(title, ano_page_link): 

    if ano_page_link: 
     page = requests.get(ano_page_link).text    
     tel = re.findall(r'\d{10}', page)[0] if re.findall(r'\d{10}', page) else "" 
     print(title, tel) 

if __name__ == '__main__': 
    for link in url_list: 
     get_link(link) 

结果我有：

Jayanagar 2nd Block, 4000 sft Plot for Sale 9845012673 
PRESTIGE OZONE TYPE D 3 B/R VILLA FOR SALE 9611226364 
T ZED HOMES 3 BEDROOM FLAT FOR SALE 9611226364 

结果我很期待：

A Flat is for sale at Cooke Town 
Prestige Sunnyside 
Jayanagar 2nd Block, 4000 sft Plot for Sale 9845012673 
PRESTIGE OZONE TYPE D 3 B/R VILLA FOR SALE 9611226364 
T ZED HOMES 3 BEDROOM FLAT FOR SALE 9611226364 

Answer 1

需要注意的是，例如，在http://bangalore.craigslist.co.in/reb/d/flat-is-for-sale-at-cooke-town/6266183606.html没有链接匹配'a.showcontact'选择器，所以下面的块

try: 
    link = base + tree.cssselect('a.showcontact')[0].attrib['href'] 
except IndexError: 
    link = "" 

将返回link = ""

然后当你调用if ano_page_link:在if块中的所有命令都被忽略的条件if ""是False并没有打印出来

你可以试试下面来代替：

def parse_doc(title, ano_page_link): 

    if ano_page_link: 
     page = requests.get(ano_page_link).text    
     tel = re.findall(r'\d{10}', page)[0] if re.findall(r'\d{10}', page) else "" 
     print(title, tel) 
    else: 
     print(title) 

Answer 2

您可以通过分离两个任务来获得更大的灵活性收集数据和打印数据。稍后想要扩展时，添加更多信息会更容易。

def collect_info(medium_link): 
    response = requests.get(medium_link).text 
    tree = html.fromstring(response) 

    title = get_title(tree) 
    contact_link = get_contact_link(tree) 
    tel = get_tel(contact_link) if contact_link else '' 

    return title, tel 


def get_title(tree): 
    try: 
     name = tree.cssselect('span#titletextonly')[0].text 
    except IndexError: 
     name = "" 
    return name 

def get_contact_link(tree): 
    try: 
     link = base + tree.cssselect('a.showcontact')[0].attrib['href'] 
    except IndexError: 
     link = "" 
    return link 

def get_tel(ano_page_link): 
    page = requests.get(ano_page_link).text 
    tel = re.findall(r'\d{10}', page)[0] if re.findall(r'\d{10}', page) else "" 
    return tel 

def print_info(title, tel): 
    if tel: 
     fmt = 'Title: {title}, Phone: {tel}' 
    else: 
     fmt = 'Title: {title}' 
    print(fmt.format(title=title, tel=tel)) 

if __name__ == '__main__': 
    for link in url_list: 
     title, tel = collect_info(link) 
     print_info(title, tel)