如何使用加载更多选项与非头部网页刮板[Instagram]

Question

我想从URL下载位置细节从Instagram刮，但我不能使用加载更多的选项来从网址刮更多的位置。

我很欣赏关于如何修改代码的建议，或者需要使用哪个新代码块来获取特定网址中的所有可用位置。

代码：

import re 
import requests 
import json 
import pandas as pd 
import numpy as np 
import csv 
from geopy.geocoders import Nominatim 

def Location_city(F_name): 
    path="D:\\Everyday_around_world\\instagram\\" 
    filename=path+F_name 
    url1="https://www.instagram.com/explore/locations/c1027234/hyderabad-india/" 
    r = requests.get(url1) 
    df3=pd.DataFrame() 
    match = re.search('window._sharedData = (.*);</script>', r.text) 
    a= json.loads(match.group(1)) 
    b=a['entry_data']['LocationsDirectoryPage'][0]['location_list'] 
    for j in range(0,len(b)): 
     z= b[j] 
     if all(ord(char) < 128 for char in z['name'])==True: 
      x=str(z['name']) 
      print (x) 
      geolocator = Nominatim() 
      location = geolocator.geocode(x,timeout=10000) 
      if location!=None: 
       #print((location.latitude, location.longitude)) 
       df3 = df3.append(pd.DataFrame({'name': z['name'], 'id':z['id'],'latitude':location.latitude, 
             'longitude':location.longitude},index=[0]), ignore_index=True) 
    df3.to_csv(filename,header=True,index=False) 
Location_city("Hyderabad_locations.csv") 

在此先感谢您的帮助..

Answer 1

对于Instagram的URL为“看多”按钮，我想你可能会描述增加了页码，你的网址像这样：https://www.instagram.com/explore/locations/c1027234/hyderabad-india/?page=2

只要你继续收到结果，你可以添加一个迭代计数器来模拟增加页码和循环。我添加一个try，除了监视没有更多结果时引发的KeyError，然后设置条件以退出循环并将数据框写入csv。

修改后的代码：

import re 
import requests 
import json 
import pandas as pd 
import numpy as np 
import csv 
from geopy.geocoders import Nominatim 

def Location_city(F_name): 
    path="D:\\Everyday_around_world\\instagram\\" 
    filename=path+F_name 
    url1="https://www.instagram.com/explore/locations/c1027234/hyderabad-india/?page=" 
    pageNumber = 1 
    r = requests.get(url1+ str(pageNumber)) #grabs page 1 
    df3=pd.DataFrame() 
    searching = True 
    while searching: 
     match = re.search('window._sharedData = (.*);</script>', r.text) 
     a= json.loads(match.group(1)) 
     try: 
      b=a['entry_data']['LocationsDirectoryPage'][0]['location_list'] 
     except KeyError: # 
      print "No more locations returned" 
      searching = False # will exit while loop 
      b = [] # avoids duplicated from previous results 
     if len(b) > 0: # skips this section if there are no results 
      for j in range(0,len(b)): 
       z= b[j] 
       if all(ord(char) < 128 for char in z['name'])==True: 
        x=str(z['name']) 
        print (x) 
        geolocator = Nominatim() 
        location = geolocator.geocode(x,timeout=10000) 
        if location!=None: 
         #print((location.latitude, location.longitude)) 
         df3 = df3.append(pd.DataFrame({'name': z['name'], 'id':z['id'],'latitude':location.latitude, 
             'longitude':location.longitude},index=[0]), ignore_index=True) 
     pageNumber += 1 
     next = url1 + str(pageNumber) # increments url 
     r = requests.get(next) # gets results for next url 
    df3.to_csv(filename,header=True,index=False) #When finished looping through pages, write dataframe to csv 
Location_city("Hyderabad_locations.csv")