使用BrowserSession和HtmlAgilityPack通过.NET登录到Facebook

Question

我试图使用Rohit Agarwal的BrowserSession类与HtmlAgilityPack一起登录并随后在Facebook上导航。

我以前通过编写我自己的HttpWebRequest来管理。然而，它只在我从我的浏览器手动获取cookie并在每次进行新的“会话”时向请求插入新的cookie字符串时才起作用。现在我试图使用BrowserSession来获得更智能的导航。

下面是当前的代码：

BrowserSession b = new BrowserSession(); 

b.Get(@"http://www.facebook.com/login.php"); 
b.FormElements["email"] = "some@email.com"; 
b.FormElements["pass"] = "xxxxxxxx"; 
b.FormElements["lsd"] = "qDhIH"; 
b.FormElements["trynum"] = "1"; 
b.FormElements["persistent_inputcheckbox"] = "1"; 

var response = b.Post(@"https://login.facebook.com/login.php?login_attempt=1"); 

上述工作正常。当我尝试再次使用此BrowserSession来获取另一个页面时出现问题。我这样做是因为BrowserSession从最后一个响应中保存了Cookie并将它们插入到下一个请求中，因此我不必手动输入从我的浏览器中获取的cookiedata。

然而，当我尝试做这样的事情：

var profilePage = b.Get(@"https://m.facebook.com/profile.php?id=1111111111"); 

我回来的文档是空的。我将不胜感激任何关于我做错的投入。

Answer 1

对不起，我不太了解你提到的HTML敏捷包或BrowserSession类。但我确实尝试了与HtmlUnit相同的场景，它工作得很好。我使用的是.NET包装（它的源代码可以发现here并解释多一点here），这里是我使用的代码（一些细节去除，以保护无辜者）：

var driver = new HtmlUnitDriver(true); 
driver.Url = @"http://www.facebook.com/login.php"; 

var email = driver.FindElement(By.Name("email")); 
email.SendKeys("some@email.com"); 

var pass = driver.FindElement(By.Name("pass")); 
pass.SendKeys("xxxxxxxx"); 

var inputs = driver.FindElements(By.TagName("input")); 
var loginButton = (from input in inputs 
        where input.GetAttribute("value").ToLower() == "login" 
        && input.GetAttribute("type").ToLower() == "submit" 
        select input).First(); 
loginButton.Click(); 

driver.Url = @"https://m.facebook.com/profile.php?id=1111111111"; 
Assert.That(driver.Title, Is.StringContaining("Title of page goes here")); 

希望这有助于。

Answer 2

你检查了他们的新API吗？ http://developers.facebook.com/docs/authentication/

可以调用一个简单的URL来获得OAuth2.0用户访问令牌，并附上您的要求休息...

https://graph.facebook.com/oauth/authorize? 
    client_id=...& 
    redirect_uri=http://www.example.com/oauth_redirect 

更改REDIRECT_URI任何你想要的网址，它会使用名为“access_token”的参数对它进行回调。获取它并进行任何你想要的自动化SDK调用。

Answer 3

您可能需要使用WatiN (Web Application Testing In .Net)或Selenium来驱动您的浏览器。这将有助于确保您不必摆弄cookies并进行任何自定义工作以使后续请求能够工作，因为您正在模拟实际用户。

Answer 4

今天我面临同样的问题。我还和HtmlAgilityPack一起合作了Rohit Agarwal的BrowserSession课。 经过一整天的试错编程后，我发现问题是由于在后续请求中没有设置正确的cookie而导致的。 我没有改变初始的BrowserSession代码来正常工作，但我添加了以下功能并略微修改了SameCookieFrom函数。最后它对我很好。

的添加/修改功能如下：

class BrowserSession{ 
    private bool _isPost; 
    private HtmlDocument _htmlDoc; 
    public CookieContainer cookiePot; //<- This is the new CookieContainer 

... 

    public string Get2(string url) 
    { 
     HtmlWeb web = new HtmlWeb(); 
     web.UseCookies = true; 
     web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest2); 
     web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse2); 
     HtmlDocument doc = web.Load(url); 
     return doc.DocumentNode.InnerHtml; 
    } 
    public bool OnPreRequest2(HttpWebRequest request) 
    { 
     request.CookieContainer = cookiePot; 
     return true; 
    } 
    protected void OnAfterResponse2(HttpWebRequest request, HttpWebResponse response) 
    { 
     //do nothing 
    } 
    private void SaveCookiesFrom(HttpWebResponse response) 
    { 
     if ((response.Cookies.Count > 0)) 
     { 
      if (Cookies == null) 
      { 
       Cookies = new CookieCollection(); 
      }  
      Cookies.Add(response.Cookies); 
      cookiePot.Add(Cookies);  //-> add the Cookies to the cookiePot 
     } 
    } 

做些什么：它基本上保存的cookies，从最初的“后反应”，并增加了相同的CookieContainer到后来叫请求。我不完全理解为什么它在初始版本中不起作用，因为它在AddCookiesTo-function中以某种方式执行相同的操作。 （如果（Cookies！= null & & Cookies.Count> 0）request.CookieContainer.Add（Cookies）;） 无论如何，使用这些附加功能它现在应该可以正常工作。

它可以像这样使用：

//initial "Login-procedure" 
BrowserSession b = new BrowserSession(); 
b.Get("http://www.blablubb/login.php"); 
b.FormElements["username"] = "yourusername"; 
b.FormElements["password"] = "yourpass"; 
string response = b.Post("http://www.blablubb/login.php"); 

所有后续调用应该使用：

response = b.Get2("http://www.blablubb/secondpageyouwannabrowseto"); 
response = b.Get2("http://www.blablubb/thirdpageyouwannabrowseto"); 
... 

我希望它可以帮助很多人都面临着同样的问题！

Answer 5

我解决了这个问题的根本原因，如果任何人关心。事实证明，Cookie正被保存在REQUEST对象的CookieContainer中，而不是响应对象。我还添加了下载文件的功能（前提是该文件是基于字符串的）。代码绝对不是线程安全的，但对象不是线程安全的，首先：

public class BrowserSession 
{ 
    private bool _isPost; 
    private bool _isDownload; 
    private HtmlDocument _htmlDoc; 
    private string _download; 

    /// <summary> 
    /// System.Net.CookieCollection. Provides a collection container for instances of Cookie class 
    /// </summary> 
    public CookieCollection Cookies { get; set; } 

    /// <summary> 
    /// Provide a key-value-pair collection of form elements 
    /// </summary> 
    public FormElementCollection FormElements { get; set; } 

    /// <summary> 
    /// Makes a HTTP GET request to the given URL 
    /// </summary> 
    public string Get(string url) 
    { 
     _isPost = false; 
     CreateWebRequestObject().Load(url); 
     return _htmlDoc.DocumentNode.InnerHtml; 
    } 

    /// <summary> 
    /// Makes a HTTP POST request to the given URL 
    /// </summary> 
    public string Post(string url) 
    { 
     _isPost = true; 
     CreateWebRequestObject().Load(url, "POST"); 
     return _htmlDoc.DocumentNode.InnerHtml; 
    } 

    public string GetDownload(string url) 
    { 
     _isPost = false; 
     _isDownload = true; 
     CreateWebRequestObject().Load(url); 
     return _download; 
    } 

    /// <summary> 
    /// Creates the HtmlWeb object and initializes all event handlers. 
    /// </summary> 
    private HtmlWeb CreateWebRequestObject() 
    { 
     HtmlWeb web = new HtmlWeb(); 
     web.UseCookies = true; 
     web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest); 
     web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse); 
     web.PreHandleDocument = new HtmlWeb.PreHandleDocumentHandler(OnPreHandleDocument); 
     return web; 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PreRequestHandler. Occurs before an HTTP request is executed. 
    /// </summary> 
    protected bool OnPreRequest(HttpWebRequest request) 
    { 
     AddCookiesTo(request);    // Add cookies that were saved from previous requests 
     if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request 
     return true; 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PostResponseHandler. Occurs after a HTTP response is received 
    /// </summary> 
    protected void OnAfterResponse(HttpWebRequest request, HttpWebResponse response) 
    { 
     SaveCookiesFrom(request, response); // Save cookies for subsequent requests 

     if (response != null && _isDownload) 
     { 
      Stream remoteStream = response.GetResponseStream(); 
      var sr = new StreamReader(remoteStream); 
      _download = sr.ReadToEnd(); 
     } 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PreHandleDocumentHandler. Occurs before a HTML document is handled 
    /// </summary> 
    protected void OnPreHandleDocument(HtmlDocument document) 
    { 
     SaveHtmlDocument(document); 
    } 

    /// <summary> 
    /// Assembles the Post data and attaches to the request object 
    /// </summary> 
    private void AddPostDataTo(HttpWebRequest request) 
    { 
     string payload = FormElements.AssemblePostPayload(); 
     byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray()); 
     request.ContentLength = buff.Length; 
     request.ContentType = "application/x-www-form-urlencoded"; 
     System.IO.Stream reqStream = request.GetRequestStream(); 
     reqStream.Write(buff, 0, buff.Length); 
    } 

    /// <summary> 
    /// Add cookies to the request object 
    /// </summary> 
    private void AddCookiesTo(HttpWebRequest request) 
    { 
     if (Cookies != null && Cookies.Count > 0) 
     { 
      request.CookieContainer.Add(Cookies); 
     } 
    } 

    /// <summary> 
    /// Saves cookies from the response object to the local CookieCollection object 
    /// </summary> 
    private void SaveCookiesFrom(HttpWebRequest request, HttpWebResponse response) 
    { 
     //save the cookies ;) 
     if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0) 
     { 
      if (Cookies == null) 
      { 
       Cookies = new CookieCollection(); 
      } 

      Cookies.Add(request.CookieContainer.GetCookies(request.RequestUri)); 
      Cookies.Add(response.Cookies); 
     } 
    } 

    /// <summary> 
    /// Saves the form elements collection by parsing the HTML document 
    /// </summary> 
    private void SaveHtmlDocument(HtmlDocument document) 
    { 
     _htmlDoc = document; 
     FormElements = new FormElementCollection(_htmlDoc); 
    } 
} 

/// <summary> 
/// Represents a combined list and collection of Form Elements. 
/// </summary> 
public class FormElementCollection : Dictionary<string, string> 
{ 
    /// <summary> 
    /// Constructor. Parses the HtmlDocument to get all form input elements. 
    /// </summary> 
    public FormElementCollection(HtmlDocument htmlDoc) 
    { 
     var inputs = htmlDoc.DocumentNode.Descendants("input"); 
     foreach (var element in inputs) 
     { 
      string name = element.GetAttributeValue("name", "undefined"); 
      string value = element.GetAttributeValue("value", ""); 

      if (!this.ContainsKey(name)) 
      { 
       if (!name.Equals("undefined")) 
       { 
        Add(name, value); 
       } 
      } 
     } 
    } 

    /// <summary> 
    /// Assembles all form elements and values to POST. Also html encodes the values. 
    /// </summary> 
    public string AssemblePostPayload() 
    { 
     StringBuilder sb = new StringBuilder(); 
     foreach (var element in this) 
     { 
      string value = System.Web.HttpUtility.UrlEncode(element.Value); 
      sb.Append("&" + element.Key + "=" + value); 
     } 
     return sb.ToString().Substring(1); 
    } 
} 

Answer 6

我有类似的症状 - 登录工作，但身份验证cookie是不存在的cookie容器，所以它不随后的请求发送。我发现这是因为web请求在内部处理Location：标头，在幕后重定向到新页面，在进程中丢失了Cookie。我通过加入以下内容来解决这个问题：

request.AllowAutoRedirect = false; // Location header messing up cookie handling! 

...到OnPreRequest（）函数。它现在看起来像这样：

protected bool OnPreRequest(HttpWebRequest request) 
    { 
     request.AllowAutoRedirect = false; // Location header messing up cookie handling! 

     AddCookiesTo(request);    // Add cookies that were saved from previous requests 
     if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request 
     return true; 
    } 

我希望这可以帮助有人遇到同样的问题。