怎么做，使用R

我试图刮掉从以下站点找到该表从网站刮网表：，我已经寻找几种不同的方法解答https://finance.yahoo.com/gainers?e=us怎么做，使用R

但是凑在这里从现场表和没有任何方法为我工作。

我曾尝试：

library(xml2) 
url <- "https://finance.yahoo.com/gainers?e=us" 
tbl <- read_html(url)

也：

library(XML) 
url <- "https://finance.yahoo.com/gainers?e=us" 
tbl <- readHTMLList(url)

和其它软件包如rvest但我不能让表显示！

来源

2016-11-06 Rime

它们将该数据存储在页面本身的JavaScript块中。你有两个选择：要么使用RSelenium，然后对付真正复杂的表格，你必须放松或从V8包得到一些帮助后，要进行一些字符串手术的一些JSON的mangling相结合：

library(V8) 
library(xml2) 
library(stringi) 
library(jsonlite) 
library(ndjson) 

pg <- read_html("https://finance.yahoo.com/gainers?e=us")

这下位从显着的<script>标记中提取数据。我正在按位置做这件事，这意味着如果Yahoo！曾经改变它的格式（比如，如果VZ曾经完成购买，那么插入无用的Verizon无线广告通过javascript）。你可以改变这个寻找文字指标，但这会导致第二个问题...

“刮”这种方式的第二个问题是，如果JavaScript的数据有变化，这也将打破。但是对于基于HTML的抓取也是如此。此方法不需要启动用于导航Web服务器的Web服务器和控制服务器。反正...

html_nodes(pg, "script")[13] %>% 
    html_text() %>% 
    stri_replace_first_fixed("(function (root) {", "var root = { App : {}};\n") %>% 
    stri_replace_last_fixed("}(this));", "") -> js

我们不得不这样做，因为^^在<script>标签javascript代码期待是在浏览器（我们没有）。现在，我们评估的是通过V8 JavaScript的大块，并提取主要数据元素：

ctx <- v8() 
ctx$eval(JS(js)) 
root <- ctx$get("root", flatten=TRUE)

中的数据元素包含所有该页面的数据（这实在是一个单页的应用程序）的。因此，我们必须找到我们关心的数据，实在是远了嵌套的JavaScript漏洞：

quotes <- root$App$main$context$dispatcher$stores$`QuoteDataStore-Immutable`$quoteData

有许多方法把这一嵌套表数据转换成一个漂亮的，长方形的数据帧。下面的方法选择“将其转换为JSON，然后将其从JSON中以完全”平坦“的格式返回。随意使用其他方法，您可以在SO上找到它。

代码首先告诉R忽略非-stocks（因为你，想必，只想要在美丽的表页面上的引号）：

discard(names(quotes), ~grepl("[\\^\\=]", .)) %>% 
    map_df(~ndjson::flatten(toJSON(quotes[[.]]))) %>% 
    glimpse() 

## Observations: 30 
## Variables: 53 
## $ averageDailyVolume3Month.fmt.0  <chr> "437,939", "541,801", "1.033M", "992,278", "1.40... 
## $ averageDailyVolume3Month.longFmt.0 <chr> "437,939", "541,801", "1,033,453", "992,278", "1... 
## $ averageDailyVolume3Month.raw.0  <dbl> 437939, 541801, 1033453, 992278, 1402175, 537906... 
## $ exchange.0       <chr> "NYQ", "NMS", "NYQ", "NGM", "NGM", "NMS", "NGM",... 
## $ exchangeTimezoneName.0    <chr> "America/New_York", "America/New_York", "America... 
## $ exchangeTimezoneShortName.0   <chr> "EST", "EST", "EST", "EST", "EST", "EST", "EST",... 
## $ fiftyTwoWeekHigh.fmt.0    <chr> "15.36", "46.12", "15.82", "7.64", "5.97", "9.86... 
## $ fiftyTwoWeekHigh.raw.0    <dbl> 15.360, 46.120, 15.820, 7.640, 5.970, 9.860, 5.0... 
## $ fiftyTwoWeekHighChange.fmt.0  <chr> "-6.96", "-17.72", "-4.19", "-2.83", "-4.29", "-... 
## $ fiftyTwoWeekHighChange.raw.0  <dbl> -6.960, -17.720, -4.190, -2.830, -4.290, -3.640,... 
## $ fiftyTwoWeekHighChangePercent.fmt.0 <chr> "-45.31%", "-38.42%", "-26.49%", "-37.04%", "-71... 
## $ fiftyTwoWeekHighChangePercent.raw.0 <dbl> -0.4531, -0.3842, -0.2649, -0.3704, -0.7186, -0.... 
## $ fiftyTwoWeekLow.fmt.0    <chr> "6.59", "23.75", "9.46", "4.22", "0.25", "4.97",... 
## $ fiftyTwoWeekLow.raw.0    <dbl> 6.590, 23.750, 9.460, 4.220, 0.251, 4.970, 0.970... 
## $ fiftyTwoWeekLowChange.fmt.0   <chr> "1.81", "4.65", "2.17", "0.59", "1.43", "1.25", ... 
## $ fiftyTwoWeekLowChange.raw.0   <dbl> 1.810, 4.650, 2.170, 0.590, 1.429, 1.250, 0.190,... 
## $ fiftyTwoWeekLowChangePercent.fmt.0 <chr> "27.47%", "19.58%", "22.94%", "13.98%", "569.32%... 
## $ fiftyTwoWeekLowChangePercent.raw.0 <dbl> 0.2747, 0.1958, 0.2294, 0.1398, 5.6932, 0.2515, ... 
## $ fullExchangeName.0     <chr> "NYSE", "NasdaqGS", "NYSE", "NasdaqGM", "NasdaqG... 
## $ gmtOffSetMilliseconds.0    <dbl> -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07, -1.8e+07... 
## $ invalid.0       <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,... 
## $ isLoading.0       <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE,... 
## $ language.0       <chr> "en-US", "en-US", "en-US", "en-US", "en-US", "en... 
## $ longName.0       <chr> "Bankrate, Inc.", "Air Methods Corp.", "Bristow ... 
## $ market.0       <chr> "us_market", "us_market", "us_market", "us_marke... 
## $ marketCap.fmt.0      <chr> "758.268M", "1.082B", "407.786M", "494.03M", "33... 
## $ marketCap.longFmt.0     <chr> "758,267,968", "1,081,625,344", "407,786,176", "... 
## $ marketCap.raw.0      <dbl> 758267968, 1081625344, 407786176, 494030272, 338... 
## $ marketState.0      <chr> "CLOSED", "CLOSED", "CLOSED", "CLOSED", "CLOSED"... 
## $ messageBoardId.0     <chr> "finmb_30061", "finmb_24494", "finmb_292980", "f... 
## $ quoteType.0       <chr> "EQUITY", "EQUITY", "EQUITY", "EQUITY", "EQUITY"... 
## $ regularMarketChange.fmt.0   <chr> "1.20", "3.65", "1.44", "0.59", "0.27", "0.80", ... 
## $ regularMarketChange.raw.0   <dbl> 1.20, 3.65, 1.44, 0.59, 0.27, 0.80, 0.15, 0.04, ... 
## $ regularMarketChangePercent.fmt.0 <chr> "16.67%", "14.75%", "14.13%", "13.98%", "19.15%"... 
## $ regularMarketChangePercent.raw.0 <dbl> 16.6667, 14.7475, 14.1315, 13.9810, 19.1489, 14.... 
## $ regularMarketDayHigh.fmt.0   <chr> "9.90", "30.25", "12.60", "5.29", "1.85", "6.30"... 
## $ regularMarketDayHigh.raw.0   <dbl> 9.9000, 30.2500, 12.6000, 5.2900, 1.8500, 6.2950... 
## $ regularMarketDayLow.fmt.0   <chr> "7.90", "25.30", "9.46", "4.30", "1.33", "5.79",... 
## $ regularMarketDayLow.raw.0   <dbl> 7.9000, 25.3000, 9.4600, 4.3000, 1.3300, 5.7900,... 
## $ regularMarketPrice.fmt.0   <chr> "8.40", "28.40", "11.63", "4.81", "1.68", "6.22"... 
## $ regularMarketPrice.raw.0   <dbl> 8.40, 28.40, 11.63, 4.81, 1.68, 6.22, 1.16, 1.20... 
## $ regularMarketTime.fmt.0    <chr> "4:02PM EDT", "4:00PM EDT", "4:00PM EDT", "4:00P... 
## $ regularMarketTime.raw.0    <dbl> 1478289722, 1478289600, 1478289614, 1478289600, ... 
## $ regularMarketVolume.fmt.0   <chr> "2.383M", "3.32M", "3.411M", "4.567M", "4.232M",... 
## $ regularMarketVolume.longFmt.0  <chr> "2,382,856", "3,320,072", "3,411,052", "4,566,79... 
## $ regularMarketVolume.raw.0   <dbl> 2382856, 3320072, 3411052, 4566790, 4232135, 185... 
## $ sharesOutstanding.fmt.0    <chr> "90.27M", "38.085M", "35.063M", "102.709M", "20.... 
## $ sharesOutstanding.longFmt.0   <chr> "90,270,000", "38,085,400", "35,063,300", "102,7... 
## $ sharesOutstanding.raw.0    <dbl> 90270000, 38085400, 35063300, 102709000, 2016800... 
## $ shortName.0       <chr> "Bankrate, Inc. Common Stock", "Air Methods Corp... 
## $ sourceInterval.0     <dbl> 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, ... 
## $ symbol.0       <chr> "RATE", "AIRM", "BRS", "CERS", "EBIO", "ELNK", "... 
## $ uuid.0        <chr> "79521cde-a3ef-383f-917d-31c49f9082f5", "f0432c1...

我觉得你还是有一些数据转换（取决于你以后）做的，但你有数据，你正在寻找。

来源

2016-11-06 13:02:22 hrbrmstr

非常感谢你！ – Rime

怎么做，使用R

回答

相关问题