MySQL：用FROM子句中的相关子查询重写MSSQL？

Question

我们有一个包含网站的页面访问量，像一个表：

time  | page_id 
----------|----------------------------- 
1256645862| pageA 
1256645889| pageB 
1256647199| pageA 
1256647198| pageA 
1256647300| pageB 
1257863235| pageA 
1257863236| pageC 

在我们的生产表，目前大约40K行。我们要生成，每天的独特网页在过去30天里浏览，60天，90天计数。因此，在结果集中，我们可以查找了一天，看到独特页是多少那一天之前的60天期限内访问。

我们能够得到一个查询的MSSQL工作：

SELECT DISTINCT 
CONVERT(VARCHAR,P.NDATE,101) AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-29,P.NDATE) AND P.NDATE) AS SUB) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-59,P.NDATE) AND P.NDATE) AS SUB) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-89,P.NDATE) AND P.NDATE) AS SUB) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

注：由于MSSQL不具备FROM_UNIXTIME功能，我们增加了测试NDATE列，它仅仅是转换time。生产表中不存在NDATE。

这个查询转换到MySQL为我们提供了“未知科拉姆P.time”错误：

SELECT DISTINCT 
FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

我明白这是因为我们不能有一个相关子查询，在外部FROM子句引用的表。但是，不幸的是，我们在如何将这个查询转换为在MySQL中工作时遇到了困难。现在，我们只需返回表中的所有DISTINCT行，并在PHP中进行后处理。 40K行花费大约2-3秒。当我们有100个1000行的行时，我担心这个表现。

是否有可能在MySQL中吗？如果是这样，我们可以期望它比我们的PHP后处理解决方案表现更好。

UPDATE： 这里的查询创建表：

CREATE TABLE `perflog` (
    `user_id` VARBINARY(40) NOT NULL , 
    `elapsed` float UNSIGNED NOT NULL , 
    `page_id` VARCHAR(255) NOT NULL , 
    `time` INT(10) UNSIGNED NOT NULL , 
    `ip` VARBINARY(40) NOT NULL , 
    `agent` VARCHAR(255) NOT NULL , 
    PRIMARY KEY ( `user_id` , `page_id` , `time` , `ip`, `agent`) 
) ENGINE MyISAM 

我司生产的表有40K〜行迄今！

Answer 1

为什么你还要埋在第二级这样子查询？试试这个：

SELECT DISTINCT 
FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

Answer 2

你可以尝试使用单选。

选择日期90天前仅之间的值。

然后使用的情况下，语句中的每个fiels以检查日期30，60，90之间落入对于每个字段，如果情况属实，那么1否则为0，并且计数的那些。

喜欢的东西

SELECT SUM(CASE WHEN p.Date IN 30 PERIOD THEN 1 ELSE 0 END) Cnt30, 
     SUM(CASE WHEN p.Date IN 60 PERIOD THEN 1 ELSE 0 END) Cnt60, 
     SUM(CASE WHEN p.Date IN 90 PERIOD THEN 1 ELSE 0 END) Cnt90 
FROM Table 
WHERE p.Date IN 90 PERIOD 

Answer 3

更改子查询到连接，因为这样：

select 
    FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
    count(distinct p30.page_id) AS '30D', 
    count(distinct p60.page_id) AS '60D', 
    count(distinct p90.page_id) AS '90D' 
from 
    perflog p 
    join perflog p30 on FROM_UNIXTIME(p30.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 
    join perflog p60 on FROM_UNIXTIME(p60.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 
    join perflog p90 on FROM_UNIXTIME(p90.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 

然而，这可能缓慢，因为堆的杀戮你的日期列任何indicies功能的运行，更好的解决方案可能是：

create temporary table perf_tmp as 
select 
    FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'VIEWDATE', 
    page_id 
from 
    perflog; 

create index perf_dt on perf_tmp (VIEWDATE); 

select 
    VIEWDATE, 
    count(distinct p30.page_id) AS '30D', 
    count(distinct p60.page_id) AS '60D', 
    count(distinct p90.page_id) AS '90D' 
from 
    perf_tmp p 
    join perf_tmp p30 on p30.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 30 DAY) AND p.VIEWDATE 
    join perf_tmp p60 on p60.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 60 DAY) AND p.VIEWDATE 
    join perf_tmp p90 on p90.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 90 DAY) AND p.VIEWDATE; 

Answer 4

这是我用来解决这个问题的PHP。理想情况下，我希望这一切都由MySQL完成（如果可以更快地完成）。我只发布任务本作进一步澄清：

function getUniqueUsage($field = 'page_id', $since = 90){ 
    //we need to add 90 days onto our date range for the 90-day sum 
    $sinceSeconds = mktime(0, 0, 0, $m , $d, $y) - (($sinceDays + 90) * (60 * 60 * 24)); 
    //==> omitting mySQL connection details<== 
    $sql = "SELECT DISTINCT From_unixtime(time,'%Y-%m-%d') AS date, $field FROM perflog WHERE time > $sinceSeconds ORDER BY date" ; 
    $sql_results = mysql_query($sql); 
    $results = array(); 
    //all page ids per date (ending-up with only unique date keys) 
    while ($row = mysql_fetch_assoc($sql_results)) 
    { 
     $results[$row['date']][] = $row[$field]; 
    } 
    $sums = array(); 
    //initialize sum array, with only unique dates (days) 
    foreach (array_keys($results) as $date){ 
     $sums[$date] = array(0,0,0); 
    } 
    //calculate the 30/60/90 day unique pages for each day 
    foreach (array_keys($sums) as $ref_date){ 
     $merges30 = array(); 
     $merges60 = array(); 
     $merges90 = array(); 
     $ref_time = strtotime($ref_date); 
     $ref_minus_30 = strtotime("-30 Days",$ref_time); 
     $ref_minus_60 = strtotime("-60 Days",$ref_time); 
     $ref_minus_90 = strtotime("-90 Days",$ref_time); 
     foreach ($results as $result_date => $pages){ 
      $compare_time = strtotime($result_date); 
      if ($compare_time >= $ref_minus_30 && $compare_time <= $ref_time){ 
       $merges30 = array_merge($merges30, $pages); 
      } 
      if ($compare_time >= $ref_minus_60 && $compare_time <= $ref_time){ 
       $merges60 = array_merge($merges60, $pages); 
      } 
      if ($compare_time >= $ref_minus_90 && $compare_time <= $ref_time){ 
       $merges90 = array_merge($merges90, $pages); 
      } 
     } 
     $sums[$ref_date] = array(count(array_unique($merges30)),count(array_unique($merges60)),count(array_unique($merges90))); 
    } 
    //truncate to only specified number of days 
    return array_slice($sums,-$since, $since, true); 
} 

正如你可以看到，有很多不幸的阵列合并，ING和阵列独特-ING的。

Answer 5

注：我在阅读@astander，@ Donnie，@longneck的解决方案后写这篇文章。

我知道性能很重要，但为什么不存储聚合？十年的每日行是3650行，每列只有几列。

TABLE dimDate (DateKey int (PK), Year int, Day int, DayOfWeek varchar(10), DayInEpoch....) 
TABLE AggVisits (DateKey int (PK,FK), Today int, Last30 int, Last60 int, Last90 int) 

这样，您只需在一天结束时仅运行一次查询，仅一天。预先计算的总量是任何高性能分析解决方案（多维数据集）的根源。

UPDATE：
你可以通过引入另一列DayInEpoch int（天数自说1990-01-01）加快这些查询。然后您可以删除所有这些日期/时间转换功能。