1
我在我正在开发的网站上安装了Sphider 1.3.5搜索引擎。我正在共享主机（Dreamhost）上工作，并且一切似乎都已安装正常，并且正常工作，除了一个问题。使用Sphider搜索引擎和robots.txt
Sphider似乎忽略了我的robots.txt文件。我已成功验证了robots.txt语法，检查并重新检查了我的路径以确保它们是正确的，并且一次又一次在管理区域重新编制了该网站的索引，不管我做了什么，Sphider都不会超出我所不允许的文件。
下面是该文件的文本：
# robots.txt for http://redesign.ladottransit.com/
User-agent: *
Disallow: /sphider-1.3.5/
Disallow: /whatsnew/
Disallow: /css/
Disallow: /js/
Disallow: /Library/
Disallow: /formtoemail/
Disallow: /Templates/
Disallow: /tmp/ # these will soon disappear
Disallow: /cc_thanks.html
Disallow: /index_static.html
Disallow: /comexp/routes/142/142_sked_all.php
Disallow: /comexp/routes/142/142_sked_ebwd.php
Disallow: /comexp/routes/142/142_sked_ebwe.php
Disallow: /comexp/routes/142/142_sked_wbwd.php
Disallow: /comexp/routes/142/142_sked_wbwe.php
Disallow: /comexp/routes/142/142_smap.php
Disallow: /comexp/routes/142/142_imap.php
Disallow: /comexp/routes/409/409_sked_all.php
Disallow: /comexp/routes/409/409_sked_downtown.php
Disallow: /comexp/routes/409/409_sked_glendale.php
Disallow: /comexp/routes/409/409_smap.php
Disallow: /comexp/routes/409/409_imap.php
Disallow: /comexp/routes/419/419_sked_all.php
Disallow: /comexp/routes/419/419_sked_chatsworth.php
Disallow: /comexp/routes/419/419_sked_downtown.php
Disallow: /comexp/routes/419/419_smap.php
Disallow: /comexp/routes/419/419_imap.php
Disallow: /comexp/routes/422/422_sked_all.php
Disallow: /comexp/routes/422/422_sked_hollywood.php
Disallow: /comexp/routes/422/422_sked_sanfernandovalley.php
Disallow: /comexp/routes/422/422_smap.php
Disallow: /comexp/routes/422/422_imap.php
Disallow: /comexp/routes/423/423_sked_all.php
Disallow: /comexp/routes/423/423_sked_encino.php
Disallow: /comexp/routes/423/423_sked_hollywood.php
Disallow: /comexp/routes/423/423_smap.php
Disallow: /comexp/routes/423/423_imap.php
Disallow: /comexp/routes/431/431_sked_all.php
Disallow: /comexp/routes/431/431_sked_downtown.php
Disallow: /comexp/routes/431/431_sked_westwood.php
Disallow: /comexp/routes/431/431_smap.php
Disallow: /comexp/routes/431/431_imap.php
Disallow: /comexp/routes/437/437_sked_all.php
Disallow: /comexp/routes/437/437_sked_culvercity.php
Disallow: /comexp/routes/437/437_sked_downtown.php
Disallow: /comexp/routes/437/437_smap.php
Disallow: /comexp/routes/437/437_imap.php
Disallow: /comexp/routes/438/438_sked_all.php
Disallow: /comexp/routes/438/438_sked_downtown.php
Disallow: /comexp/routes/438/438_sked_redondo.php
Disallow: /comexp/routes/438/438_smap.php
Disallow: /comexp/routes/438/438_imap.php
Disallow: /comexp/routes/448/448_sked_all.php
Disallow: /comexp/routes/448/448_sked_downtown.php
Disallow: /comexp/routes/448/448_sked_ranchopv.php
Disallow: /comexp/routes/448/448_smap.php
Disallow: /comexp/routes/448/448_imap.php
Disallow: /comexp/routes/534/534_sked_all.php
Disallow: /comexp/routes/534/534_sked_downtown.php
Disallow: /comexp/routes/534/534_sked_westla.php
Disallow: /comexp/routes/534/534_smap.php
Disallow: /comexp/routes/534/534_imap.php
Disallow: /comexp/routes/549/549_sked_all.php
Disallow: /comexp/routes/549/549_sked_burbank.php
Disallow: /comexp/routes/549/549_sked_glendale.php
Disallow: /comexp/routes/549/549_imap.php
Disallow: /comexp/routes/549/549_smap.php
Disallow: /comexp/routes/573/573_sked_all.php
Disallow: /comexp/routes/573/573_sked_encino.php
Disallow: /comexp/routes/573/573_sked_westwood.php
Disallow: /comexp/routes/573/573_smap.php
Disallow: /comexp/routes/573/573_imap.php
Disallow: /comexp/routes/574/574_sked_all.php
Disallow: /comexp/routes/574/574_sked_encino.php
Disallow: /comexp/routes/574/574_sked_lax.php
Disallow: /comexp/routes/574/574_smap.php
Disallow: /comexp/routes/574/574_imap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_all.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_wdam.php
Disallow: /comexp/routes/UnionBunker/unionbunker_sked_wdpm.php
Disallow: /comexp/routes/UnionBunker/unionbunker_gmap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_imap.php
Disallow: /comexp/routes/UnionBunker/unionbunker_smap.php
Disallow: /dash/routes/beachwood/beachwood_sked_all.php
Disallow: /dash/routes/beachwood/beachwood_sked_tohollywood.php
Disallow: /dash/routes/beachwood/beachwood_sked_tohollywood_sat.php
Disallow: /dash/routes/beachwood/beachwood_sked_tosunset.php
Disallow: /dash/routes/beachwood/beachwood_sked_tosunset_sat.php
Disallow: /dash/routes/beachwood/beachwood_imap.php
Disallow: /dash/routes/beachwood/beachwood_smap.php
Disallow: /dash/routes/beachwood/beachwood_gmap.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_all.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_north_sat.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_north_weekday.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_south_sat.php
Disallow: /dash/routes/boyleheights/boyleheights_sked_south_weekday.php
Disallow: /dash/routes/boyleheights/boyleheights_imap.php
Disallow: /dash/routes/boyleheights/boyleheights_gmap.php
Disallow: /dash/routes/boyleheights/boyleheights_smap.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_all.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_ebwd.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_ebwe.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_wbwd.php
Disallow: /dash/routes/chesterfield/chesterfield_sked_wbwe.php
Disallow: /dash/routes/chesterfield/chesterfield_imap.php
Disallow: /dash/routes/chesterfield/chesterfield_gmap.php
Disallow: /dash/routes/chesterfield/chesterfield_smap.phpchesterfield_sked_all.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_all.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_ccw.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_ccwsat.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_cw.php
Disallow: /dash/routes/crenshaw/crenshaw_sked_cwsat.php
Disallow: /dash/routes/crenshaw/crenshaw_imap.php
Disallow: /dash/routes/crenshaw/crenshaw_gmap.php
Disallow: /dash/routes/crenshaw/crenshaw_smap.php
Disallow: /dash/routes/downtown/downtown_sked_all.php
Disallow: /dash/routes/downtown/downtown_sked_a.php
Disallow: /dash/routes/downtown/downtown_sked_b.php
Disallow: /dash/routes/downtown/downtown_sked_d.php
Disallow: /dash/routes/downtown/downtown_sked_e.php
Disallow: /dash/routes/downtown/downtown_sked_f.php
Disallow: /dash/routes/downtown/downtown_we_sked_e.php
Disallow: /dash/routes/downtown/downtown_we_sked_f.php
Disallow: /dash/routes/downtown/downtown_imap.php
Disallow: /dash/routes/downtown/downtown_gmap.php
Disallow: /dash/routes/downtown/downtown_smap.php
Disallow: /dash/routes/elsereno/elsereno_sked_all.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbsat.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbsun.php
Disallow: /dash/routes/elsereno/elsereno_sked_nbwd.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbsat.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbsun.php
Disallow: /dash/routes/elsereno/elsereno_sked_sbwd.php
Disallow: /dash/routes/elsereno/elsereno_gmap.php
Disallow: /dash/routes/elsereno/elsereno_imap.php
Disallow: /dash/routes/elsereno/elsereno_smap.php
Disallow: /dash/routes/fairfax/fairfax_sked_all.php
Disallow: /dash/routes/fairfax/fairfax_sked_robertson.php
Disallow: /dash/routes/fairfax/fairfax_sked_robertson_sat.php
Disallow: /dash/routes/fairfax/fairfax_sked_wilshire.php
Disallow: /dash/routes/fairfax/fairfax_sked_wilshire_sat.php
Disallow: /dash/routes/fairfax/fairfax_gmap.php
Disallow: /dash/routes/fairfax/fairfax_imap.php
Disallow: /dash/routes/fairfax/fairfax_smap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_all.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_ebwd.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_ebwe.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_wbwd.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_sked_wbwe.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_gmap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_imap.php
Disallow: /dash/routes/HighlandPark_EagleRock/highland_smap.php
Disallow: /dash/routes/hollywood/hollywood_sked_all.php
Disallow: /dash/routes/hollywood/hollywood_sked_ccw.php
Disallow: /dash/routes/hollywood/hollywood_sked_ccw_ssh.php
Disallow: /dash/routes/hollywood/hollywood_sked_cw.php
Disallow: /dash/routes/hollywood/hollywood_sked_cw_ssh.php
Disallow: /dash/routes/hollywood/hollywood_gmap.php
Disallow: /dash/routes/hollywood/hollywood_imap.php
Disallow: /dash/routes/hollywood/hollywood_smap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_all.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_larchmont.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_north.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_sked_south.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_gmap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_imap.php
Disallow: /dash/routes/hollywood_wilshire/hollywilshire_smap.php
Disallow: /dash/routes/KingEast/kingeast_sked_all.php
Disallow: /dash/routes/KingEast/kingeast_sked_ccw_sat.php
Disallow: /dash/routes/KingEast/kingeast_sked_ccwwd.php
Disallow: /dash/routes/KingEast/kingeast_sked_cw_sat.php
Disallow: /dash/routes/KingEast/kingeast_sked_cwwd.php
Disallow: /dash/routes/KingEast/kingeast_gmap.php
Disallow: /dash/routes/KingEast/kingeast_imap.php
Disallow: /dash/routes/KingEast/kingeast_smap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_all.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_ccwsd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_ccwwd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_cwsd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_sked_cwwd.php
Disallow: /dash/routes/Leimert_Slauson/leimert_gmap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_imap.php
Disallow: /dash/routes/Leimert_Slauson/leimert_smap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_all.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_ccwwd.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_ccwwe.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_cwwd.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_sked_cwwe.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_gmap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_imap.php
Disallow: /dash/routes/LincolnHeights_Chinatown/lincolnheights_smap.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_all.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_nbwd.php
Disallow: /dash/routes/LosFeliz/losfeliz_sked_sbwd.php
Disallow: /dash/routes/LosFeliz/losfeliz_gmap.php
Disallow: /dash/routes/LosFeliz/losfeliz_imap.php
Disallow: /dash/routes/LosFeliz/losfeliz_smap.php
Disallow: /dash/routes/Midtown/midtown_sked_all.php
Disallow: /dash/routes/Midtown/midtown_sked_nbwd.php
Disallow: /dash/routes/Midtown/midtown_sked_nbwe.php
Disallow: /dash/routes/Midtown/midtown_sked_sbwd.php
Disallow: /dash/routes/Midtown/midtown_sked_sbwe.php
Disallow: /dash/routes/Midtown/midtown_gmap.php
Disallow: /dash/routes/Midtown/midtown_imap.php
Disallow: /dash/routes/Midtown/midtown_smap.php
Disallow: /dash/routes/Northridge/northridge_sked_all.php
Disallow: /dash/routes/Northridge/northridge_sked_sat.php
Disallow: /dash/routes/Northridge/northridge_sked_wd.php
Disallow: /dash/routes/Northridge/northridge_gmap.php
Disallow: /dash/routes/Northridge/northridge_imap.php
Disallow: /dash/routes/Northridge/northridge_smap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_all.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_ccwwd.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_ccwwe.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_cwwd.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_sked_cwwe.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_gmap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_imap.php
Disallow: /dash/routes/PanoramaCity_VanNuys/pcityvannuys_smap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_all.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_northssh.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_northweekday.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_southssh.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_sked_southweekday.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_gmap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_imap.php
Disallow: /dash/routes/PicoUnion_EchoPk/picounion_smap.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_all.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbsat.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbsun.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_nbwd.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbsat.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbsun.php
Disallow: /dash/routes/SanPedro/sanpedro_sked_sbwd.php
Disallow: /dash/routes/SanPedro/sanpedro_gmap.php
Disallow: /dash/routes/SanPedro/sanpedro_imap.php
Disallow: /dash/routes/SanPedro/sanpedro_smap.php
Disallow: /dash/routes/Southeast/southeast_sked_all.php
Disallow: /dash/routes/Southeast/southeast_sked_ccwwd.php
Disallow: /dash/routes/Southeast/southeast_sked_ccwwe.php
Disallow: /dash/routes/Southeast/southeast_sked_cwwd.php
Disallow: /dash/routes/Southeast/southeast_sked_cwwe.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrnsat.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrnwd.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrssat.php
Disallow: /dash/routes/Southeast/southeast_sked_pdrswd.php
Disallow: /dash/routes/Southeast/southeast_gmap.php
Disallow: /dash/routes/Southeast/southeast_imap.php
Disallow: /dash/routes/Southeast/southeast_smap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_all.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_ccwsat.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_ccwwd.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_cwsat.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_sked_cwwd.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_gmap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_imap.php
Disallow: /dash/routes/VanNuys_StudioCty/vannuysstudiocity_smap.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_all.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_ccwsat.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_ccwwd.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_cwsat.php
Disallow: /dash/routes/VermontMain/vermontmain_sked_cwwd.php
Disallow: /dash/routes/VermontMain/vermontmain_gmap.php
Disallow: /dash/routes/VermontMain/vermontmain_imap.php
Disallow: /dash/routes/VermontMain/vermontmain_smap.php
Disallow: /dash/routes/Watts/watts_sked_all.php
Disallow: /dash/routes/Watts/watts_sked_ccwsat.php
Disallow: /dash/routes/Watts/watts_sked_ccwwd.php
Disallow: /dash/routes/Watts/watts_sked_cwsat.php
Disallow: /dash/routes/Watts/watts_sked_cwwd.php
Disallow: /dash/routes/Watts/watts_gmap.php
Disallow: /dash/routes/Watts/watts_imap.php
Disallow: /dash/routes/Watts/watts_smap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_all.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_ccwsat.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_ccwwd.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_cwsat.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_sked_cwwd.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_gmap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_imap.php
Disallow: /dash/routes/Wilmington_Bannings/wilmington_smap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_all.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_ccwsat.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_ccwwd.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_cwsat.php
Disallow: /dash/routes/Wilshire/wilshirekorea_sked_cwwd.php
Disallow: /dash/routes/Wilshire/wilshirekorea_gmap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_imap.php
Disallow: /dash/routes/Wilshire/wilshirekorea_smap.php
Disallow: /map/cemap_imap.php
Disallow: /map/cemap_smap.php
Disallow: /map/cemap_gmap.php
Disallow: /map/dashmap_smap.php
Disallow: /map/dashmap_gmap.php
Disallow: /map/dashmap_imap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_smap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_gmap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_imap.php
Disallow: /other/observatoryshuttle/observatoryshuttle_all.php
Disallow: /other/observatoryshuttle/observatoryshuttle_nbwe.php
Disallow: /other/observatoryshuttle/observatoryshuttle_sbwe.php
Disallow: /KML/
Disallow: /store_template.html
我注意到谷歌网站管理员工具目前看到该文件的案文：
# robots.txt for http://www.ladottransit.com/
User-agent: *
Disallow: /sphider-1.3.5/
Disallow: /whatsnew/
Disallow: /css/
Disallow: /js/
Disallow: /Library/
Disallow: /formtoemail/
Disallow: /Templates/
Disallow: /tmp/ # these will soon disappear
Disallow: /cc_thanks.html
Disallow: /index_static.html
这是文件，直到这个状态早上当我添加其他不允许的文件时。直接链接到robots.txt会显示全文。我认为谷歌没有重新编制索引，这就是为什么它看到了旧文件。但是我成功地通过Sphider管理员手动重新编制索引，似乎忽略了这些规则。
任何人都可以看到问题所在？我很难过。
我无法看到链接的论坛帖子中提到的任何补丁。帮我。 – Ameer