问题

老高最近遇到一个需求,linuxcentos下,使用selenium技术抓取数据。本来很简单的问题,但是由于内存限制,安装X window不现实,所以一个BT的想法诞生了,是否可以在centos命令行界面运行一个虚拟的桌面,然后使用selenium控制Firefox浏览器完成一些操作,Firefox运行在虚拟的桌面中,一切操作都在命令行中完成。

Google之,发现了Xvfb,他可以新建一个虚拟的X窗口,再配合python的pyvirtualdisplay,简直就是神器!

安装

centos下:

# 安装Xvfb和pyvirtualdisplay
yum install xorg-x11-server-Xvfb
pip install pyvirtualdisplay

安装firefox和selenium

yum install firefox
pip install selenium

代码

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

参考网站:

http://selenium-python.readthedocs.org/en/latest/getting-started.html
http://nullege.com/codes/search/selenium.webdriver.Remote.find_elements_by_class_name
http://www.opsview.com/forum/opsview-core/how-do-i/how-do-i-install-selenium-centos-server
https://gist.github.com/textarcana/5855427
http://scraping.pro/use-headless-firefox-scraping-linux/
http://serverfault.com/questions/363827/how-can-i-run-firefox-on-centos-with-no-display
https://realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/
https://pypi.python.org/pypi/selenium

http://selenium.googlecode.com/git/docs/api/py/selenium/selenium.selenium.html#module-selenium.selenium

http://www.ibm.com/developerworks/cn/opensource/os-php-designptrns/
http://www.cnblogs.com/fnng/p/3230768.html
http://www.cnblogs.com/fnng/p/3157639.html
http://www.cnblogs.com/fnng/p/3157639.html

标签: centos, python, Linux, headless, selenium

已有 6 条评论

  1. [...]linux无界面(headless)使用selenium抓取数据[...]

  2. 求教:运行demo时一直提示 raise WebDriverException("The browser appears to have exited " selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details. 是什么原因?

    1. 依赖包都安装了吗?

      1. 链接错误 大神为啥啊

        [root@spider ~]# python3 firefox.py
        Traceback (most recent call last):
        File "firefox.py", line 6, in
        driver = webdriver.Firefox(executable_path=r'./geckodriver')
        File "/usr/lib/python3.4/site-packages/selenium/webdriver/firefox/webdriver.py", line 152, in __init__
        keep_alive=True)
        File "/usr/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 98, in __init__
        self.start_session(desired_capabilities, browser_profile)
        File "/usr/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 188, in start_session
        response = self.execute(Command.NEW_SESSION, parameters)
        File "/usr/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
        self.error_handler.check_response(response)
        File "/usr/lib/python3.4/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
        raise exception_class(message, screen, stacktrace)
        selenium.common.exceptions.WebDriverException: Message: connection refused

    2. 今天早上又可以了,可能是由于我手动设置了xvfb。写的很清楚,赞

      1. cheyo Hou cheyo Hou

        我现在也出现了如下这个错误。现在整个程序卡在这里,进行不下去。请问你是如何解决的? Xvfb除在文章中的python代码外,需要手工启动或者配置吗?

        相关依赖包都装了

        多谢!

        selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

添加新评论