问题

老高最近遇到一个需求，linux\centos下，使用selenium技术抓取数据。本来很简单的问题，但是由于内存限制，安装X window不现实，所以一个BT的想法诞生了，是否可以在centos命令行界面运行一个虚拟的桌面，然后使用selenium控制Firefox浏览器完成一些操作，Firefox运行在虚拟的桌面中，一切操作都在命令行中完成。

Google之，发现了Xvfb，他可以新建一个虚拟的X窗口，再配合python的pyvirtualdisplay，简直就是神器！

安装

centos下：

# 安装Xvfb和pyvirtualdisplay
yum install xorg-x11-server-Xvfb
pip install pyvirtualdisplay

安装firefox和selenium

yum install firefox
pip install selenium

代码

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

参考网站：

http://selenium-python.readthedocs.org/en/latest/getting-started.html http://nullege.com/codes/search/selenium.webdriver.Remote.find_elements_by_class_name http://www.opsview.com/forum/opsview-core/how-do-i/how-do-i-install-selenium-centos-server https://gist.github.com/textarcana/5855427 http://scraping.pro/use-headless-firefox-scraping-linux/ http://serverfault.com/questions/363827/how-can-i-run-firefox-on-centos-with-no-display https://realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/ https://pypi.python.org/pypi/selenium

http://selenium.googlecode.com/git/docs/api/py/selenium/selenium.selenium.html#module-selenium.selenium

http://www.ibm.com/developerworks/cn/opensource/os-php-designptrns/ http://www.cnblogs.com/fnng/p/3230768.html http://www.cnblogs.com/fnng/p/3157639.html http://www.cnblogs.com/fnng/p/3157639.html

删除自带web、数据库、php

yum remove httpd* php* mysql-server mysql* php-mysql -y

删除不需要的程序

yum remove Deployment_Guide-en-US finger cups-libs cups ypbind bluez-libs desktop-file-utils ppp rp-pppoe wireless-tools irda-utils sendmail* samba* talk-server finger-server bind* xinetd nfs-utils nfs-utils-lib rdate fetchmail eject ksh mkbootdisk mtools syslinux tcsh startup-notification talk apmd rmt dump setserial portmap yp-tools -y
yum groupremove "Mail Server" "Games and Entertainment" "X Window System" "X Software Development" "Development Libraries" "Dialup Networking Support" "Games and Entertainment" "Sound and Video" "Graphics" "Editors" "Text-based Internet" "GNOME Desktop Environment" "GNOME Software Development" -y
yum -y groupremove "FTP Server" "PostgreSQL Database client" "PostgreSQL Database server" "MySQL Database server" "MySQL Database client" "Web Server" "Office Suite and Productivity" "E-mail server" "Ruby Support" "Printing client"

关闭selinux

setenforce 0
sed -i 's/^SELINUX=.*$/SELINUX=disabled/' /etc/selinux/config

对系统进行一些优化

sed -i 's/^id:.*$/id:3:initdefault:/' /etc/inittab
init q
[ -z "`cat ~/.bashrc | grep ^PS1`" ] && echo 'PS1="[e[37;40m][[e[32;40m]u[e[37;40m]@h [e[35;40m]W[e[0m]]\$ "' >> ~/.bashrc
sed -i 's/^HISTSIZE=.*$/HISTSIZE=10/' /etc/profile
[ -z "`cat ~/.bashrc | grep history-timestamp`" ] && echo "export PROMPT_COMMAND='{ msg=$(history 1 | { read x y; echo $y; });user=$(whoami); echo $(date "+%Y-%m-%d %H:%M:%S"):$user:`pwd`/:$msg ---- $(who am i); } >> /tmp/`hostname`.`whoami`.history-timestamp'" >> ~/.bashrc
[ -z "`cat /etc/security/limits.conf | grep 'nproc 65535'`" ] && cat >> /etc/security/limits.conf <> /etc/rc.local
[ "$(hostname -i | awk '{print $1}')" != "127.0.0.1" ] && sed -i "s@^127.0.0.1(.*)@127.0.0.1   `hostname` 1@" /etc/hosts
[ -z "`cat /etc/pam.d/system-auth | grep 'pam_tally2.so'`" ] && sed -i '4a auth        required      pam_tally2.so deny=5 unlock_time=180' /etc/pam.d/system-auth

输入vi自动打开vim

[ -z "`cat ~/.bashrc | grep 'alias vi='`" ] && sed -i "s@alias mv=(.*)@alias mv=1nalias vi=vim@" ~/.bashrc && echo 'syntax on' >> /etc/vimrc

升级系统

yum -y update
yum clean all

美化命令行，其实在之前优化那里就美化了，这里只是让它生效而已

. /etc/profile
. ~/.bashrc

删除不需要的服务，这里自行考虑，反正我是只保留几个的~

service modules_dep stop
chkconfig modules_dep off
chkconfig --del modules_dep
service netconsole stop
chkconfig netconsole off
chkconfig --del netconsole
service netfs stop
chkconfig netfs off
chkconfig --del netfs
service nscd stop
chkconfig nscd off
chkconfig --del nscd
service quota_nld stop
chkconfig quota_nld off
chkconfig --del quota_nld
service rdisc stop
chkconfig rdisc off
chkconfig --del rdisc
service restorecon stopd
chkconfig restorecond off
chkconfig --del restorecond
service saslauthd stop
chkconfig saslauthd off
chkconfig --del saslauthd
service snmpd stop
chkconfig snmpd off
chkconfig --del snmpd
service snmptrapd stop
chkconfig snmptrapd off
chkconfig --del snmptrapd
service gpm stop
chkconfig gpm off
chkconfig --del gpm
service iscsi stop
chkconfig iscsi off
chkconfig --del iscsi
service iscsid stop
chkconfig iscsid off
chkconfig --del iscsid
service lm_sensors stop
chkconfig lm_sensors off
chkconfig --del lm_sensors
service lvm2-monitor stop
chkconfig lvm2-monitor off
chkconfig --del lvm2-monitor
service mcstrans stop
chkconfig mcstrans off
chkconfig --del mcstrans
service messagebus stop
chkconfig messagebus off
chkconfig --del messagebus
service multipathd stop
chkconfig multipathd off
chkconfig --del multipathd
service netconsole stop
chkconfig netconsole off
chkconfig --del netconsole
service netfs stop
chkconfig netfs off
chkconfig --del netfs
service netplugd stop
chkconfig netplugd off
chkconfig --del netplugd
service nscd stop
chkconfig nscd off
chkconfig --del nscd
service rawdevices stop
chkconfig rawdevices off
chkconfig --del rawdevices
service rdisc stop
chkconfig rdisc off
chkconfig --del rdisc
service restorecond stop
chkconfig restorecond off
chkconfig --del restorecond
service ntpd stop
chkconfig ntpd off
chkconfig --del ntpd
service ip6tables stop
chkconfig ip6tables off
chkconfig --del ip6tables
service ntpdate stop
chkconfig ntpdate off
chkconfig --del ntpdate
service portreserve stop
chkconfig portreserve off
chkconfig --del portreserve
service udev-post stop
chkconfig udev-post off
chkconfig --del udev-post
service exim stop
chkconfig exim off
chkconfig --del exim

转自http://shuang.ca/centos-optimizer/

放假闲来无事，研究了一下centos 7.0，发现变化真大。。。。。。ifconfig竟然都被废了，以后看IP就要用ip addr了。

下面记录一下需要注意的地方。

阅读剩余部分

源码安装：

点击这里下载源码

解压前线装一下必要的包

    yum install python-devel mysql-devel zlib-devel openssl-devel

解压

    tar xf MySQL-python-1.2.3.tar.gz
    cd MySQL-python-1.2.3

修改mysite.cfg，指定mysql-config的路径

使用whereis mysql-config找路径

    vi mysite.cfg
    #找到mysql_config = XXX
    改为你的路径

继续安装

    python setup.py build
    python setup.py install

工具安装：

sudo easy_install mysql-python

sudo pip install mysql-python

出现错误的解决办法
---

```bash
vi ~/.bash_profile
# add
export DYLD_LIBRARY_PATH='/usr/local/mysql/lib';
PATH="$PATH:/usr/local/mysql/bin"

enjoy~

Win下请直接下载编译好的安装包：

请输入链接描述

有时候centos使用yum命令的时候软件下载速度会很慢，还好国内有很多镜像站供大家选择，在此博主为大家总结了一些centos国内镜像站，如果你需要下载centos的iso文件或者相关软件，可以到以下网站下载，速度相当快！

当然，如果你想直接替换掉centos系统内置的yum源地址，可以参考这篇文章Centos源设置，将你的源设为比较快的地址！

起因

刚刚装好的centos网卡也配好了，就是解析不了mirror.centos.org，估计是DNS的问题，索性换了163源。

网易源地址

使用帮助

方法

很简单，注意备份即可

cd /etc/yum.repos.d
mv CentOS-Base.repo CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS5-Base-163.repo #centos5
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.163.com/.help/CentOS6-Base-163.repo #centos6
yum clean all
yum makecache

意外

在执行update的时候又报错

GPG key retrieval failed: [Errno 12] Timeout on http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6: (28, 'connect() timed out!')

貌似需要在源服务器验证RPM的合法性 本办法，只好祭出终极武器，host大法!

vi /etc/hosts

加入 74.53.10.146 mirror.centos.org 解决！

阿里源

阿里的速度也不错！

repo下载地址： http://mirrors.aliyun.com/repo/

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

# 根据系统版本下载对应版本的repo
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo

或

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo


# 更新缓存
yum clean all
yum makecache

webkitgtk+的编译要用到4.8+的GCC 人家ubuntu都有编译好的gcc。。。可怜我centos只能编译源代码了。。。

废话不说，速度干活

下载

先下载需要的GCC源码，http://gcc.gnu.org/

以gcc-4.8.3为例 先解压文件

cd ~
mkdir src
cd src
wget http://自己写吧
tar xf gcc-4.8.3.tar.gz

到这一步停下，还有软件包没有下载安装，难点重点就在这儿了，这三个软件包的安装顺序不能乱，而且安装后一个软件之前必须指定前面安装的路径，很麻烦！

下载源代码可以执行gcc-4.8.3/contrib下的download_prerequisites脚本，即可自动下载，到时候也好一起编译

阅读剩余部分