开始（Start)
控制面板（Control Panel）
地区和语言（Region and Languages）
在弹出的对话框里选择管理（Administrative），如下图所示：
选择设置系统场所（Change Sysem locale ... )
然后选择中文。（Chinese-RPC）如下图所示：
设置完成后系统需要重启。
重启之后就一切正常了。
Hadoop as Big Data Platform
Hadoop was designed as a big data system; it is distributed
and can handle massive data sets by coordinating the computation power of a large
number of commodity machines. I will introduce the Hadoop big data framework.
The Hadoop Big Data Framework
Hadoop is one of the representative approaches for dealing
with big data challenges. Compared with traditional data frameworks, Hadoop redefines
the way data is managed and analyzed by leveraging the power of computing
resources composed of commodity hardware.
What is Hadoop?
Hadoop was first developed as a big data processing platform
from 2006 at Yahoo! The idea is based on Google’s MapReduce, which was first
published by Google based on their proprietary MapReduce implementation. Hadoop
is implemented using java and was open sourced under Apache license. In the
past few years, Hadoop has become a widely used platform and runtime
environment for the deployment of Big Data applications.
Hadoop Properties
Hadoop was designed to be parallel and resilient. It was
built to work on commodity hardware and can automatically recover from failures.
Despite of its success as being a big data platform, Hadoop is notorious for its
difficulty in configuration and management. A few Hadoop oriented companies
were founded to provide enterprise ready Hadoop distributions and Hadoop based
big data solutions.
Although designed for processing big data, sometimes the distributed
parallel computing framework can be useful for dealing with small data but big computation
problems. One such example is the computation of pi (π) shipped with the Hadoop
sample package.
Next I will introduce the data management and
computation framework, identifying their differences from traditional data systems.
Data Management with HDFS
Hadoop Distributed File System (HDFS) is the default
filesystem of Hadoop. It was designed as a distributed filesystem that provides
high-throughput access to application data. Data on HDFS is stored as data blocks.
These data blocks are replicated on several nodes and checksums of the blocks
are computed. In case of checksum error or system failure, erroneous or lost
data blocks can be recovered from backup blocks located on other nodes.
HDFS has two types of nodes, NameNode and DataNode. NameNode
keeps track of the metadata such as location of data blocks in main memory. DataNode
holds data blocks, and communicates with clients for reading and writing of the
blocks. In addition, DataNode periodically reports the list of its hosting
blocks to the NameNode.
Some HDFS configurations contain a SecondaryNameNode, which
maintains a backup of the HDFS filesystem image, once the NameNode crashes, the
whole filesystem can be recovered from the SecondaryNameNode. Unfortunately, SecondaryNameNode
is not a hot backup of the NameNode. Newer versions of Hadoop have introduced a
feature called high availability (HA), which has two or more NameNodes, one of
which is the primary and others are hot backup of the primary. When the primary
NameNode fails, one of the backups can automatically become the new primary NameNode,
reducing the system downtime to the minimum.
Unlike traditional database systems, where data records are
managed by a DataBase Management System (DBMS), no schema constraints exist
among the data sets. Due to this reason, HDFS is also known as a NoSQL
database.
Data Processing with MapReduce
MapReduce provides a programming model that transforms a
complex computation into the computation over a set of keys and values. Applications
coordinate the processing of tasks on a cluster of nodes by scheduling jobs,
monitoring activity, and re-executing failed tasks.
 |
| Figure 1. The MapReduce Data Processing Framework. |
The MapReduce framework has two types of nodes, Master Node
and Slave Node. Jobtracker is the daemon on a master node, and Tasktracker is
the daemon on a slave node. The master node is the manager of MapReduce jobs;
it splits jobs into smaller tasks, including map tasks and reduce tasks. After
splitting, the master node assigns tasks to slave nodes to run. When a slave
node gets a task from master node, the Tasktracker on the slave node will fork
a java process to run the task. The Tasktracker is also responsible for
tracking and reporting the progress of individual tasks. MapReduce takes data
locality into consideration so that a node will process its locally hosted data
first. This reduces the data transfer cost over the network. It is different
from traditional data processing system, which retrieves data for processing in
a centralized system.
The anatomy of a MapReduce job is described in the following
Figure. Multiple mappers on the slave nodes are executed in parallel. Results from
mappers will be buffered on local machine. Once some or sometimes all of the
mapper tasks have finished, the random shuffling process begins, which
aggregates the mapper outputs and shuffle intermediate partitions into reducer machine(s).
The reducers will run on the partitioned data generating final results which
are written to HDFS. Once the job finishes, the result will reside in multiple
files, depending on the number of reducers used in the job.
By default MapReduce programs are written in java. The
programming model of a basic MapReduce job is simple and straightforward. Assuming
all settings using default, application programmers only need to implement a
mapper function and a reducer function, the framework will automatically handle
the input, output and shuffle of data. Programmers don’t need to consider task
failures or resource contention either. And if your programming language is not
java, MapReduce streaming makes it even simpler to use MapReduce.
The Hadoop Big Data Ecosystem
The name Hadoop appears as the synonym of Google’s
MapReduce. Gradually, with more and more features and subsystems being added,
Hadoop is becoming the umbrella of a full-fledged big data ecosystem. Besides
the distributed file system (HDFS) and computing framework (MapReduce) that we have
introduced in the previous section, the ecosystem also includes common
utilities, a column oriented data storage table (HBase), high level data
management systems (Pig and Hive), a big data analytics library (Mahout), a
distributed coordination system (Zookeeper), a data integration framework
(Sqoop) and a workflow management module (Oozie). As the ecosystem grows, more complementary
services or higher-level abstractions systems are being added.
 |
| Figure 2. The Hadoop Big Data System |
Common
Hadoop common is a collection
of components and interfaces for the foundation of Hadoop based big data
platform. It provides the following modules: interfaces for distributed
filesystem and I/O operations, general parallel computation interfaces, logging
and security management modules.
HBase
HBase is an open source,
distributed, versioned, column oriented data store. It was built on top of
Hadoop and HDFS. HBase supports random, real time access to big data. It can
scale to be hosting very large table, containing billions of rows and millions
of columns.
Mahout
Mahout is an open source scalable machine learning library
based on Hadoop. It has a very active community and is still under active
development. Currently, the library supports four use cases: recommendation
mining, clustering, classification and frequent item set mining.
Pig
Apache Pig is a high level
system for expressing big data analysis programs. It supports big data by
compiling the pig statement into a sequence of MapReduce jobs. Pig uses Pig
Latin as the programming language, which has the feature of easy to program,
build in optimization for execution and extensibility.
Hive
Hive is a high level system for management
and analysis of big data stored in Hadoop based systems. It uses a SQL-like
language called HiveQL. Similar to Apache Pig, the Hive runtime engine
translates the HiveQL statements into a sequence of MapReduce jobs for
execution.
ZooKeeper
ZooKeeper is a centralized
coordination service for large scale distributed systems. It maintains the configuration
and naming information and provides distributed synchronization and group
services for applications in distributed environment.
Sqoop
Hadoop was designed to process
data on non-relational database systems (such as HDFS and HBase). Sometimes, there
are requirements of integrating data on relational database systems with data
on non-relational ones. Sqoop is an open source tool for extracting structured
data from relational database to non-relation data systems.
Oozie
Oozie is a scalable workflow management and coordination
service for Hadoop jobs. It is data aware and manages and coordinates jobs
based on their dependencies. In addition, Oozie has been integrated with the
Hadoop stack and can support all types of Hadoop jobs.
Hadoop Alternatives
What is Hadoop NOT for?
Hadoop has been successful as a big data platform, but this
does not mean that it is a good fit for all data problems. For example, Hadoop
is not an optimal choice for the following cases:
- small structured data sets that require interactive queries
- data processing that requires transaction
- streaming data analytics
- real time data analytics
Especially, Hadoop was designed as a batch processing engine.
Sometimes, real time processing of the data is required. For example, online
big data querying, stream data processing and interactive big data analytics
all requires low latency responses.
As a big data solution manager, you always need to consider
a lot of technical and non-technical situations before making the final
decision. For example, you need to consider the ease of use and management of
the system, the functional ability of meeting data processing needs and cost of
system deployment etc. Fortunately, as the big data industry develops, alternatives
of Hadoop are emerging. In the next section, we will introduce these Hadoop
alternatives.
Next, I am going to introduce a few Hadoop alternatives,
including improvements over current Hadoop implementation and alternative big
data implementations that aims to tackle the fundamental drawbacks of Hadoop.
As open source software, Hadoop is difficult to configure and
manage, mainly due to the instability of the software and the lack of proper documentation
and support. To work as a Hadoop system administrator, you need to have
sufficient Unix/Linux and network management skills. Sometimes even for an
experience system administrator, it is still hard to configure a large cluster
in limited time. Fortunately, if your organization has sufficient budget, there
are several Hadoop oriented companies that provide enterprise ready Hadoop
solutions. We will introduce several such companies including Cloudera, Hadapt,
MapR and Horntonworks.
On the other hand, Hadoop was not designed to work on real
time big data problems. We will introduce Spark and Storm as alternatives to
deal with such problems. In the end, we introduce Message Passing Interface
(MPI) and High Performance Computing Cluster (HPCC).
Enterprise Hadoop
Cloudera is one of the first few companies that do
enterprise Hadoop big data solutions. This company provides Hadoop consulting,
training and certification services. It is also one of the biggest contributors
of the Hadoop code base. The Cloudera big data solution provides Cloudera
Desktop as cluster manager. It simplifies the installation, management and
monitoring of the Hadoop clusters. You can visit the corporate website of
Cloudera through www.cloudera.com.
Hadapt differentiate itself from the other Hadoop oriented
companies by the goal of integrating structured, semi-structured and
unstructured data into a uniform data manipulation platform. The Hadoop based
data platform by Hadapt unifies SQL and Hadoop which makes it easy to handle
different variety of data. You can visit the Hadapt corporate website at http://hadapt.com/.
Other well-known Hadoop companies include MapR and
Hortonworks, which are founded to provide more stable Hadoop distribution and
Hadoop based big data solutions.
Spark
Spark is a real time in memory big data processing platform.
It can be up to 40 times faster than Hadoop. So it is ideal for iteration
intensive machine learning and real time online big data analytics. Spark can
be integrated with Hadoop, and the Hadoop compatible storage APIs enables it to
access any Hadoop supported systems such as HDFS.
Another famous real time big data processing platform is
Storm, which is developed and open-sourced by Twitter.
MPI
MPI is a library specification for message passing.
Different from Hadoop, MPI was designed for high performance on both massively
parallel machines and on workstation clusters.
In addition, MPI lacks fault tolerance, and performance will be bounded
when data becomes large.
HPCC
HPCC is an open source big data platform developed by
LexisNexis Risk Solutions. It achieves high performance by clustering commodity
hardware. The system includes configurations for both parallel batch processing
and high performance online query applications using indexed data files.
The HPCC platform contains two cluster processing subsystems,
Data Refinery subsystem and Data Delivery subsystem. The Data Refinery
subsystem is responsible for the general processing of massive raw data and the
Data Delivery subsystem is responsible for the delivery of clean data for online
queries and analytics.
Summary
As big data continues to flood the whole world, handling of
big data becomes important. Hadoop becomes successful as a resilient and powerful
parallel cluster computing framework on commodity hardware. And the Hadoop
based big data ecosystem, including a number of subsystems and services, are
being widely accepted by the industry. Hadoop alternatives can help us deal with problems that the open source community Hadoop is not an optimal choice.
Big Data
In the past few years, we have witnessed the rapid increase of all kinds of data, from social network data continuously created by millions of users to large corporate transactional data and to real time streaming sensor data from ubiquitous sensors in our surrounding environment etc. Then, what is Big Data? And what does Big Data look like? Is Big Data really that critical and valuable for business players to make it the new frontier for business innovations? In this post, I will formally define the big data problem, identifying its features, discussing its value for organizations and in the end introducing challenges of managing and using big data.
What is Big Data?
Today, every organization across the globe is faced with an
unprecedented growth of data. The most general definition of big data is that
data with sizes go beyond the ability of commonly-used software tools to
collect, manage, and process within a tolerable elapsed time.
What does big data look like?
The most important and obvious property of big data is that
the data is orders of magnitude larger than data managed in traditional storage
and analytical systems. More formally, data can be described with three features: volume, velocity, variety. And for Big Data, the size of data is so big or
data moves so fast or there are so many varieties of data sources that
traditional data management and analytical systems cannot handle.
Volume
The volume of data generated has skyrocketed in the past
decade, and the measurement rises from megabytes to gigabyte, to terabytes, to petabytes
and to the historical exabytes now. Today, it is common for production Big Data
implementations to process petabytes and even exabytes of data on a daily
basis. And the data size is expected to be measured in zettabytes in the next
few years and predicted to double every two years.
Some examples of big data:
- The Human Genomes Project has generated more than 200 terabytes of data, which is equivalent to more than 30,000 standard DVDs.
- Microsoft’s search engine hosts over 100 petabytes of data to deliver high quality search results.
- Billions of pieces of information were created by more than 600 million Facebook users every day.
- The online game company, Zynga, processes 1 petabyte of content for players every day.
- With the rapid growth of network devices and internet users, the internet will soon have a daily traffic of bigger than one exabytes.
Velocity
The second important feature of Big Data is that it is being
generated in a much faster speed than ever before. Actually, in a lot of use
cases, data is generated in real time. And many Big Data use cases
require the processing of a real-time data stream to make time-sensitive
decisions.
An example of high velocity data is that data systems process click streams from web sites, and make real time updates to the
contents to serve the users in a timely manner.
Variety
The third feature of Big Data is the variety of data, which
is largely caused by the variety of data sources. For example, the increased use of digital cameras and smart phones makes it much easier to generate high
definition (HD) images and videos; ubiquitous sensors, such as utility meters,
traffic and security cameras and medical devices are becoming important sources of big
data etc.
The increased variety of big data has changed the format of
data. Traditionally, the majority of data is well formatted based on some
well-designed schema. In the big data world, unstructured data becomes
dominate, comprising of more than 80% of the whole data sets. Unstructured data
usually comes in many formats such as text, image and video etc. Sometimes, companies
need to integrate information from multiple data sources, for example, from
third parties.
Does Big Data Means Big Value for Business Players?
Today, more and more companies and joining the Big Data battle field. Amazon, Microsoft, Google, Facebook, IBM etc are releasing their big data systems and publishing their own perspectives on Big Data. So, why, in the past few years, the Big Data landscape is becoming so interesting to attract so many IT giants to join this community? Well, from their perspectives, it is the value of Big Data. They expect Big Data to serve so important role in the operation of businesses, including, optimizing corporate management, identifying and extracting new values from their customers for better customer services, or in other words, Big Data can help them to get richer, deeper,
and more accurate insights to help differentiate themselves from their
competitors and gain competitive advantage.
Big data is also an important source
of innovation for scientific research, for example, the data from human
genome project has the potential of revealing mechanisms of many complicated
diseases. Due to the great potential of big data, the US Federal Agencies are committing more than $200 million to a collaborative effort to develop core technologies and other resources needed by researchers to manage and analyze enormous data sets.
In summary, big data can unlock significant value; not only help companies
do better business but also help scientists tackle hard scientific problems.
The Value of Big Data from Technical Perspective
For the past decades, Business Intelligence (BI) has been helping companies to reveal values from their operation data. As the data is changing from "Small" and well structured data to "Big Data", will it change, or more radically, revolutionize the way to do Business Intelligence? The answer is "Yes". Then, how can we use Big Data, can we just simply fit the Big Data into our legacy Business Intelligence systems? Although, the answer will depend on the scale and power of your system and properties data is, generally, you can not do it. Or if you follow this path, your will suffer, because most of the legacy system are not designed to handle such big data in a graceful manor. We will discuss the challenges of handling Big Data later.Diving to Learning Theory
According to the "Big Data Theorem", the more data we have, the more precise and stable the pattern that we can learn from the data will be. In learning theory, if we have a small data set, we can not use very complicated models, otherwise, we will suffer from overfitting (the model fits almost precisely to our learning data set, leading to very small training error). Now, Big Data can help us alleviate this problem. When data increases, we can use more complicated model (for example, model with much higher order), which in the end can lead to more details of the patterns in the data. Figure 1 shows the comparison of learning errors with different model complexity with increasing data set size.
 |
| Figure 1. Learning Error Comparison for Different Model Complexity with increasing Data Size. [1] |
From Figure 1, we can see that, if the data set is small, complex model (here is H10 with order of 10) will suffer from serious over fitting (the testing error is too large and the training error is to small); and the simple model (H2 with highest order of 2) does suffer from overfitting. But when the data set increases, the complex model will converge to lower errors than simple model, meaning that complex models incurs lower bias than simple ones. This the beauty of Big Data, which enables us to use more complex models leading to more faithful fitting of the patterns in the data.
Big Data Processing Challenges
The tremendous opportunities to gain new and exciting value
from big data are compelling for most organizations, but the challenge of
managing and transforming it into insights requires new approaches to deal with the dynamic data sources and multiple contexts for big data.
The fact that big data contains mostly unstructured data
makes traditional database systems no longer a good fit for big data operations.
We need a new system that can deal with the massive unstructured data. On the
other hand, structured data will continue to be critical for organizations. Thus
the integration of data stored in traditional systems with the big data system becomes
a concern when designing big data systems.
Table 1 is a comparison of traditional data system
and big data system:
Operations
|
Traditional Data System
|
Big Data System
|
Data Storage
|
Centralized relational database (SQL) storage and
management
|
Distributed, non-relational (NoSQL) storage.
|
Data Processing
|
Centralized processing on single computer
|
Batch and real time, Distributed, parallel processing on
large cluster
|
Table 1. Comparison of Traditional Data System and Big Data System
Hadoop was designed as a big data system; it is distributed
and can handle massive data sets by coordinating the computation power of a large
number of commodity machines. In the next post, I will introduce Hadoop and explain the features that enable its success in the Big Data community.
References
[1] Learning from Data, Slides of Overfitting. http://bit.ly/13nSL5F
[2] The humane genome project data available on AWS. http://1.usa.gov/Xqqs3j
[3] NSF big data initiative. http://1.usa.gov/ZCFZN3
|
Linux常用命令总结
Linux命令总结 (updated Jan. 9th 2013).
- Install a printer from the web: localhost:631
- To play encrypted DVD using under linux, you need the libreadcss library.
- nl ，给文件编上行号；
- tac，将文件以逆序行号输出；
- 删除以-开头的文件：rm -- -foo 或 rm ./-foo
- 内容替换命令：tr (翻译和替换命令)，例如将文件中的大写字母替换称小写字母使用的命令是：cat foo | tr A-Z a-z
- nice 用来调整进程的优先级，默认情况下进程的优先级是10，优先级范围是[-20--19].
- banner命令，将输入的字符以大标题的形式显示出来。
- finger命令显示用户的信息。
- 给本系统或其他系统的用户发送邮件，例如：$ mail xxx@gmail.com 或：$ mail xxx,(其中xxx是本地一个用户。)
- write命令，给某个用户发送及时消息，用户必须在线，例如：$ write gsm, $ write xxx@gmail.com
- wall命令向系统中的所有用户发送广播消息，例如：$ wall "hello, good day"
- talk命令，向某个用户发送聊天邀请。例如: $ talk lili
- pg命令用来翻页阅读文件，例如 $ pg file_name
- cat -vte将不可见字符显示为特殊字符；-v 将非打印字符显示为可见字符，-t将tab键解释为^I，-e在行的末尾添加一个$.
- lsof 查看打开的文件，包括各种文件，例如pipe，socket，各种库文件等，如果想查看一个进程打开的文件的话这个命令就很有用了，例如查看2098号进程打开的文件，使用的命令为lsof -pn 2098
- lwp-download, lwp-mirror and lwp-rget commands can be used to retrieve website using (http) requests.
- sudo command can let common system users work as root on some privileged commands such as services, network management and so on.
- cut 命令可以将一个复杂的字符串根据指定的分隔符分割成几个部分，然后分别对这几个部分进行处理；
- join 将两个文件的内容合成一个；
- rev 将一个字符串进行反转，得到一个反向的字符串，这种操作对于取出带有全路径的文件名非常有用的。
- mtr ping和traceroute的组合体，用于检测网络的性能。
- atop，top的增强版，显示更多的信息，包括内存，cpu的summary等。
- scp rcp远程拷贝命令,例如scp hostname:/home/add/sohu ./
- nmap扫描机器的端口；
- 删除0字节文件 find -type f -size 0 -exec rm -rf {} \;
- 查看进程按内存从大到小排列 ps -e -o "%C : %p : %z : %a"|sort -k5 -nr
- 按cpu利用率从大到小排列 ps -e -o "%C : %p : %z : %a"|sort -nr
- 打印说cache里的URL : grep -r -a jpg /data/cache/* | strings | grep "http:" | awk -F'http:' '{print "http:"$2;}'
- 查看http的并发请求数及其TCP连接状态： netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
- sed -i '/Root/s/no/yes/' /etc/ssh/sshd_config sed在这个文里Root的一行，匹配Root一行，将no替换成yes.
- 如何杀掉mysql进程： ps aux|grep mysql|grep -v grep|awk '{print $2}'|xargs kill -9 (从中了解到awk的用途) killall -TERM mysqld kill -9 `cat /usr/local/apache2/logs/httpd.pid` 试试查杀进程PID
- 显示运行3级别开启的服务: ls /etc/rc3.d/S* |cut -c 15- (从中了解到cut的用途，截取数据)
- 如何在编写SHELL显示多个信息，用EOF cat << EOF +---------+ | === Welcome to Tunoff services === | +-------+ EOF
- for 的巧用(如给mysql建软链接) cd /usr/local/mysql/bin for i in * do ln /usr/local/mysql/bin/$i /usr/bin/$i done
- 取IP地址： ifconfig eth0 |grep "inet addr:" |awk '{print $2}'|cut -c 6- 或者 ifconfig | grep 'inet addr:'| grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $1}'
- 内存的大小: free -m |grep "Mem" | awk '{print $2}'
- netstat -an -t | grep ":80" | grep ESTABLISHED | awk '{printf "%s %s\n",$5,$6}' | sort
- 查看Apache的并发请求数及其TCP连接状态：
- netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
- 因为同事要统计一下服务器下面所有的jpg的文件的大小,写了个shell给他来统计.原来用xargs实现,但他一次处理一部分,搞的有多个总和....,下面的命令就能解决啦. find / -name *.jpg -exec wc -c {} \;|awk '{print $1}'|awk '{a+=$1}END{print a}' CPU的数量（多核算多个CPU，cat /proc/cpuinfo |grep -c processor）越多，系统负载越低，每秒能处理的请求数也越多。
- CPU负载 # cat /proc/loadavg 检查前三个输出值是否超过了系统逻辑CPU的4倍。
- CPU负载 #mpstat 1 1 检查%idle是否过低(比如小于5%)
- 内存空间 # free 检查free值是否过低 也可以用 # cat /proc/meminfo
- swap空间 # free 检查swap used值是否过高 如果swap used值过高，进一步检查swap动作是否频繁： # vmstat 1 5 观察si和so值是否较大
- 磁盘空间 # df -h 检查是否有分区使用率(Use%)过高(比如超过90%) 如发现某个分区空间接近用尽，可以进入该分区的挂载点，用以下命令找出占用空间最多的文件或目录： # du -cks * | sort -rn | head -n 10
- 磁盘I/O负载: # iostat -x 1 2 检查I/O使用率(%util)是否超过100% 23 网络负载 # sar -n DEV 检查网络流量(rxbyt/s, txbyt/s)是否过高
- 网络错误 # netstat -i 检查是否有网络错误(drop fifo colls carrier) 也可以用命令：# cat /proc/net/dev
- 网络连接数目 # netstat -an | grep -E “^(tcp)” | cut -c 68- | sort | uniq -c | sort -n
- 进程总数 # ps aux | wc -l 检查进程个数是否正常 (比如超过250)
- 可运行进程数目 # vmwtat 1 5 列给出的是可运行进程的数目，检查其是否超过系统逻辑CPU的4倍
- 进程 # top -id 1 观察是否有异常进程出现
- 网络状态 检查DNS, 网关等是否可以正常连通
- 用户 # who | wc -l 检查登录用户是否过多 (比如超过50个) 也可以用命令：# uptime
- 系统日志 # cat /var/log/rflogview/*errors
- 检查是否有异常错误记录 也可以搜寻一些异常关键字，例如： # grep -i error /var/log/messages # grep -i fail /var/log/messages
- 核心日志 # dmesg 检查是否有异常错误记录
- 系统时间 # date 检查系统时间是否正确
- 打开文件数目 # lsof | wc -l 检查打开文件总数是否过多
- 日志 # logwatch –print 配置/etc/log.d/logwatch.conf，将 Mailto 设置为自己的email 地址，启动mail服务 (sendmail或者postfix)，这样就可以每天收到日志报告了。缺省logwatch只报告昨天的日志，可以用# logwatch –print –range all 获得所有的日志分析结果。可以用# logwatch –print –detail high 获得更具体的日志分析结果(而不仅仅是出错日志)。
- 杀掉80端口相关的进程 lsof -i :80|grep -v "PID"|awk '{print "kill -9",$2}'|sh
- 清除僵死进程。ps -eal | awk '{ if ($2 == "Z") {print $4}}' | kill -9
- tcpdump 抓包 ，用来防止80端口被人攻击时可以分析数据 # tcpdump -c 10000 -i eth0 -n dst port 80 > /root/pkts
- 然后检查IP的重复数 并从小到大排序 注意 "-t\ +0" 中间是两个空格: # less pkts | awk {'printf $3"\n"'} | cut -d. -f 1-4 | sort | uniq -c | awk {'printf $1" "$2"\n"'} | sort -n -t\ +0
- 查看有多少个活动的php-cgi进程: # netstat -anp | grep php-cgi | grep ^tcp | wc -l chkconfig --list | awk '{if ($5=="3:on") print $1}'
- kudzu查看网卡型号 kudzu --probe --class=network
- 设定默认的编辑器：$EDITOR 环境变量。
- 清空或创建一个文件 > file.txt
- 用ssh创建端口转发通道 ssh -N -L2001:remotehost:80 user@somemachine 这个命令在本机打开了2001端口，对本机2001端口的请求通过somemachine作为跳板，转到remotehost的 80端口上。实现效果跟术语反向代理是相似的，实际上就是端口转发，注意上面的描述涉及了3台主机，但当然somemachine可 以变成localhost。这个命令比较抽象，但有时候是很有用的，比如因为众所周知的原因国内的IP的80端口无法使用，又或者公司的防火墙只给外网开了ssh端口，需要访 问内部服务器一个web应用，以及需要访问某些限定了来源IP的服务，就可以用上这个方法了。举一个具体例子，运行：ssh -f -N -L 0.0.0.0:443:twitter.com:443 shell.cjb.netssh -f -N -L 0.0.0.0:80:twitter.com:80 shell.cjb.net 然后在/etc/hosts里面添加127.0.0.1 twitter.com.
- 重置终端 reset
- 在午夜的时候执行某命令 echo cmd | at midnight
- 远程传送麦克风语音 dd if=/dev/dsp | ssh username@host dd of=/dev/dsp
- 映射一个内存目录 mount -t tmpfs -o size=1024m tmpfs /mnt/ram
- 用diff对比远程文件跟本地文件 ssh user@host cat /path/to/remotefile | diff /path/to/localfile -
- 查看系统中占用端口的进程 netstat -tulnp
- Netstat是很常用的用来查看Linux网络系统的工具之一，这个参数可以背下来：
- -t: 显示TCP链接信息
- -u: 显示UDP链接信息
- -l: 显示监听状态的端口
- -n: 直接显示ip，不做名称转换
- -p: 显示相应的进程PID以及名称（要root权限）
- 如果要查看关于sockets更详细占用信息等，可以使用lsof工具。
- Linux command line editing:
- Ctrl-k kill from cursor to the end of line.
- Ctrl-u kill from cursor to the beginning of line.
- Ctrl-y paste content to cursor position.
- Ctrl-t exchange character.
- Alt-t exchange words.
- Alt-u Uppercase word.
- Alt-l Lowercase word.
- Alt-? List possible completions.
- Alt-* Insert all possible completions.
- sar 查看系统状态的综合命令。
- iostat 查看 CPU 负载,硬盘状况
- mpstat 查看多处理器状态。
- iptraf 实时网络状态监控。
- Stop/Halt a user named didi. : skill -STOP -u didi
- resume a halted user: skill -CONT -u didi
- kill and logout user called didi. skill -KILL -u didi
- kill and logout all users. skill -KILL -v /dev/pts/*
- kill a program by it's name. kill PROG_NAME
- disable enforcement of selinux. setenforce 0 sudo echo 0 > /selinux/enforce
- lpstat query status of printing jobs
- lpq show the queue status;
- Use hwclock --systohc --localtime to change the time from UTC to local time;
- In xinetd.conf, bind rather than server is used to sign the address of the service;
- The /etc/nsswitch.conf defines the source order for security configuration.
- Encryption: When A uses encryption to send data specifically to B, it encrypts the data using B's public key, so that B can decrypt the message after receiving it from A.
- cupsdisable will disable print service and cups enables the disabled service
- vmstat Monitor virtual memory
- pmap Display/examine memory map and libraries (so). Usage: pmap pid
- sar -B Show statistics on page swapping.
- time -v date Show system page size, page faults, etc of a process during execution. Note you must fully qualify the command as "/usr/bin/time" to avoid using the bash shell command "time".
- cat /proc/sys/vm/freepages Display virtual memory "free pages". One may increase/decrease this limit: echo 300 400 500 > /proc/sys/vm/freepages
- cat /proc/meminfo Show memory size and usage.
- To create a background process that will not hang after user exist, we can use command nohup.
- To put a process into background, use three steps 1) ctrl-z 2)bg 3) disown -h
- Print each item in a CSV file: for line in `cat dpgdp.csv`; do for i in `echo $line | tr ',' ' '`; do echo $i; done; echo '============='; done;
EIGRP Lab Experiments
EIGRP Lab Experiments
This lab includes the following topics:
Ø IGRP packet/message
types;
Ø EIGRP terminologies;
Ø EIGRP Data Structures;
Ø EIGRP summarization,
load balancing, authentication;
Ø EIGRP parameter tunning;
Ø EIGRP trouble shooting
and miscellaneous topics;
The configuration of this lab is show as in the following diagram.
Router R1 is the Head quarter R2 and R3 are connected to R1 using
Frame-Relay. R4 is redundant link between R1 and R3 with ppp connection. R1 is
connected with LAN 10.1.1-3.0/24 and R2 and R3 are connected to LAN
192.168.1.0/24.
Configuration
Steps:
Step 1:
Configure the routers as the diagram shows, also note that in
order to simulate a LAN environment on R1, I used loopback interface. So you
need to configure the loopback interface from 0 to 2. Also, don't forget to
issue the "no shutdown command". In this lab, the Frame-Relay configuration
is the one I want to note here. In order to ease the effort of configuration, I
used Point-To-Point sub-interfaces on routers R1, R2 and R3.
Step 2:
Configure the EIGRP routing protocol on all the routers which you
want to participate in the EIGRP process. Example, for Router R1, you need to
issue the following command on the router.
router eigrp 1
network 10.0.0.0 network 190.16.0.0 network 220.1.100.0 no auto-summary |
After that, EIGRP should work. This can be verified with the
following commands.
Check the Routing table.
R1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route Gateway of last resort is not set 16.0.0.0/30 is subnetted, 1 subnets C 16.1.3.0 is directly connected, Serial0/0.103 220.1.100.0/30 is subnetted, 1 subnets C 220.1.100.0 is directly connected, Serial0/0.102 190.16.0.0/16 is variably subnetted, 3 subnets, 3 masks C 190.16.8.2/32 is directly connected, Serial0/1 C 190.16.8.0/30 is directly connected, Serial0/1 D 190.16.0.0/16 [90/3196416] via 220.1.100.2, 00:01:12, Serial0/0.102 172.16.0.0/16 is variably subnetted, 3 subnets, 3 masks D EX 172.16.1.1/32 [170/2684416] via 220.1.100.2, 00:01:12, Serial0/0.102 D 172.16.0.0/16 [90/2681856] via 190.16.8.2, 00:01:28, Serial0/1 D 172.16.1.0/30 [90/2684416] via 220.1.100.2, 00:01:12, Serial0/0.102 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks C 10.1.3.0/24 is directly connected, Loopback2 C 10.1.2.0/24 is directly connected, Loopback1 C 10.1.1.0/24 is directly connected, Loopback0 D 10.1.0.0/22 is a summary, 00:01:41, Null0 D 192.168.1.0/24 [90/2172416] via 220.1.100.2, 00:01:13, Serial0/0.102 |
Check EIGRP
neighbors
R1# show ip eigrp neighbors
IP-EIGRP neighbors for process 1 H Address Interface Hold Uptime SRTT RTO Q Seq (sec) (ms) Cnt Num 1 220.1.100.2 Se0/0.102 13 00:04:12 32 200 0 7 0 190.16.8.2 Se0/1 10 00:04:27 413 2478 0 10 |
Check EIGRP topology data structure.
R1#show ip eigrp topology
IP-EIGRP Topology Table for AS(1)/ID(10.1.3.1) Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply, r - reply Status, s - sia Status P 10.1.3.0/24, 1 successors, FD is 128256 via Connected, Loopback2 P 10.1.2.0/24, 1 successors, FD is 128256 via Connected, Loopback1 P 10.1.1.0/24, 1 successors, FD is 128256 via Connected, Loopback0 P 10.1.0.0/22, 1 successors, FD is 128256 via Summary (128256/0), Null0 P 16.1.3.0/30, 0 successors, FD is Inaccessible via 220.1.100.2 (2684416/2172416), Serial0/0.102 via 190.16.8.2 (3193856/2681856), Serial0/1 P 192.168.1.0/24, 1 successors, FD is 2172416 via 220.1.100.2 (2172416/28160), Serial0/0.102 P 190.16.8.2/32, 1 successors, FD is 2169856 via Rconnected (2169856/0) P 190.16.8.0/30, 1 successors, FD is 2169856 via Connected, Serial0/1 P 190.16.8.1/32, 0 successors, FD is Inaccessible --- OMITTED FOR CONCISE --- |
Step 3:
We will investigate the process of how EIGRP neibhbors are formed.
This can be done using the debug command. Also note that if you can't see the
debug messages you need to enable console monitoring and logging under the
configuration mode. In this step we also want to see EIGRP hello, update,
query, reply and ack messages with the help of the debug command.
We first go to router R1 and issue the "debug eigrp
neighbor" command, then go to router R4 and shutdown interface Serial0/0.
When hold-down timer expires on R1, we can see the neighbor going down.
R1#debug eigrp neighbors
*Mar 1
00:18:38.895: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 190.16.8.2
(Serial0/1) is down: holding time expired
*Mar 1
00:18:38.911: EIGRP: Neighbor 190.16.8.2 went down on Serial0/1
*Mar 1
00:30:15.163: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 190.16.8.2
(Serial0/1) is down: holding time expired
|
After that, we bring up the Serial 0/0 interface on R4 and will notice
the process of how the neighbors are formed.
*Mar 1
00:30:26.623: IP-EIGRP: Callback: redist frm connected AS 0 190.16.8.2/32
*Mar 1
00:30:26.623: into: eigrp AS
1 event: 2
*Mar 1
00:30:26.691: IP-EIGRP: Callback: redist frm connected AS 0 190.16.8.2/32
*Mar 1
00:30:26.695: into: eigrp AS
1 event: 1
*Mar 1
00:30:26.695: IP-EIGRP(Default-IP-Routing-Table:0): Callback: redist
connected (config change) Serial0/1
*Mar 1
00:30:28.723: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 190.16.8.2
(Serial0/1) is up: new adjacency
|
Next we will investigate EIGRP messages.
R1#debug eigrp packets
hello
EIGRP Packets debugging is on
(HELLO)
R1#
*Mar 1
00:40:31.991: EIGRP: Sending HELLO on Serial0/1
*Mar 1
00:40:31.991: AS 1, Flags 0x0, Seq
0/0 idbQ 0/0 iidbQ un/rely 0/0
*Mar 1
00:40:31.995: EIGRP: Sending HELLO on Loopback2
*Mar 1
00:40:31.995: AS 1, Flags 0x0, Seq
0/0 idbQ 0/0 iidbQ un/rely 0/0
*Mar 1
00:40:31.995: EIGRP: Received HELLO on Loopback2 nbr 10.1.3.1
*Mar 1
00:40:31.995: AS 1, Flags 0x0, Seq
0/0 idbQ 0/0
*Mar 1
00:40:32.239: EIGRP: Received HELLO on Serial0/0.102 nbr 220.1.100.2
*Mar 1
00:40:32.239: AS 1, Flags 0x0, Seq
0/0 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
*Mar 1
00:40:34.015: EIGRP: Received HELLO on Serial0/1 nbr 190.16.8.2
|
In order to see QUERY, REPLY and ACK messages, we need to simulate
a real world link outage. Here, we again shut down the Serial 0/0 interface of
R4 and then bring it back up.
*Mar 1 00:46:12.867: EIGRP: Received QUERY on
Serial0/0.102 nbr 220.1.100.2
*Mar 1
00:46:12.867: AS 1, Flags 0x0, Seq
153/101 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/0
*Mar 1
00:46:12.871: EIGRP: Enqueueing ACK on Serial0/0.102 nbr 220.1.100.2
*Mar 1
00:46:12.871: Ack seq 153 iidbQ
un/rely 0/0 peerQ un/rely 1/0
*Mar 1
00:46:12.875: EIGRP: Sending ACK on Serial0/0.102 nbr 220.1.100.2
*Mar 1
00:46:12.879: AS 1, Flags 0x0, Seq
0/153 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 1/0
*Mar 1
00:46:12.883: EIGRP: Enqueueing QUERY on Serial0/1 iidbQ un/rely 0/1 serno
86-86
*Mar 1
00:46:12.883: EIGRP: Enqueueing QUERY on Serial0/0.102 iidbQ un/rely 0/1
serno 86-86
*Mar 1
00:46:12.887: EIGRP: Enqueueing QUERY on Serial0/1 nbr 190.16.8.2 iidbQ
un/rely 0/0 peerQ un/rely 0/0 serno 86-86
*Mar 1
00:46:12.891: EIGRP: Enqueueing QUERY on Serial0/0.102 nbr 220.1.100.2 iidbQ
un/rely 0/0 peerQ un/rely 0/0 serno 86-86
*Mar 1
00:46:12.899: EIGRP: Sending QUERY on Serial0/1 nbr 190.16.8.2
*Mar 1
00:46:12.899: AS 1, Flags 0x0, Seq
103/101 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 86-86
*Mar 1
00:46:13.155: EIGRP: Sending QUERY on Serial0/1 nbr 190.16.8.2, retry 1, RTO
369
*Mar 1
00:46:25.055: EIGRP: Sending QUERY on Serial0/1 nbr 190.16.8.2, retry 8, RTO
5000
*Mar 1
00:46:25.059: AS 1, Flags 0x0, Seq
103/101 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 86-86
*Mar 1
00:46:26.643: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 190.16.8.2
(Serial0/1) is down: holding time expired
*Mar 1
00:46:26.663: EIGRP: Enqueueing REPLY on Serial0/0.102 nbr 220.1.100.2 iidbQ
un/rely 0/1 peerQ un/rely 0/0 serno 87-87
*Mar 1
00:46:26.671: EIGRP: Sending REPLY on Serial0/0.102 nbr 220.1.100.2
*Mar 1
00:46:26.675: AS 1, Flags 0x0, Seq
105/153 idbQ 0/0 iidbQ un/rely 0/0 peerQ un/rely 0/1 serno 87-87
*Mar 1
00:46:26.707: EIGRP: Received ACK on Serial0/0.102 nbr 220.1.100.2
|
We can show the EIGRP traffic statistics with the show ip eigrp
traffic command.
R1#show ip eigrp traffic
IP-EIGRP Traffic Statistics for AS 1
Hellos
sent/received: 3265/3166
Updates
sent/received: 72/75
Queries sent/received: 73/11
Replies sent/received: 12/27
Acks
sent/received: 89/89
Input
queue high water mark 3, 0 drops
SIA-Queries sent/received: 0/0
SIA-Replies sent/received: 0/0
Hello
Process ID: 134
PDM Process
ID: 133
|
Step 4:
We will get familiar with FD, AD, successor, feasible successor.
R1#show ip eigrp topology
IP-EIGRP Topology Table for AS(1)/ID(10.1.3.1)
Codes: P - Passive, A - Active, U - Update, Q
- Query, R - Reply,
r
- reply Status, s - sia Status
P 10.1.3.0/24, 1 successors, FD is 128256
via Connected, Loopback2
P 10.1.2.0/24, 1 successors, FD is 128256
via Connected, Loopback1
P 10.1.1.0/24, 1 successors, FD is 128256
via Connected, Loopback0
P 10.1.0.0/22, 1 successors, FD is 128256
via Summary (128256/0), Null0
P 16.1.3.0/30, 0 successors, FD is
Inaccessible
via 190.16.8.2 (3193856/2681856), Serial0/1
via 220.1.100.2 (2684416/2172416), Serial0/0.102
P 192.168.1.0/24, 1 successors, FD is 2172416
via 220.1.100.2 (2172416/28160), Serial0/0.102
P 190.16.8.2/32, 1 successors, FD is 2169856
via Rconnected (2169856/0)
P 190.16.8.0/30, 1 successors, FD is 2169856
via Connected, Serial0/1
P 190.16.8.1/32, 0 successors, FD is
Inaccessible
via 190.16.8.2 (2681856/2169856), Serial0/1
|
Step 5:
We will investigate summarization, authentication and load
balancing.
I am going to configure R1 so that it only advertise the
summarized address of networks 10.1.1-3.0/24 to router R4.
interface Serial0/1
ip
address 190.16.8.1 255.255.255.252
encapsulation ppp
ip
summary-address eigrp 1 10.1.0.0 255.255.252.0 5
serial
restart-delay 0
end
|
Then under R4, let's check the result.
R4#show ip route
-- Omitted for Simplicity. --
10.0.0.0/22 is subnetted, 1 subnets
D
10.1.0.0 [90/2297856] via 190.16.8.1, 00:01:22, Serial0/0
|
Authentication of EIGRP neighbors utilizes key chain. So, first we
need to define key chain.
key chain R1TOR4
key 1
key-string cisco
accept-lifetime 10:00:00 Mar 8 2010 11:00:00 Mar 8 2010
send-lifetime 11:00:00 Mar 8 2010 12:00:00 Mar 8 2010
key 2
key-string simon
accept-lifetime 11:00:00 Mar 8 2010 12:00:00 Mar 8 2010
send-lifetime 11:00:00 Mar 8 2010 12:00:00 Mar 8 2010
|
After that we need to configure the authentication under the
interface of the neighbor.
ip authentication key-chain eigrp 1 R1TOR4
|
Also please note, we have to define the same configuration on the
other side of the neighbor relationship. And in order for authentication to
work, the two neighbors should have synchronized system clock. This can be done
with the ntp service. For more information, please check related documents.
Next let's work on load-balancing. By default, EIGRP will load
balance 4 equal cost routes. We can change this number by tunning the maximum-paths parameter.
More importantly, we can configure to let EIGRP do unequal cost load-balancing
with the variance command.
We can show the current path number by the following command
R3#show ip protocols
Routing Protocol is "eigrp 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP
metric weight K1=1, K2=0, K3=1, K4=0, K5=0
EIGRP
maximum hopcount 100
EIGRP
maximum metric variance 1
Redistributing: eigrp 1
EIGRP
NSF-aware route hold timer is 240s
Automatic
network summarization is not in effect
Maximum
path: 2
Routing for Networks:
16.0.0.0
172.16.0.0
192.168.1.0
Routing Information Sources:
Gateway Distance Last Update
192.168.1.1 90 00:00:03
172.16.1.1 90 00:00:03
Distance: internal 90 external 170
|
Then, let's evaluate the unequal cost load balancing on router R3.
Variance = 1
D
10.1.0.0 [90/2300416] via 192.168.1.1, 00:00:03, FastEthernet1/0
Variance = 2
D
10.1.0.0 [90/2300416] via 192.168.1.1, 00:00:03, FastEthernet1/0
[90/2809856] via 172.16.1.1, 00:00:03,
Serial0/1
|
Step 6:
Finally, let's tune the timers, metrics and other miscellaneous
parameters of EIGRP.
In order to change timers, we need to go to the specific interface,
and issue: "ip hello-interval eigrp 1 10" or "ip hold-time eigrp
1 40".
Metrics are lated to the K value of eigrp routing protocol. It can
be tuned with "metric weights 0 1 0 1 0 0"
Setting default network is always very helpful sometimes. Under
EIGRP, we can use the default network command to do this. This doesn't work on
my lab environment.
Also, we can use the classic "ip route 0.0.0.0 0.0.0.0 "
command to do this.
First, let's set the default route under router R1 and the default route is to interface loopback 3. Then we propogate this default route to R1' neighbor R4. Then we can check the result using the "show ip route" command.
First, let's set the default route under router R1 and the default route is to interface loopback 3. Then we propogate this default route to R1' neighbor R4. Then we can check the result using the "show ip route" command.
conf t
ip route 0.0.0.0 0.0.0.0 lo 3
router eigrp 1
network 0.0.0.0
R1#show ip route
Gateway of last resort is 0.0.0.0 to network
0.0.0.0
S*
0.0.0.0/0 is directly connected, Loopback3
R4#show ip route
Gateway of last resort is 190.16.8.1 to
network 0.0.0.0
D*
0.0.0.0/0 [90/2297856] via 190.16.8.1, 00:10:48, Serial0/0
|
Subscribe to:
Posts (Atom)