??斗地主捕鱼电竞提现秒到 广告位招租 - 15元/月全站展示
??支付宝搜索579087183领大额红包 ??伍彩集团官网直营彩票
??好待遇→招代理 ??伍彩集团官网直营彩票
??络茄网 广告位招租 - 15元/月全站展示
大数据之:Flume安装详解

转载   2018-05-17   浏览量:152


0. 软件版本下载
//mirror.bit.edu.cn/apache/flume/

1. 集群环境
Master 172.16.11.97
Slave1 172.16.11.98
Slave2 172.16.11.99

2. 下载软件包
#Master
wget //mirror.bit.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
tar zxvf apache-flume-1.6.0-bin.tar.gz

3. 修改Flume配置
#NetCat
vim conf/flume-netcat.conf
# Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1

# Describe/configuration the source
agent.sources.r1.type = netcat
agent.sources.r1.bind = 127.0.0.1
agent.sources.r1.port = 44444

# Describe the sink
agent.sinks.k1.type = logger

# Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1

#验证
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-netcat.conf --name=agent -Dflume.root.logger=INFO,console
#Client
telnet master 44444

#Exec
vim conf/flume-exec.conf
# Name the components on this agent
agent.sources = r1
agent.sinks = k1
agent.channels = c1

# Describe/configuration the source
agent.sources.r1.type = exec
agent.sources.r1.command = tail -f /data/hadoop/flume/test.txt

# Describe the sink
agent.sinks.k1.type = logger

# Use a channel which buffers events in memory
agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-exec.conf --name=agent -Dflume.root.logger=INFO,console
#Client
while true;do echo `date` >> /data/hadoop/flume/test.txt ; sleep 1; done


#Avro
vim conf/flume-avro.conf
# Define a memory channel called c1 on agent
agent.channels.c1.type = memory

# Define an avro source alled r1 on agent and tell it
agent.sources.r1.channels = c1
agent.sources.r1.type = avro
agent.sources.r1.bind = 127.0.0.1
agent.sources.r1.port = 44444

# Describe/configuration the source
agent.sinks.k1.type = hdfs
agent.sinks.k1.channel = c1
agent.sinks.k1.hdfs.path = hdfs://master:9000/flume_data_pool
agent.sinks.k1.hdfs.filePrefix = events-
agent.sinks.k1.hdfs.fileType = DataStream
agent.sinks.k1.hdfs.writeFormat = Text
agent.sinks.k1.hdfs.rollSize = 0
agent.sinks.k1.hdfs.rollCount= 600000
agent.sinks.k1.hdfs.rollInterval = 600

agent.channels = c1
agent.sources = r1
agent.sinks = k1
#验证
#Server
bin/flume-ng agent --conf conf --conf-file conf/flume-netcat.conf --name=agent -Dflume.root.logger=DEBUG,console
#Client
telnet master 44444

转载自:https://blog.csdn.net/knidly/article/details/80348867

招聘 不方便扫码就复制添加关注:程序员招聘谷,微信号:jobs1024


下一篇:

flume学习与总结记录
1.什么是flume??Cloudera开发的框架,实时收集数据??Flume学习的核心:agent的设计??官方文档,//flume.apache.org/FlumeUserGuide.html2.flume环境的搭建?下载地址://archive.cloudera.com/cdh5/??---选择对应版本?常规的解压安装操作??1)配置环境变量????e...
大数据之:Flume安装详解
0.软件版本下载//mirror.bit.edu.cn/apache/flume/1.集群环境Master172.16.11.97Slave1172.16.11.98Slave2172.16.11.992.下载软件包#Masterwget//mirror.bit.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-bin.ta...
flume实战教学练习新手必看
使用Flume关键就是写配置文件需求一:从指定网络端口采集数据输出到控制台conf的书写exampleconf:Asingle-nodeFlumeconfigurationNa
flume源码分析
flume源码分析,Flume的程序入口是orgapacheflumenodeApplicationmain,进入后会先进行命令行参数的解析及核对,使用的组件是orgapachecommonscli?;故呛芎糜玫?。
云计算大数据技术之Flume安装教程
云计算大数据技术之Flume安装教程。
Error:Couldnotfindorloadmainclassorg.apache.flume.tools.GetJavaProperty
Error:CouldnotfindorloadmainclassorgapacheflumetoolsGetJavaProperty。
Flume的几个基础概念
Flume的几个基础概念,从数据发生器接收数据,并将接收的数据以Flume的event格式传递给一个或者多个通道channal,Flume提供多种数据接收的方式,比如Avro,Thrift,exec等。
nginx+flume+hdfs搭建实时日志收集系统
nginx+flume+hdfs搭建实时日志收集系统。
Ceontos7安装Flume和问题
Ceontos7安装Flume及问题。Flume是一个强大的采集日志信息的工具,它适用大部分的日志采集场景。它的安装配置也非常简单,下面就一起来看看吧!
flume1.7TailDirsource重复获取数据集不释放资源解决办法
flume17TailDirsource重复获取数据集不释放资源解决办法。