Tachyon 0.7.1伪分布式集群安装与测试

2024-02-01 12:08

本文主要是介绍Tachyon 0.7.1伪分布式集群安装与测试,希望对大家解决编程问题提供一定的参考价值,需要的开发者们随着小编来一起学习吧!

Tachyon是一个高容错的分布式文件系统,允许文件以内存的速度在集群框架中进行可靠的共享,就像Spark和 MapReduce那样。通过利用信息继承,内存侵入,Tachyon获得了高性能。Tachyon工作集文件缓存在内存中,并且让不同的 Jobs/Queries以及框架都能内存的速度来访问缓存文件。因此,Tachyon可以减少那些需要经常使用的数据集通过访问磁盘来获得的次数。

源码下载

源码地址:https://github.com/amplab/tachyon
git clone https://github.com/amplab/tachyon.git

编译

进入 TACHYON_HOME

git tagv0.1.0
v0.2.0
v0.2.1
v0.3.0
v0.4.0
v0.4.1
v0.4.1-thrift
v0.5.0
v0.6.0
v0.6.1
v0.6.2
v0.6.3
v0.6.4
v0.7.0
v0.7.0-rc1
v0.7.1

切换到v0.7.1版本

git checkout v0.7.1 
Note: checking out 'v0.7.1'.You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:git checkout -b new_branch_nameHEAD 目前位于 a2b69c9... [maven-release-plugin] prepare release v0.7.1

执行如下命令:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn clean package  -Dhadoop.version=2.3.0-cdh5.1.0 -Dspark.version=1.4.1 -DskipTests

这里写图片描述

配置

  1. 创建journal目录
    mkdir -p /home/cluster/apps/tachyon/journal
  2. 创建ramdisk
    mkdir -p /home/cluster/apps/tachyon/ramdisk
  3. 将tachyon_home/conf/tachyon-env.sh.template 修改为tachyon_home/conf/tachyon-env.sh
    mv conf/tachyon-env.sh.template conf/tachyon-env.sh
  4. 编辑tachyon_home/conf/tachyon-env.sh 增加如下内容
export JAVA_HOME=/home/cluster/share/java1.7
export HADOOP_HOME=/home/cluster/apps/hadoop
export TACHYON_HOME=/home/cluster/apps/tachyon
export TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdisk
export TACHYON_MASTER_ADDRESS=master
  1. 修改tachyon日志存放路径:
    tachyon_home/下创建logs目录
    mkdir logs
    vim conf/log4j.properties 增加如下内容:
    tachyon.logs.dir=/home/cluster/apps/tachyon/logs
  2. 将tachyon_home/conf/core-site.xml.template 修改为tachyon_home/conf/core-site.xml
    mv conf/core-site.xml.template conf/core-site.xml
    编辑 conf/core-site.xml,增加如下内容:

    <configuration>
    <property>
      <name>fs.tachyon.impl</name>
      <value>tachyon.hadoop.TFS</value>
     </property>
    </configuration>
  3. 格式化Tachyon
bin/tachyon format
Connecting to localhost as wangyue...
Formatting Tachyon Worker @ wangyue-um
Connection to localhost closed.
Formatting Tachyon Master @ master

启动

bin/tachyon-start.sh local
Killed 0 processes on wangyue-um
Killed 0 processes on wangyue-um
Connecting to localhost as wangyue...
Killed 0 processes on wangyue-um
Connection to localhost closed.
[sudo] password for wangyue: 
Formatting RamFS: /home/cluster/apps/tachyon/ramdisk (1gb)
Starting master @ master
Starting worker @ wangyue-umjps下:
21025 ZeppelinServer
22214 Worker
21987 Master
21360 TachyonWorker
32212 RemoteInterpreterServer
23361 NameNode
29603 RemoteInterpreterServer
21324 TachyonMaster
23493 DataNode
21412 Jps
27005 RemoteInterpreterServer已经启动TachyonMaster & TachyonWorker

tachyon-start.sh local命令将同时在本地启动Master和Worker进程。需要注意的是,运行tachyon-start.sh local命令一定要拥有切换root的密码,否者会无法启动。这是因为RamFS的格式化需要root权限。

测试

webui:http://master:19999
这里写图片描述

/home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH
/default_tests_files/BasicFile_CACHE_THROUGH has been removed
2015-09-09 20:00:17,911 INFO   (MasterClient.java:connect) - Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:19998
2015-09-09 20:00:17,938 INFO   (MasterClient.java:connect) - User registered with the master @ master/127.0.0.1:19998; got UserId 5
2015-09-09 20:00:17,964 INFO   (CommonUtils.java:printTimeTakenMs) - createFile with fileId 3 took 57 ms.
2015-09-09 20:00:17,999 INFO   (WorkerClient.java:connect) - Trying to get local worker host : wangyue-um.local
2015-09-09 20:00:18,013 INFO   (WorkerClient.java:connect) - Connecting local worker @ wangyue-um.local/10.32.24.78:29998
2015-09-09 20:00:18,098 INFO   (BlockOutStream.java:get) - Writing with local stream. tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH
2015-09-09 20:00:18,136 INFO   (CommonUtils.java:createBlockPath) - Folder /home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was created!
2015-09-09 20:00:18,140 INFO   (LocalBlockOutStream.java:<init>) - /home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! tachyonFile: /default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 536870912
2015-09-09 20:00:18,203 INFO   (CommonUtils.java:printTimeTakenMs) - writeFile to file /default_tests_files/BasicFile_CACHE_THROUGH took 238 ms.
2015-09-09 20:00:18,246 INFO   (CommonUtils.java:printTimeTakenMs) - readFile file /default_tests_files/BasicFile_CACHE_THROUGH took 43 ms.
Passed the test!执行  bin/tachyon runTests 循环测试

停止

wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh 
Killed 1 processes on wangyue-um
Killed 1 processes on wangyue-um
Connecting to localhost as wangyue...
Killed 0 processes on wangyue-um
Connection to localhost closed.执行jps:
21025 ZeppelinServer
22214 Worker
21987 Master
32212 RemoteInterpreterServer
23361 NameNode
29603 RemoteInterpreterServer
27469 Jps
23493 DataNode
27005 RemoteInterpreterServer
没有tachyon相关进程

尊重原创,拒绝转载,http://blog.csdn.net/stark_summer/article/details/48321605

这篇关于Tachyon 0.7.1伪分布式集群安装与测试的文章就介绍到这儿,希望我们推荐的文章对编程师们有所帮助!



http://www.chinasem.cn/article/667223

相关文章

最详细安装 PostgreSQL方法及常见问题解决

《最详细安装PostgreSQL方法及常见问题解决》:本文主要介绍最详细安装PostgreSQL方法及常见问题解决,介绍了在Windows系统上安装PostgreSQL及Linux系统上安装Po... 目录一、在 Windows 系统上安装 PostgreSQL1. 下载 PostgreSQL 安装包2.

Maven如何手动安装依赖到本地仓库

《Maven如何手动安装依赖到本地仓库》:本文主要介绍Maven如何手动安装依赖到本地仓库问题,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录一、下载依赖二、安装 JAR 文件到本地仓库三、验证安装四、在项目中使用该依赖1、注意事项2、额外提示总结一、下载依赖登

如何在Mac上安装并配置JDK环境变量详细步骤

《如何在Mac上安装并配置JDK环境变量详细步骤》:本文主要介绍如何在Mac上安装并配置JDK环境变量详细步骤,包括下载JDK、安装JDK、配置环境变量、验证JDK配置以及可选地设置PowerSh... 目录步骤 1:下载JDK步骤 2:安装JDK步骤 3:配置环境变量1. 编辑~/.zshrc(对于zsh

深入理解Apache Kafka(分布式流处理平台)

《深入理解ApacheKafka(分布式流处理平台)》ApacheKafka作为现代分布式系统中的核心中间件,为构建高吞吐量、低延迟的数据管道提供了强大支持,本文将深入探讨Kafka的核心概念、架构... 目录引言一、Apache Kafka概述1.1 什么是Kafka?1.2 Kafka的核心概念二、Ka

如何在pycharm安装torch包

《如何在pycharm安装torch包》:本文主要介绍如何在pycharm安装torch包方式,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录在pycharm安装torch包适http://www.chinasem.cn配于我电脑的指令为适用的torch包为总结在p

在PyCharm中安装PyTorch、torchvision和OpenCV详解

《在PyCharm中安装PyTorch、torchvision和OpenCV详解》:本文主要介绍在PyCharm中安装PyTorch、torchvision和OpenCV方式,具有很好的参考价值,... 目录PyCharm安装PyTorch、torchvision和OpenCV安装python安装PyTor

Python Transformer 库安装配置及使用方法

《PythonTransformer库安装配置及使用方法》HuggingFaceTransformers是自然语言处理(NLP)领域最流行的开源库之一,支持基于Transformer架构的预训练模... 目录python 中的 Transformer 库及使用方法一、库的概述二、安装与配置三、基础使用:Pi

如何解决mmcv无法安装或安装之后报错问题

《如何解决mmcv无法安装或安装之后报错问题》:本文主要介绍如何解决mmcv无法安装或安装之后报错问题,具有很好的参考价值,希望对大家有所帮助,如有错误或未考虑完全的地方,望不吝赐教... 目录mmcv无法安装或安装之后报错问题1.当我们运行YOwww.chinasem.cnLO时遇到2.找到下图所示这里3.

Python 安装和配置flask, flask_cors的图文教程

《Python安装和配置flask,flask_cors的图文教程》:本文主要介绍Python安装和配置flask,flask_cors的图文教程,本文通过图文并茂的形式给大家介绍的非常详细,... 目录一.python安装:二,配置环境变量,三:检查Python安装和环境变量,四:安装flask和flas

Redis分片集群的实现

《Redis分片集群的实现》Redis分片集群是一种将Redis数据库分散到多个节点上的方式,以提供更高的性能和可伸缩性,本文主要介绍了Redis分片集群的实现,具有一定的参考价值,感兴趣的可以了解一... 目录1. Redis Cluster的核心概念哈希槽(Hash Slots)主从复制与故障转移2.