分类

链接

2022 年 5 月
一 二 三 四 五 六 日
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

近期文章

热门标签

新人福利，免费薅羊毛

现在位置:    首页 ＞ 大数据 ＞ 正文
共享办公室出租
hive3.1.2 on spark
 2022-05-17 14:49 大数据 暂无评论 阅读(119)

1.  安装java(openjdk8)

 

2.安装Mysql

wget https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm
rpm -ivh mysql57-community-release-el7-8.noarch.rpm
cd /etc/yum.repos.d/
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum -y install mysql-server
systemctl start mysqld
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p
set global validate_password_policy=LOW;
set global validate_password_length=4;
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root'; 


mysql -uroot -proot
use mysql;
update user set Host='%' where User='root';
flush privileges;
exit;

3.  安装hadoop3.23

Core-site.xml文件配置
注意：以下所有文件均在对应文件夹下面 可以命令行编辑 也可在vscode 编辑

<span class="token punctuation">&lt;</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
  <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">fs</span><span class="token punctuation">.</span><span class="token variable">default</span><span class="token punctuation">.</span><span class="token variable">name</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
    <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token variable">hdfs:</span><span class="token punctuation">/</span><span class="token punctuation">/</span><span class="token variable">localhost:</span><span class="token number">8020</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>

Hdfs-site.xml文件配置

<span class="token punctuation">&lt;</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">dfs</span><span class="token punctuation">.</span><span class="token variable">replication</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token number">1</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>

<span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
  <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">dfs</span><span class="token punctuation">.</span><span class="token variable">name</span><span class="token punctuation">.</span><span class="token variable">dir</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
    <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token punctuation">/</span><span class="token variable">root</span><span class="token punctuation">/</span><span class="token variable">hadoop-3.2.3</span><span class="token punctuation">/</span><span class="token variable">data</span><span class="token punctuation">/</span><span class="token variable">namenode</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>

<span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
  <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">dfs</span><span class="token punctuation">.</span><span class="token variable">data</span><span class="token punctuation">.</span><span class="token variable">dir</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
    <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token punctuation">/<span class="token variable">root</span></span><span class="token punctuation">/</span><span class="token variable">hadoop-3.2.3</span><span class="token punctuation">/data</span><span class="token punctuation">/</span><span class="token variable">datanode</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">configuration</span><span class="token punctuation">&gt;

&lt;property&gt;
&lt;name&gt;dfs.http.address&lt;/name&gt;
&lt;value&gt;0.0.0.0:50070&lt;/value&gt;
&lt;/property&gt; 
</span>

Mapred-site.xml文件配置

<span class="token punctuation">&lt;</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
  <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">mapreduce</span><span class="token punctuation">.</span><span class="token variable">framework</span><span class="token punctuation">.</span><span class="token variable">name</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
   <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token variable">yarn</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>

Yarn-site.xml文件配置

<span class="token punctuation">&lt;</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
  <span class="token punctuation">&lt;</span><span class="token variable">name</span><span class="token punctuation">&gt;</span><span class="token variable">yarn</span><span class="token punctuation">.</span><span class="token variable">nodemanager</span><span class="token punctuation">.</span><span class="token variable">aux-services</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">name</span><span class="token punctuation">&gt;</span>
    <span class="token punctuation">&lt;</span><span class="token variable">value</span><span class="token punctuation">&gt;</span><span class="token variable">mapreduce_shuffle</span><span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">value</span><span class="token punctuation">&gt;</span>
 <span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">property</span><span class="token punctuation">&gt;</span>
<span class="token punctuation">&lt;</span><span class="token punctuation">/</span><span class="token variable">configuration</span><span class="token punctuation">&gt;</span>

 

重启格式化HDFS

hdfs namenode -format

start-dfs.sh

 

 

4.  安装hive3.1.2

hadoop fs -mkdir /spark-jars

hadoop fs -mkdir /spark-history

vim hive-3.1.2/conf/spark-defaults.conf

spark.master yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://localhost:8020/spark-history
spark.executor.memory 512m
spark.driver.memory 512m

conf 下新建 hive-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- jdbc 连接的 URL -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?useSSL=false</value>
</property>
<!-- jdbc 连接的 Driver-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!-- jdbc 连接的 username-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!-- jdbc 连接的 password -->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property>
<!-- Hive 元数据存储版本的验证 -->
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<!--元数据存储授权-->
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
</property>
<!-- 指定存储元数据要连接的地址 -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9083</value>
</property>
<!-- Hive 默认在 HDFS 的工作目录 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<!-- 指定 hiveserver2 连接的 host -->
<property>
<name>hive.server2.thrift.bind.host</name>
<value>localhost</value>
</property>
<!-- 指定 hiveserver2 连接的端口号 -->
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<!-- 打印表头 -->
<name>hive.cli.print.header</name>
<value>true</value>
</property>
<property>
<name>hive.cli.print.current.db</name>
<value>true</value>
</property>
</configuration>

 

5.安装spark2.4.7

https://archive.apache.org/dist/spark/spark-2.4.7/

 

https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7.tgz

https://archive.apache.org/dist/spark/spark-2.4.7/spark-2.4.7-bin-without-hadoop.tgz

hadoop fs -put spark-2.4.7-bin-without-hadoop/jars/* /spark-jars

 

vim /root/spark-2.4.7-bin-without-hadoop/conf/spark-env.sh

export TERM=xterm-color
export JAVA_HOME=/root/java8
export HADOOP_HOME=/root/hadoop-3.2.3
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export SPARK_HOME=/root/spark-2.4.7-bin-without-hadoop
export SPARK_DIST_CLASSPATH=$(hdfs classpath)
export MASTER_WEBUI_PORT=8079
export SPARK_LOG_DIR=/root/spark-2.4.7-bin-without-hadoop/logs
export SPARK_LIBRARY_PATH=${SPARK_HOME}/jars

export SPARK_SSH_OPTS="-p 22000"

vim ./spark-defaults.conf

spark.executor.memory 256m

spark.driver.cores 1

spark.driver.maxResultSize 0

 

6.  配置环境变量

vim /etc/profile.d/my_env.sh

export JAVA_HOME=/root/java8
export HADOOP_HOME=/root/hadoop-3.2.3
export HIVE_HOME=/root/hive-3.1.2
export SPARK_HOME=/root/spark-2.4.7

export PATH=$PATH:$JAVA_HOME/bin/:$HADOOP_HOME/bin/:$HIVE_HOME/bin:$SPARK_HOME/bin

 

source /etc/profile

 

 

 

 

 

============ 欢迎各位老板打赏~ ===========

关键字: ,
【上篇】
【下篇】

与本文相关的文章

该日志由 Bruce 于2022年05月17日发表在大数据分类下

本文版权归Bruce's Blog所有，转载引用请完整注明以下信息：
本文作者：Bruce
本文地址：hive3.1.2 on spark | Bruce's Blog

发表评论

留言无头像?