活动介绍

sqoop: command not found

时间: 2023-10-08 09:05:08 浏览: 923
引用中的错误提示"sqoop: command not found"通常是因为系统无法找到sqoop命令。这个问题可能是由于以下几个原因导致的: 1. 没有正确安装sqoop或者未将sqoop添加到系统的PATH环境变量中。请确保sqoop已经成功安装,并且在命令行中可以通过输入"sqoop"来执行。 2. 您可能正在使用错误的命令或拼写错误。请检查您的shell脚本中的sqoop命令是否正确,并确保没有拼写错误或语法错误。 3. 您可能没有以正确的方式调用sqoop命令。请确保您在shell脚本中正确地使用了sqoop命令的参数和选项。 如果您确定已经正确安装并配置了sqoop,但仍然遇到"sqoop: command not found"错误,请尝试以下方法进行排查: 1. 在命令行中直接运行sqoop命令,看是否可以正常执行。 2. 检查系统的PATH环境变量,确保已经添加了sqoop的安装目录。 3. 检查您的shell脚本中是否正确设置了sqoop的路径。可以尝试使用绝对路径来调用sqoop命令,例如"/usr/local/sqoop/bin/sqoop"。 如果以上方法都无法解决问题,您可以尝试重新安装sqoop或者在sqoop的官方论坛或社区中寻求帮助。
相关问题

bash: sqoop: command not found怎么办

### 解决方案 当遇到 `sqoop` 命令未找到的问题时,通常是因为环境变量中缺少 Sqoop 的路径设置。为了确保能够正常使用 Sqoop 命令,需要按照以下方法进行配置。 #### 修改环境变量 将 Sqoop 的二进制文件路径添加到系统的 `$PATH` 变量中可以有效解决问题。具体操作如下: 编辑 `.bashrc` 文件,在其中加入 Sqoop 路径: ```bash vim ~/.bashrc ``` 在文件末尾追加以下内容: ```bash export PATH=$PATH:/usr/local/sqoop/bin ``` 保存并关闭文件后,通过以下命令使更改立即生效: ```bash source ~/.bashrc ``` 此时再次尝试运行 `sqoop help` 应该不会再提示命令未找到了[^4]。 另外一种方式是直接编辑全局配置文件 `/etc/profile` 来永久性地增加 Sqoop 的路径支持: ```bash vim /etc/profile ``` 同样地,在此文件里加上相同的内容来扩展 `$PATH` 并应用变更: ```bash export PATH=$PATH:/usr/local/sqoop/bin source /etc/profile ``` 以上两种办法都可以实现让 Bash 认识新的命令工具的目的;对于单用户的临时需求推荐前者,而对于多用户共享主机的情况则更适合后者[^3]。 #### 验证安装与配置 完成上述步骤之后,可以通过执行下面这条指令验证是否已经正确设置了 Sqoop 环境: ```bash sqoop help ``` 如果一切正常,则会显示 Sqoop 提供的帮助信息列表而不是之前的错误消息[^1]。

怎么定时执行sqoop任务 sqoop: command not found

### 定时执行 Sqoop 任务并解决 `sqoop: command not found` 的方法 当尝试通过 `cron` 或者 Shell 脚本来定时运行 Sqoop 任务时,可能会遇到诸如 `sqoop: command not found` 这样的错误。这通常是由于环境变量未正确设置或者路径不匹配所导致的。 #### 环境变量配置的重要性 Cron 使用的是有限的默认环境变量集合[^1],因此即使在终端中可以正常运行 Sqoop 命令,在 Cron 中可能仍然会报错。为了确保 Cron 可以成功调用 Sqoop,需要显式指定其可执行文件的位置或将其路径加入到脚本中的 `$PATH` 环境变量里。 #### 编写 Shell 脚本 可以通过编写一个简单的 Shell 脚本来封装 Sqoop 任务,并在此过程中明确定义所需的环境变量: ```bash #!/bin/bash # 设置必要的环境变量 export HADOOP_HOME=/path/to/your/hadoop-installation-directory export SQOOP_HOME=/path/to/your/sqoop-installation-directory export PATH=$SQOOP_HOME/bin:$HADOOP_HOME/bin:$PATH # 执行 Sqoop 导入任务 $SQOOP_HOME/bin/sqoop import \ --connect jdbc:mysql://localhost/testdb \ --username root \ --password password \ --table employees \ --target-dir /user/data/sqoop_imports/employees ``` 此脚本设置了两个主要的环境变量:`HADOOP_HOME` 和 `SQOOP_HOME`,并将它们对应的 bin 目录添加到了系统的 `$PATH` 中。这样能够保证无论是在交互式的 Bash 终端还是非交互式的 Cron 环境下都能找到 Sqoop 和其他依赖工具的二进制文件。 #### 配置 Crontab 来调度任务 编辑用户的 crontab 文件来安排定期的任务执行: ```bash crontab -e ``` 然后增加一行用于每天凌晨两点触发上面提到的那个脚本(假设该脚本保存为 `/home/user/scripts/run_sqoop.sh`) : ```text 0 2 * * * /home/user/scripts/run_sqoop.sh >> /home/user/logs/sqoop.log 2>&1 ``` 这里重定向了标准输出和错误流至日志文件以便于后续排查任何潜在的问题。 #### 检查权限与所有权 确认脚本具有足够的执行权限并且由合适的用户拥有。如果必要的话更改这些属性: ```bash chmod +x /home/user/scripts/run_sqoop.sh chown user:usergroup /home/user/scripts/run_sqoop.sh ``` 以上步骤应该能有效防止因缺少适当环境而导致的 “command not found” 错误发生。 ### 注意事项 - 如果使用的是特定版本的 Hadoop/Sqoop,则需调整相应的安装路径。 - 对于敏感数据如数据库密码考虑采用更安全的方式存储而不是硬编码在脚本内部。
阅读全文

相关推荐

sqoop import \ > --connect jdbc:mysql://hadoop01:3306/test \ > --username 'root' \ > --password '123456' \ > --query "select id,hno from emp_conn where /$CONDITIONS" \ > --targer-dir '/user/hive/warehouse/tx.db/emp_conn' \ > -m 1 Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /export/server/zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 25/03/18 21:01:50 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1 25/03/18 21:01:50 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 25/03/18 21:01:50 ERROR tool.BaseSqoopTool: Error parsing arguments for import: 25/03/18 21:01:50 ERROR tool.BaseSqoopTool: Unrecognized argument: --targer-dir 25/03/18 21:01:50 ERROR tool.BaseSqoopTool: Unrecognized argument: /user/hive/warehouse/tx.db/emp_conn 25/03/18 21:01:50 ERROR tool.BaseSqoopTool: Unrecognized argument: -m 25/03/18 21:01:50 ERROR tool.BaseSqoopTool: Unrecognized argument: 1 Try --help for usage instructions.

sqoop import --connect jdbc:mysql://zhaosai:3306/mydb --username root --password jqe6b6 --table news --target-dir /user/news --fields-terminated-by “;” --hive-import --hive-table news -m 1出现错误Warning: /opt/programs/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /opt/programs/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/programs/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /opt/programs/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 23/06/10 16:18:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 23/06/10 16:18:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 23/06/10 16:18:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 23/06/10 16:18:23 INFO tool.CodeGenTool: Beginning code generation Sat Jun 10 16:18:23 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 23/06/10 16:18:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM news AS t LIMIT 1 23/06/10 16:18:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM news AS t LIMIT 1 23/06/10 16:18:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/programs/hadoop-2.7.6 注: /tmp/sqoop-root/compile/84ba419f00fa83cb5d16dba722729d01/news.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 23/06/10 16:18:25 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/84ba419f00fa83cb5d16dba722729d01/news.jar 23/06/10 16:18:25 WARN manager.MySQLManager: It looks like you are importing from mysql. 23/06/10 16:18:25 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 23/06/10 16:18:25 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 23/06/10 16:18:25 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 23/06/10 16:18:25 ERROR tool.ImportTool: Import failed: No primary key could be found for table news. Please specify one with --split-by or perform a sequential import with '-m 1'.

[root@node ~]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.42 MySQL Community Server - GPL Copyright (c) 2000, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE weblog_db; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CREATE DATABASE weblog_db' at line 1 mysql> CREATE DATABASE weblog_db; ERROR 1007 (HY000): Can't create database 'weblog_db'; database exists mysql> USE weblog_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> DROP DATABASE IF EXISTS weblog_db; Query OK, 2 rows affected (0.02 sec) mysql> CREATE DATABASE weblog_db; Query OK, 1 row affected (0.01 sec) mysql> USE weblog_db; Database changed mysql> CREATE TABLE page_visits ( -> page VARCHAR(255) , -> visits BIGINT -> ); ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'TABLE page_visits ( page VARCHAR(255) , visits BIGINT )' at line 1 mysql> CREATE TABLE page_visits ( -> page VARCHAR(255), -> visits BIGINT -> ); Query OK, 0 rows affected (0.02 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | page_visits | +---------------------+ 1 row in set (0.00 sec) mysql> ^C mysql> q -> quit -> exit -> ^C mysql> ^C mysql> ^C mysql> ^DBye [root@node ~]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 7bb79582-cc2b-49b6-abc7-020dcdc46542 Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = 15d9da52-e18e-40b2-a80f-e76eda81df4c hive> DESCRIBE FORMATTED page_visits; OK # col_name data_type comment page string visits bigint # Detailed Table Information Database: default OwnerType: USER Owner: root CreateTime: Tue Jul 08 01:43:42 CST 2025 LastAccessTime: UNKNOWN Retention: 0 Location: hdfs://node:9000/hive/warehouse/page_visits Table Type: MANAGED_TABLE Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} bucketing_version 2 numFiles 1 numRows 4 rawDataSize 56 totalSize 60 transient_lastDdlTime 1751910222 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat: org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 Time taken: 0.785 seconds, Fetched: 32 row(s) hive> [root@node ~]# [root@node ~]# sqoop export \ > --connect jdbc:mysql://localhost/weblog_db \ > --username root \ > --password Aa@123456 \ > --table page_visits \ > --export-dir hdfs://node:9000/hive/warehouse/page_visits \ > --input-fields-terminated-by '\001' \ > --num-mappers 1 Warning: /home/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 2025-07-08 15:28:12,550 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-07-08 15:28:12,587 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-07-08 15:28:12,704 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-07-08 15:28:12,708 INFO tool.CodeGenTool: Beginning code generation Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-07-08 15:28:13,225 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM page_visits AS t LIMIT 1 2025-07-08 15:28:13,266 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM page_visits AS t LIMIT 1 2025-07-08 15:28:13,280 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop3.3 Note: /tmp/sqoop-root/compile/363869e21c2078b9742685122c43a3cc/page_visits.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 2025-07-08 15:28:16,377 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/363869e21c2078b9742685122c43a3cc/page_visits.jar 2025-07-08 15:28:16,391 INFO mapreduce.ExportJobBase: Beginning export of page_visits 2025-07-08 15:28:16,391 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-07-08 15:28:16,484 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-07-08 15:28:17,339 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-07-08 15:28:17,342 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:28:17,343 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-07-08 15:28:17,555 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node/192.168.196.122:8032 2025-07-08 15:28:17,782 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1751959003014_0001 2025-07-08 15:28:26,026 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:28:26,029 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:28:26,495 INFO mapreduce.JobSubmitter: number of splits:1 2025-07-08 15:28:26,528 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:28:26,619 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1751959003014_0001 2025-07-08 15:28:26,620 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-07-08 15:28:26,805 INFO conf.Configuration: resource-types.xml not found 2025-07-08 15:28:26,805 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-07-08 15:28:27,226 INFO impl.YarnClientImpl: Submitted application application_1751959003014_0001 2025-07-08 15:28:27,264 INFO mapreduce.Job: The url to track the job: http://node:8088/proxy/application_1751959003014_0001/ 2025-07-08 15:28:27,264 INFO mapreduce.Job: Running job: job_1751959003014_0001 2025-07-08 15:28:34,334 INFO mapreduce.Job: Job job_1751959003014_0001 running in uber mode : false 2025-07-08 15:28:34,335 INFO mapreduce.Job: map 0% reduce 0% 2025-07-08 15:28:38,374 INFO mapreduce.Job: map 100% reduce 0% 2025-07-08 15:28:38,381 INFO mapreduce.Job: Job job_1751959003014_0001 failed with state FAILED due to: Task failed task_1751959003014_0001_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-07-08 15:28:38,448 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2061 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2061 Total vcore-milliseconds taken by all map tasks=2061 Total megabyte-milliseconds taken by all map tasks=2110464 2025-07-08 15:28:38,456 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-07-08 15:28:38,457 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 21.1033 seconds (0 bytes/sec) 2025-07-08 15:28:38,462 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-07-08 15:28:38,462 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-07-08 15:28:38,462 ERROR mapreduce.ExportJobBase: Export job failed! 2025-07-08 15:28:38,463 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [root@node ~]# sqoop export \ > --connect jdbc:mysql://localhost/weblog_db \ > --username root \ > --password Aa@123456 \ > --table page_visits \ > --export-dir hdfs://node:9000/hive/warehouse/page_visits \ > --input-fields-terminated-by ',' \ > --num-mappers 1 Warning: /home/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 2025-07-08 15:30:31,174 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 2025-07-08 15:30:31,218 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 2025-07-08 15:30:31,333 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2025-07-08 15:30:31,336 INFO tool.CodeGenTool: Beginning code generation Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2025-07-08 15:30:31,771 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM page_visits AS t LIMIT 1 2025-07-08 15:30:31,814 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM page_visits AS t LIMIT 1 2025-07-08 15:30:31,821 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop3.3 Note: /tmp/sqoop-root/compile/ab00e36d1f5084a0f7d522b4e9a975e5/page_visits.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 2025-07-08 15:30:33,116 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/ab00e36d1f5084a0f7d522b4e9a975e5/page_visits.jar 2025-07-08 15:30:33,129 INFO mapreduce.ExportJobBase: Beginning export of page_visits 2025-07-08 15:30:33,129 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2025-07-08 15:30:33,212 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 2025-07-08 15:30:33,877 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative 2025-07-08 15:30:33,880 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:30:33,880 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 2025-07-08 15:30:34,097 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at node/192.168.196.122:8032 2025-07-08 15:30:34,310 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1751959003014_0002 2025-07-08 15:30:39,127 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:30:39,131 INFO input.FileInputFormat: Total input files to process : 1 2025-07-08 15:30:39,995 INFO mapreduce.JobSubmitter: number of splits:1 2025-07-08 15:30:40,022 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative 2025-07-08 15:30:40,532 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1751959003014_0002 2025-07-08 15:30:40,532 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2025-07-08 15:30:40,689 INFO conf.Configuration: resource-types.xml not found 2025-07-08 15:30:40,689 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2025-07-08 15:30:40,746 INFO impl.YarnClientImpl: Submitted application application_1751959003014_0002 2025-07-08 15:30:40,783 INFO mapreduce.Job: The url to track the job: http://node:8088/proxy/application_1751959003014_0002/ 2025-07-08 15:30:40,784 INFO mapreduce.Job: Running job: job_1751959003014_0002 2025-07-08 15:30:46,847 INFO mapreduce.Job: Job job_1751959003014_0002 running in uber mode : false 2025-07-08 15:30:46,848 INFO mapreduce.Job: map 0% reduce 0% 2025-07-08 15:30:50,893 INFO mapreduce.Job: map 100% reduce 0% 2025-07-08 15:30:51,905 INFO mapreduce.Job: Job job_1751959003014_0002 failed with state FAILED due to: Task failed task_1751959003014_0002_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2025-07-08 15:30:51,973 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=2058 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2058 Total vcore-milliseconds taken by all map tasks=2058 Total megabyte-milliseconds taken by all map tasks=2107392 2025-07-08 15:30:51,979 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 2025-07-08 15:30:51,980 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 18.0828 seconds (0 bytes/sec) 2025-07-08 15:30:51,983 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2025-07-08 15:30:51,983 INFO mapreduce.ExportJobBase: Exported 0 records. 2025-07-08 15:30:51,983 ERROR mapreduce.ExportJobBase: Export job failed! 2025-07-08 15:30:51,984 ERROR tool.ExportTool: Error during export: Export job failed! at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) [root@node ~]# 6.2 Sqoop导出数据 6.2.1从Hive将数据导出到MySQL 6.2.2sqoop导出格式 6.2.3导出page_visits表 6.2.4导出到ip_visits表 6.3验证导出数据 6.3.1登录MySQL 6.3.2执行查询

[root@node ~]# start-dfs.sh Starting namenodes on [node] Last login: 二 7月 8 16:00:18 CST 2025 from 192.168.1.92 on pts/0 Starting datanodes Last login: 二 7月 8 16:00:38 CST 2025 on pts/0 Starting secondary namenodes [node] Last login: 二 7月 8 16:00:41 CST 2025 on pts/0 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# start-yarn.sh Starting resourcemanager Last login: 二 7月 8 16:00:45 CST 2025 on pts/0 Starting nodemanagers Last login: 二 7月 8 16:00:51 CST 2025 on pts/0 [root@node ~]# mapred --daemon start historyserver [root@node ~]# jps 3541 ResourceManager 4007 Jps 2984 NameNode 3944 JobHistoryServer 3274 SecondaryNameNode [root@node ~]# mkdir -p /weblog [root@node ~]# cat > /weblog/access.log << EOF > 192.168.1.1,2023-06-01 10:30:22,/index.html > 192.168.1.2,2023-06-01 10:31:15,/product.html > 192.168.1.1,2023-06-01 10:32:45,/cart.html > 192.168.1.3,2023-06-01 11:45:30,/checkout.html > 192.168.1.4,2023-06-01 12:10:05,/index.html > 192.168.1.2,2023-06-01 14:20:18,/product.htm > EOF [root@node ~]# ls /weblog access.log [root@node ~]# hdfs dfs -mkdir -p /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -put /weblog/access.log /weblog/raw/ SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node ~]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node ~]# cd /weblog [root@node weblog]# mkdir weblog-mapreduce [root@node weblog]# cd weblog-mapreduce [root@node weblog-mapreduce]# touch CleanMapper.java [root@node weblog-mapreduce]# vim CleanMapper.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanMapper extends Mapper<LongWritable, Text, Text, NullWritable> { public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = line.split(","); if(fields.length == 3) { String ip = fields[0]; String time = fields[1]; String page = fields[2]; if(ip.matches("\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}")) { String outputLine = ip + "," + time + "," + page; context.write(new Text(outputLine), NullWritable.get()); } } } } [root@node weblog-mapreduce]# touch CleanReducer.java [root@node weblog-mapreduce]# vim CleanReducer.java import java.io.IOException; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class CleanReducer extends Reducer<Text, NullWritable, Text, NullWritable> { public void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException { context.write(key, NullWritable.get()); } } [root@node weblog-mapreduce]# touch LogCleanDriver.java [root@node weblog-mapreduce]# vim LogCleanDriver.java import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class LogCleanDriver { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Web Log Cleaner"); job.setJarByClass(LogCleanDriver.class); job.setMapperClass(CleanMapper.class); job.setReducerClass(CleanReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# javac -classpath $(hadoop classpath) -d . *.java [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# jar cf logclean.jar *.class [root@node weblog-mapreduce]# ls /weblog/weblog-mapreduce CleanMapper.class CleanReducer.class LogCleanDriver.class logclean.jar CleanMapper.java CleanReducer.java LogCleanDriver.java [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/raw SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 1 items -rw-r--r-- 3 root supergroup 269 2025-07-08 16:03 /weblog/raw/access.log [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. ls: /weblog/output': No such file or directory [root@node weblog-mapreduce]# hadoop jar logclean.jar LogCleanDriver /weblog/raw /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mapred job -status job_1751961655287_0001 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Job: job_1751961655287_0001 Job File: hdfs://node:9000/tmp/hadoop-yarn/staging/history/done/2025/07/08/000000/job_1751961655287_0001_conf.xml Job Tracking URL : http://node:19888/jobhistory/job/job_1751961655287_0001 Uber job : false Number of maps: 1 Number of reduces: 1 map() completion: 1.0 reduce() completion: 1.0 Job state: SUCCEEDED retired: false reason for failure: Counters: 54 File System Counters FILE: Number of bytes read=287 FILE: Number of bytes written=552699 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=372 HDFS: Number of bytes written=269 HDFS: Number of read operations=8 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 HDFS: Number of bytes read erasure-coded=0 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=1848 Total time spent by all reduces in occupied slots (ms)=2016 Total time spent by all map tasks (ms)=1848 Total time spent by all reduce tasks (ms)=2016 Total vcore-milliseconds taken by all map tasks=1848 Total vcore-milliseconds taken by all reduce tasks=2016 Total megabyte-milliseconds taken by all map tasks=1892352 Total megabyte-milliseconds taken by all reduce tasks=2064384 Map-Reduce Framework Map input records=6 Map output records=6 Map output bytes=269 Map output materialized bytes=287 Input split bytes=103 Combine input records=0 Combine output records=0 Reduce input groups=6 Reduce shuffle bytes=287 Reduce input records=6 Reduce output records=6 Spilled Records=12 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=95 CPU time spent (ms)=1050 Physical memory (bytes) snapshot=500764672 Virtual memory (bytes) snapshot=5614292992 Total committed heap usage (bytes)=379584512 Peak Map Physical memory (bytes)=293011456 Peak Map Virtual memory (bytes)=2803433472 Peak Reduce Physical memory (bytes)=207753216 Peak Reduce Virtual memory (bytes)=2810859520 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=269 File Output Format Counters Bytes Written=269 [root@node weblog-mapreduce]# hdfs dfs -ls /weblog/output SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Found 2 items -rw-r--r-- 3 root supergroup 0 2025-07-08 16:34 /weblog/output/_SUCCESS -rw-r--r-- 3 root supergroup 269 2025-07-08 16:34 /weblog/output/part-r-00000 [root@node weblog-mapreduce]# hdfs dfs -cat /weblog/output/part-r-00000 | head -5 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. 192.168.1.1,2023-06-01 10:30:22,/index.html 192.168.1.1,2023-06-01 10:32:45,/cart.html 192.168.1.2,2023-06-01 10:31:15,/product.html 192.168.1.2,2023-06-01 14:20:18,/product.htm 192.168.1.3,2023-06-01 11:45:30,/checkout.html [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 5199f37c-a381-428a-be1b-0a2afaab8583 Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f38c99b3-ff7c-4f61-ae07-6b21d86d7160 hive> CREATE EXTERNAL TABLE weblog ( > ip STRING, > access_time TIMESTAMP, > page STRING > ) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > LOCATION '/weblog/output'; OK Time taken: 1.274 seconds hive> select * from weblog; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html 192.168.1.4 2023-06-01 12:10:05 /index.html Time taken: 1.947 seconds, Fetched: 6 row(s) hive> select * from weblog limit 5; OK 192.168.1.1 2023-06-01 10:30:22 /index.html 192.168.1.1 2023-06-01 10:32:45 /cart.html 192.168.1.2 2023-06-01 10:31:15 /product.html 192.168.1.2 2023-06-01 14:20:18 /product.htm 192.168.1.3 2023-06-01 11:45:30 /checkout.html Time taken: 0.148 seconds, Fetched: 5 row(s) hive> hive> CREATE TABLE page_visits AS > SELECT > page, > COUNT(*) AS visits > FROM weblog > GROUP BY page > ORDER BY visits DESC; Query ID = root_20250708183002_ec44d1b4-af24-403c-bb67-380dfb6961c3 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0002, Tracking URL = http://node:8088/proxy/application_1751961655287_0002/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0002 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:12,692 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:30:16,978 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.8 sec 2025-07-08 18:30:23,184 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.66 sec MapReduce Total cumulative CPU time: 3 seconds 660 msec Ended Job = job_1751961655287_0002 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0003, Tracking URL = http://node:8088/proxy/application_1751961655287_0003/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0003 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:30:35,969 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:30:41,155 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.23 sec 2025-07-08 18:30:46,313 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 2.95 sec MapReduce Total cumulative CPU time: 2 seconds 950 msec Ended Job = job_1751961655287_0003 Moving data to directory hdfs://node:9000/hive/warehouse/page_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.66 sec HDFS Read: 12379 HDFS Write: 251 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 2.95 sec HDFS Read: 7308 HDFS Write: 150 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 610 msec OK Time taken: 46.853 seconds hive> hive> describe page_visits; OK page string visits bigint Time taken: 0.214 seconds, Fetched: 2 row(s) hive> CREATE TABLE ip_visits AS > SELECT > ip, > COUNT(*) AS visits > FROM weblog > GROUP BY ip > ORDER BY visits DESC; Query ID = root_20250708183554_da402d08-af34-46f9-a33a-3f66ddd1a580 Total jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0004, Tracking URL = http://node:8088/proxy/application_1751961655287_0004/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0004 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:04,037 Stage-1 map = 0%, reduce = 0% 2025-07-08 18:36:09,250 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.57 sec 2025-07-08 18:36:14,393 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 3.3 sec MapReduce Total cumulative CPU time: 3 seconds 300 msec Ended Job = job_1751961655287_0004 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1751961655287_0005, Tracking URL = http://node:8088/proxy/application_1751961655287_0005/ Kill Command = /home/hadoop/hadoop3.3/bin/mapred job -kill job_1751961655287_0005 Hadoop job information for Stage-2: number of mappers: 1; number of reducers: 1 2025-07-08 18:36:27,073 Stage-2 map = 0%, reduce = 0% 2025-07-08 18:36:31,215 Stage-2 map = 100%, reduce = 0%, Cumulative CPU 1.25 sec 2025-07-08 18:36:36,853 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 3.27 sec MapReduce Total cumulative CPU time: 3 seconds 270 msec Ended Job = job_1751961655287_0005 Moving data to directory hdfs://node:9000/hive/warehouse/ip_visits MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 3.3 sec HDFS Read: 12445 HDFS Write: 216 SUCCESS Stage-Stage-2: Map: 1 Reduce: 1 Cumulative CPU: 3.27 sec HDFS Read: 7261 HDFS Write: 129 SUCCESS Total MapReduce CPU Time Spent: 6 seconds 570 msec OK Time taken: 44.523 seconds hive> [root@node weblog-mapreduce]# hive> [root@node weblog-mapreduce]# describe ip_visite; bash: describe: command not found... [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = 57dafc2a-afe2-41a4-8159-00f8d44b5add Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive Session ID = f866eae4-4cb4-4403-b7a2-7a52701c5a74 Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. hive> describe ip_visite; FAILED: SemanticException [Error 10001]: Table not found ip_visite hive> describe ip_visits; OK ip string visits bigint Time taken: 0.464 seconds, Fetched: 2 row(s) hive> SELECT * FROM page_visits; OK /index.html 2 /product.html 1 /product.htm 1 /checkout.html 1 /cart.html 1 Time taken: 2.095 seconds, Fetched: 5 row(s) hive> SELECT * FROM ip_visits; OK 192.168.1.2 2 192.168.1.1 2 192.168.1.4 1 192.168.1.3 1 Time taken: 0.176 seconds, Fetched: 4 row(s) hive> hive> [root@node weblog-mapreduce]# [root@node weblog-mapreduce]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 48 Server version: 8.0.42 MySQL Community Server - GPL Copyright (c) 2000, 2025, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CREATE DATABASE IF NOT EXISTS weblog_db; Query OK, 1 row affected (0.06 sec) mysql> USE weblog_db; Database changed mysql> CREATE TABLE IF NOT EXISTS page_visits ( -> page VARCHAR(255), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.05 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | page_visits | +---------------------+ 1 row in set (0.00 sec) mysql> DESCRIBE page_visits; +--------+--------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+--------------+------+-----+---------+-------+ | page | varchar(255) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+--------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> CREATE TABLE IF NOT EXISTS ip_visits ( -> ip VARCHAR(15), -> visits BIGINT -> ) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected, 1 warning (0.02 sec) mysql> SHOW TABLES; +---------------------+ | Tables_in_weblog_db | +---------------------+ | ip_visits | | page_visits | +---------------------+ 2 rows in set (0.01 sec) mysql> DESC ip_visits; +--------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+-------------+------+-----+---------+-------+ | ip | varchar(15) | YES | | NULL | | | visits | bigint | YES | | NULL | | +--------+-------------+------+-----+---------+-------+ 2 rows in set (0.00 sec) mysql> ^C mysql> [root@node weblog-mapreduce]# hive SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Hive Session ID = f34e6971-71ae-4aa5-aa22-895061f33bdf Logging initialized using configuration in jar:file:/home/hive-3.1.3/lib/hive-common-3.1.3.jar!/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Hive Session ID = f7a06e76-e117-4fbb-9ee8-09fdfd002104 hive> DESCRIBE FORMATTED page_visits; OK # col_name data_type comment page string visits bigint # Detailed Table Information Database: default OwnerType: USER Owner: root CreateTime: Tue Jul 08 18:30:47 CST 2025 LastAccessTime: UNKNOWN Retention: 0 Location: hdfs://node:9000/hive/warehouse/page_visits Table Type: MANAGED_TABLE Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"} bucketing_version 2 numFiles 1 numRows 5 rawDataSize 70 totalSize 75 transient_lastDdlTime 1751970648 # Storage Information SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe InputFormat: org.apache.hadoop.mapred.TextInputFormat OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Compressed: No Num Buckets: -1 Bucket Columns: [] Sort Columns: [] Storage Desc Params: serialization.format 1 Time taken: 1.043 seconds, Fetched: 32 row(s) hive> 到这里就不会了 6.2.2sqoop导出格式 6.2.3导出page_visits表 6.2.4导出到ip_visits表 6.3验证导出数据 6.3.1登录MySQL 6.3.2执行查询

大家在看

recommend-type

商品条形码及生产日期识别数据集

商品条形码及生产日期识别数据集,数据集样本数量为2156,所有图片已标注为YOLO txt格式,划分为训练集、验证集和测试集,能直接用于YOLO算法的训练。可用于跟本识别目标相关的蓝桥杯比赛项目
recommend-type

7.0 root.rar

Android 7.0 MTK MT8167 user 版本root权限修改,super权限修改,当第三方APP想要获取root权限时,会弹出窗口访问是否给与改APP root权限,同意后该APP可以得到root权限,并操作相关内容
recommend-type

RK3308开发资料

RK3308全套资料,《06 RK3308 硬件设计介绍》《07 RK3308 软件方案介绍》《08 RK3308 Audio开发介绍》《09 RK3308 WIFI-BT功能及开发介绍》
recommend-type

即时记截图精灵 v2.00.rar

即时记截图精灵是一款方便易用,功能强大的专业截图软件。   软件当前版本提供以下功能:   1. 可以通过鼠标选择截图区域,选择区域后仍可通过鼠标进行边缘拉动或拖拽来调整所选区域的大小和位置。   2. 可以将截图复制到剪切板,或者保存为图片文件,或者自动打开windows画图程序进行编辑。   3. 保存文件支持bmp,jpg,png,gif和tif等图片类型。   4. 新增新浪分享按钮。
recommend-type

WinUSB4NuVCOM_NUC970+NuWriter.rar

NUC970 USB启动所需的USB驱动,已经下载工具NuWriter,可以用于裸机启动NUC970调试,将USB接电脑后需要先安装WinUSB4NuVCOM_NUC970驱动,然后使用NuWriter初始化硬件,之后就可以使用jlink或者ulink调试。

最新推荐

recommend-type

C#类库封装:简化SDK调用实现多功能集成,构建地磅无人值守系统

内容概要:本文介绍了利用C#类库封装多个硬件设备的SDK接口,实现一系列复杂功能的一键式调用。具体功能包括身份证信息读取、人证识别、车牌识别(支持臻识和海康摄像头)、LED显示屏文字输出、称重数据读取、二维码扫描以及语音播报。所有功能均被封装为简单的API,极大降低了开发者的工作量和技术门槛。文中详细展示了各个功能的具体实现方式及其应用场景,如身份证读取、人证核验、车牌识别等,并最终将这些功能整合到一起,形成了一套完整的地磅称重无人值守系统解决方案。 适合人群:具有一定C#编程经验的技术人员,尤其是需要快速集成多种硬件设备SDK的应用开发者。 使用场景及目标:适用于需要高效集成多种硬件设备SDK的项目,特别是那些涉及身份验证、车辆管理、物流仓储等领域的企业级应用。通过使用这些封装好的API,可以大大缩短开发周期,降低维护成本,提高系统的稳定性和易用性。 其他说明:虽然封装后的API极大地简化了开发流程,但对于一些特殊的业务需求,仍然可能需要深入研究底层SDK。此外,在实际部署过程中,还需考虑网络环境、硬件兼容性等因素的影响。
recommend-type

基于STM32F1的BLDC无刷直流电机与PMSM永磁同步电机源码解析:传感器与无传感器驱动详解

基于STM32F1的BLDC无刷直流电机和PMSM永磁同步电机的驱动实现方法,涵盖了有传感器和无传感两种驱动方式。对于BLDC电机,有传感器部分采用霍尔传感器进行六步换相,无传感部分则利用反电动势过零点检测实现换相。对于PMSM电机,有传感器部分包括霍尔传感器和编码器的方式,无传感部分则采用了滑模观测器进行矢量控制(FOC)。文中不仅提供了详细的代码片段,还分享了许多调试经验和技巧。 适合人群:具有一定嵌入式系统和电机控制基础知识的研发人员和技术爱好者。 使用场景及目标:适用于需要深入了解和实现BLDC和PMSM电机驱动的开发者,帮助他们掌握不同传感器条件下的电机控制技术和优化方法。 其他说明:文章强调了实际调试过程中可能遇到的问题及其解决方案,如霍尔传感器的中断触发换相、反电动势过零点检测的采样时机、滑模观测器的参数调整以及编码器的ABZ解码等。
recommend-type

基于Java的跨平台图像处理软件ImageJ:多功能图像编辑与分析工具

内容概要:本文介绍了基于Java的图像处理软件ImageJ,详细阐述了它的跨平台特性、多线程处理能力及其丰富的图像处理功能。ImageJ由美国国立卫生研究院开发,能够在多种操作系统上运行,包括Windows、Mac OS、Linux等。它支持多种图像格式,如TIFF、PNG、GIF、JPEG、BMP、DICOM、FITS等,并提供图像栈功能,允许多个图像在同一窗口中进行并行处理。此外,ImageJ还提供了诸如缩放、旋转、扭曲、平滑处理等基本操作,以及区域和像素统计、间距、角度计算等高级功能。这些特性使ImageJ成为科研、医学、生物等多个领域的理想选择。 适合人群:需要进行图像处理的专业人士,如科研人员、医生、生物学家,以及对图像处理感兴趣的普通用户。 使用场景及目标:适用于需要高效处理大量图像数据的场合,特别是在科研、医学、生物学等领域。用户可以通过ImageJ进行图像的编辑、分析、处理和保存,提高工作效率。 其他说明:ImageJ不仅功能强大,而且操作简单,用户无需安装额外的运行环境即可直接使用。其基于Java的开发方式确保了不同操作系统之间的兼容性和一致性。
recommend-type

MATLAB语音识别系统:基于GUI的数字0-9识别及深度学习模型应用 · GUI v1.2

内容概要:本文介绍了一款基于MATLAB的语音识别系统,主要功能是识别数字0到9。该系统采用图形用户界面(GUI),方便用户操作,并配有详尽的代码注释和开发报告。文中详细描述了系统的各个组成部分,包括音频采集、信号处理、特征提取、模型训练和预测等关键环节。此外,还讨论了MATLAB在此项目中的优势及其面临的挑战,如提高识别率和处理背景噪音等问题。最后,通过对各模块的工作原理和技术细节的总结,为未来的研究和发展提供了宝贵的参考资料。 适合人群:对语音识别技术和MATLAB感兴趣的初学者、学生或研究人员。 使用场景及目标:适用于希望深入了解语音识别技术原理的人群,特别是希望通过实际案例掌握MATLAB编程技巧的学习者。目标是在实践中学习如何构建简单的语音识别应用程序。 其他说明:该程序需要MATLAB 2019b及以上版本才能正常运行,建议使用者确保软件环境符合要求。
recommend-type

c语言通讯录管理系统源码.zip

C语言项目源码
recommend-type

Teleport Pro教程:轻松复制网站内容

标题中提到的“复制别人网站的软件”指向的是一种能够下载整个网站或者网站的特定部分,然后在本地或者另一个服务器上重建该网站的技术或工具。这类软件通常被称作网站克隆工具或者网站镜像工具。 描述中提到了一个具体的教程网址,并提到了“天天给力信誉店”,这可能意味着有相关的教程或资源可以在这个网店中获取。但是这里并没有提供实际的教程内容,仅给出了网店的链接。需要注意的是,根据互联网法律法规,复制他人网站内容并用于自己的商业目的可能构成侵权,因此在此类工具的使用中需要谨慎,并确保遵守相关法律法规。 标签“复制 别人 网站 软件”明确指出了这个工具的主要功能,即复制他人网站的软件。 文件名称列表中列出了“Teleport Pro”,这是一款具体的网站下载工具。Teleport Pro是由Tennyson Maxwell公司开发的网站镜像工具,允许用户下载一个网站的本地副本,包括HTML页面、图片和其他资源文件。用户可以通过指定开始的URL,并设置各种选项来决定下载网站的哪些部分。该工具能够帮助开发者、设计师或内容分析人员在没有互联网连接的情况下对网站进行离线浏览和分析。 从知识点的角度来看,Teleport Pro作为一个网站克隆工具,具备以下功能和知识点: 1. 网站下载:Teleport Pro可以下载整个网站或特定网页。用户可以设定下载的深度,例如仅下载首页及其链接的页面,或者下载所有可访问的页面。 2. 断点续传:如果在下载过程中发生中断,Teleport Pro可以从中断的地方继续下载,无需重新开始。 3. 过滤器设置:用户可以根据特定的规则过滤下载内容,如排除某些文件类型或域名。 4. 网站结构分析:Teleport Pro可以分析网站的链接结构,并允许用户查看网站的结构图。 5. 自定义下载:用户可以自定义下载任务,例如仅下载图片、视频或其他特定类型的文件。 6. 多任务处理:Teleport Pro支持多线程下载,用户可以同时启动多个下载任务来提高效率。 7. 编辑和管理下载内容:Teleport Pro具备编辑网站镜像的能力,并可以查看、修改下载的文件。 8. 离线浏览:下载的网站可以在离线状态下浏览,这对于需要测试网站在不同环境下的表现的情况十分有用。 9. 备份功能:Teleport Pro可以用来备份网站,确保重要数据的安全。 在实际使用此类工具时,需要注意以下几点: - 著作权法:复制网站内容可能侵犯原作者的版权,因此在使用此类工具时,必须确保有合法权利去下载和使用目标网站的内容。 - 服务条款:许多网站的服务条款明确禁止未经授权的网站克隆。因此,在使用此类软件之前,应当仔细阅读并遵守目标网站的服务条款。 - 数据隐私:下载含有个人数据的网站可能触及隐私保护法律,特别是在欧洲通用数据保护条例(GDPR)等法规的环境下。 - 网络安全:随意下载网站可能导致恶意软件和病毒的传播,用户应当使用最新的反病毒软件,并在安全的环境中进行操作。 综上所述,虽然“复制别人网站的软件”在技术上是可行的,且可以用于多种正当目的,但在使用此类工具时,必须遵守法律法规和道德规范,不可侵犯他人的合法权益。
recommend-type

【跨平台开发者的必读】:解决Qt5Widgetsd.lib目标计算机类型冲突终极指南

# 1. 跨平台开发与Qt框架简介 跨平台开发已经成为现代软件开发中不可或缺的一部分,它使得开发者能够构建出能够在多种操作系统上运行的应用程序,从而触及更广泛的用户群体。在这其中,Qt框架作为一套成熟的跨平台应用程序和用户界面框架,扮演着关键角色。Qt不仅仅是一个跨平台的工具集,它的设计哲学还包含了模块化和可扩展性,使其能够适应各种规模和复杂度的项目。本章节我们将简要介绍
recommend-type

普通RNN结构和特点

<think>首先,用户的问题是:“我想了解普通RNN的结构及其特点请问RNN循环神经网络结构特点工作原理”。这涉及到RNN的结构、特点和工作原理。根据系统级指令:-所有行内数学表达式必须使用$...$格式。-独立公式必须使用$$...$$格式并单独成段。-LaTeX语法正确。-使用中文回答。-生成相关问题。-回答中引用的段落末尾自然地添加引用标识。用户可见层指令:-回答结构清晰,帮助用户逐步解决问题。-保证回答真实可靠。参考站内引用:-引用[1]:关于RNN的基本介绍,为什么需要RNN。-引用[2]:关于RNN的工作原理、结构图,以及与其他网络的比较。用户上一次的问题和我的回答:用户是第一次
recommend-type

探讨通用数据连接池的核心机制与应用

根据给定的信息,我们能够推断出讨论的主题是“通用数据连接池”,这是一个在软件开发和数据库管理中经常用到的重要概念。在这个主题下,我们可以详细阐述以下几个知识点: 1. **连接池的定义**: 连接池是一种用于管理数据库连接的技术,通过维护一定数量的数据库连接,使得连接的创建和销毁操作更加高效。开发者可以在应用程序启动时预先创建一定数量的连接,并将它们保存在一个池中,当需要数据库连接时,可以直接从池中获取,从而降低数据库连接的开销。 2. **通用数据连接池的概念**: 当提到“通用数据连接池”时,它意味着这种连接池不仅支持单一类型的数据库(如MySQL、Oracle等),而且能够适应多种不同数据库系统。设计一个通用的数据连接池通常需要抽象出一套通用的接口和协议,使得连接池可以兼容不同的数据库驱动和连接方式。 3. **连接池的优点**: - **提升性能**:由于数据库连接创建是一个耗时的操作,连接池能够减少应用程序建立新连接的时间,从而提高性能。 - **资源复用**:数据库连接是昂贵的资源,通过连接池,可以最大化现有连接的使用,避免了连接频繁创建和销毁导致的资源浪费。 - **控制并发连接数**:连接池可以限制对数据库的并发访问,防止过载,确保数据库系统的稳定运行。 4. **连接池的关键参数**: - **最大连接数**:池中能够创建的最大连接数。 - **最小空闲连接数**:池中保持的最小空闲连接数,以应对突发的连接请求。 - **连接超时时间**:连接在池中保持空闲的最大时间。 - **事务处理**:连接池需要能够管理不同事务的上下文,保证事务的正确执行。 5. **实现通用数据连接池的挑战**: 实现一个通用的连接池需要考虑到不同数据库的连接协议和操作差异。例如,不同的数据库可能有不同的SQL方言、认证机制、连接属性设置等。因此,通用连接池需要能够提供足够的灵活性,允许用户配置特定数据库的参数。 6. **数据连接池的应用场景**: - **Web应用**:在Web应用中,为了处理大量的用户请求,数据库连接池可以保证数据库连接的快速复用。 - **批处理应用**:在需要大量读写数据库的批处理作业中,连接池有助于提高整体作业的效率。 - **微服务架构**:在微服务架构中,每个服务可能都需要与数据库进行交互,通用连接池能够帮助简化服务的数据库连接管理。 7. **常见的通用数据连接池技术**: - **Apache DBCP**:Apache的一个Java数据库连接池库。 - **C3P0**:一个提供数据库连接池和控制工具的开源Java框架。 - **HikariCP**:目前性能最好的开源Java数据库连接池之一。 - **BoneCP**:一个高性能的开源Java数据库连接池。 - **Druid**:阿里巴巴开源的一个数据库连接池,提供了对性能监控的高级特性。 8. **连接池的管理与监控**: 为了保证连接池的稳定运行,开发者需要对连接池的状态进行监控,并对其进行适当的管理。监控指标可能包括当前活动的连接数、空闲的连接数、等待获取连接的请求队列长度等。一些连接池提供了监控工具或与监控系统集成的能力。 9. **连接池的配置和优化**: 连接池的性能与连接池的配置密切相关。需要根据实际的应用负载和数据库性能来调整连接池的参数。例如,在高并发的场景下,可能需要增加连接池中连接的数量。另外,适当的线程池策略也可以帮助连接池更好地服务于多线程环境。 10. **连接池的应用案例**: 一个典型的案例是电商平台在大型促销活动期间,用户访问量激增,此时通用数据连接池能够保证数据库操作的快速响应,减少因数据库连接问题导致的系统瓶颈。 总结来说,通用数据连接池是现代软件架构中的重要组件,它通过提供高效的数据库连接管理,增强了软件系统的性能和稳定性。了解和掌握连接池的原理及实践,对于任何涉及数据库交互的应用开发都至关重要。在实现和应用连接池时,需要关注其设计的通用性、配置的合理性以及管理的有效性,确保在不同的应用场景下都能发挥出最大的效能。
recommend-type

【LabVIEW网络通讯终极指南】:7个技巧提升UDP性能和安全性

# 摘要 本文系统介绍了LabVIEW在网络通讯中的应用,尤其是针对UDP协议的研究与优化。首先,阐述了UDP的原理、特点及其在LabVIEW中的基础应用。随后,本文深入探讨了通过调整数据包大小、实现并发通信及优化缓冲区管理等技巧来优化UDP性能的LabVIEW方法。接着,文章聚焦于提升UDP通信安全性,介绍了加密技术和认证授权机制在LabVIEW中的实现,以及防御网络攻击的策略。最后,通过具体案例展示了LabVIEW在实时数据采集和远程控制系统中的高级应用,并展望了LabVIEW与UDP通讯技术的未来发展趋势及新兴技术的影响。 # 关键字 LabVIEW;UDP网络通讯;性能优化;安全性;