准备
环境
- CDH 5.5.2, Parcel
- HDFS
- YARN
- Zookeeper
- Kafka
- Spark (on YARN)
下载最新版本
-
进入下载页面,分别下载
- compute-classpath.sh
- oryx-batch-2.1.2.jar
- oryx-run.sh
- oryx-serving-2.1.2.jar
- oryx-speed-2.1.2.jar
-
下载conf文件
-
以ALS为例,在源码目录oryx/app/conf下als-example.conf
-
并将其重命名为oryx.conf(注:文件需要和每个层的JAR文件放在同一个目录下)
-
修改conf文件
-
# 现阶段只需要修改这么几个字段就OK了 kafka-brokers = "data-mining-46.slave:9092,data-mining-47.slave:9092,data-mining-48.slave:9092,data-mining-49.master:9092" zk-servers = "data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka" hdfs-base = "hdfs:///Oryx"
oryx { id = "ALSExample" als { rescorer-provider-class = null } input-topic { broker = ${kafka-brokers} lock = { master = ${zk-servers} } ...开始
[root@data-mining-49 oryx]# ./oryx-run.sh kafka-setupInput ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-46.slave:9092,data-mining-47.slave:9092,data-mining-48.slave:9092,data-mining-49.master:9092 topic OryxInputUpdate ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-46.slave:9092,data-mining-47.slave:9092,data-mining-48.slave:9092,data-mining-49.master:9092 topic OryxUpdate
All available topics:
Input topic OryxInput does not exist. Create it? yCreating topic OryxInputError while executing topic command : org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids[2016-04-08 16:53:09,383] ERROR org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =NoNode for /brokers/ids at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:995) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:671) at kafka.utils.ZkUtils.getChildren(ZkUtils.scala:537) at kafka.utils.ZkUtils.getSortedBrokerList(ZkUtils.scala:172) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:243) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:107) at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) at kafka.admin.TopicCommand.main(TopicCommand.scala)Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:114) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:678) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:985) ... 8 more (kafka.admin.TopicCommand$)Status of topic OryxInput:yUpdate topic OryxUpdate does not exist. Create it?Creating topic OryxUpdateError while executing topic command : org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids[2016-04-08 16:53:11,094] ERROR org.I0Itec.zkclient.exception.ZkNoNodeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:995) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.getChildren(ZkClient.java:671) at kafka.utils.ZkUtils.getChildren(ZkUtils.scala:537) at kafka.utils.ZkUtils.getSortedBrokerList(ZkUtils.scala:172) at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:243) at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:107) at kafka.admin.TopicCommand$.main(TopicCommand.scala:60) at kafka.admin.TopicCommand.main(TopicCommand.scala)Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1496) at org.I0Itec.zkclient.ZkConnection.getChildren(ZkConnection.java:114) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:678) at org.I0Itec.zkclient.ZkClient$4.call(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:985) ... 8 more (kafka.admin.TopicCommand$)Error while executing topic command : Topic OryxUpdate does not exist on ZK path data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka[2016-04-08 16:53:11,833] ERROR java.lang.IllegalArgumentException: Topic OryxUpdate does not exist on ZK path data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:119) at kafka.admin.TopicCommand$.main(TopicCommand.scala:62) at kafka.admin.TopicCommand.main(TopicCommand.scala) (kafka.admin.TopicCommand$)Status of topic OryxUpdate:
[root@data-mining-49 oryx]#此时出现了一个问题:
Error while executing topic command : org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/ids这个问题应该是:CDH没有启动Kafka服务没有启动。通过CDH后台启动Kafka服务(如果没有添加Kafka服务,则需要先添加)。

此时重新执行setup命令:
[root@data-mining-49 oryx]# ./oryx-run.sh kafka-setupInput ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-47.slave:9092,data-mining-48.slave:9092 topic OryxInputUpdate ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-47.slave:9092,data-mining-48.slave:9092 topic OryxUpdate
All available topics:
Input topic OryxInput does not exist. Create it? yCreating topic OryxInputCreated topic "OryxInput".Status of topic OryxInput:Topic:OryxInput PartitionCount:4 ReplicationFactor:1 Configs: Topic: OryxInput Partition: 0 Leader: 140 Replicas: 140 Isr: 140 Topic: OryxInput Partition: 1 Leader: 141 Replicas: 141 Isr: 141 Topic: OryxInput Partition: 2 Leader: 140 Replicas: 140 Isr: 140 Topic: OryxInput Partition: 3 Leader: 141 Replicas: 141 Isr: 141
Update topic OryxUpdate does not exist. Create it? yCreating topic OryxUpdateCreated topic "OryxUpdate".Updated config for topic "OryxUpdate".Status of topic OryxUpdate:Topic:OryxUpdate PartitionCount:1 ReplicationFactor:1 Configs:retention.ms=86400000,max.message.bytes=16777216 Topic: OryxUpdate Partition: 0 Leader: 140 Replicas: 140 Isr: 140
[root@data-mining-49 oryx]#OK!
需要将如上的二进制脚本都同步到其他集群的节点上了
[root@data-mining-49 data]# scp -r oryx/ root@192.168.1.48:/data/compute-classpath.sh 100% 1893 1.9KB/s 00:00oryx-run.sh 100% 13KB 13.2KB/s 00:00oryx-batch-2.1.2.jar 100% 26MB 26.0MB/s 00:00oryx-serving-2.1.2.jar 100% 33MB 33.0MB/s 00:01oryx-speed-2.1.2.jar 100% 26MB 26.0MB/s 00:00oryx.conf 100% 1884 1.8KB/s 00:00接下来我们需要找一个可用的数据集:
将这个数据集的格式修改下:
[root@data-mining-49 ml-100k]# tr '\t' ',' < u.data > data.csv[root@data-mining-49 ml-100k]# tail data.csv806,421,4,882388897676,538,4,892685437721,262,3,877137285913,209,2,881367150378,78,3,880056976880,476,3,880175444716,204,5,879795543276,1090,1,87479579513,225,2,88239915612,203,3,879959583[root@data-mining-49 ml-100k]#数据准备完毕,我们可以通过
[root@data-mining-49 oryx]# ./oryx-run.sh kafka-input --input-file data.csv命令导入系统, 同时打开另一个终端窗口,通过命令
[root@data-mining-49 oryx]# ./oryx-run.sh kafka-tailInput ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-47.slave:9092,data-mining-48.slave:9092 topic OryxInputUpdate ZK data-mining-46.slave:2181,data-mining-47.slave:2181,data-mining-48.slave:2181/kafka Kafka data-mining-47.slave:9092,data-mining-48.slave:9092 topic OryxUpdate
279,64,1,875308510646,750,3,888528902654,370,2,887863914617,582,4,883789294913,690,3,880824288660,229,2,891406212421,498,4,892241344495,1091,4,888637503806,421,4,882388897676,538,4,892685437721,262,3,877137285913,209,2,881367150378,78,3,880056976880,476,3,880175444716,204,5,879795543276,1090,1,87479579513,225,2,88239915612,203,3,879959583实时跟踪输入导入情况,当数据全部导入完毕后,用户可以手动的通过Ctrl-C关闭这个命令。
如果以上全部成功了,可以关闭这些进程。集群至此已经准备好运行Oryx了。
启动服务
./oryx-run.sh batch./oryx-run.sh speed./oryx-run.sh servingblablabla…..好多输出,先不要管他。
确认服务层启动成功以后,可以导入数据(难道刚才导数据是测试用的?)
wget --quiet --post-file data.csv --output-document - --header "Content-Type: text/csv" http://192.168.1.49:8080/ingest批处理层已经开始触发一个新的计算了。
此时看看HDFS中的数据是这样的:
[root@data-mining-49 oryx]# hdfs dfs -ls /OryxFound 2 itemsdrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:40 /Oryx/datadrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:40 /Oryx/model^[[A[root@data-mining-49 oryx]# hdfs dfs -ls /Oryx/dataFound 1 itemsdrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:40 /Oryx/data/oryx-1460112000000.data[root@data-mining-49 oryx]# hdfs dfs -ls /Oryx/modelFound 3 itemsdrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:41 /Oryx/model/.checkpointdrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:40 /Oryx/model/.temporarydrwxr-xr-x - hdfs supergroup 0 2016-04-08 18:40 /Oryx/model/1460112018440[root@data-mining-49 oryx]#表明数据已经ok了,通过API确认计算是否完成
[root@data-mining-49 ~]# wget --quiet --output-document - --server-response http://192.168.1.49:8080/ready HTTP/1.1 200 OK Content-Length: 0 Date: Fri, 08 Apr 2016 10:43:27 GMT Server: OryxOK!
我们来查看下推荐结果
[root@data-mining-49 ~]# wget --quiet --output-document - http://192.168.1.49:8080/recommend/1750,0.8234792673029006127,0.7889597890898585181,0.7442612769082189275,0.7263787714764476258,0.7164891492575407121,0.700639194343239115,0.697495789732784288,0.6787936894688755285,0.669647410511970525,0.6603603088587988[root@data-mining-49 ~]# wget --quiet --output-document - http://192.168.1.49:8080/recommend/806173,1.08470471354667164,1.01426794147118937,0.9939510896801949183,0.991823517833836422,0.964462065137922869,0.945814429782331202,0.942016797140240711,0.9270770070143044135,0.9227914617804345191,0.9215727103874087[root@data-mining-49 ~]#完美结束。
这貌似是走的批处理逻辑,中途实时计算层挂鸟
16/04/08 18:42:38 INFO SparkContext: Successfully stopped SparkContext16/04/08 18:42:38 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, data-mining-47.slave): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0 at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1177) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0 at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138) at scala.Option.getOrElse(Option.scala:120)咱们下一篇再找找原因吧。