Hi experts,

I have attached to this post dataset
sample.txt

And I am trying to extract some association rules using Spark Mllib:
val transactions = sc.textFile("DATA")

import org.apache.spark.mllib.fpm.AssociationRules

import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset

val freqItemsets = transactions.map(_.split(",")).flatMap(xs => (xs.combinations(1) ++ xs.combinations(2) ++ xs.combinations(3) ++ xs.combinations(4) ++ xs.combinations(5)).filter(_.nonEmpty).map(x => (x.toList, 1L)) ).reduceByKey(_ + _).map{case (xs, cnt) => new FreqItemset(xs.toArray, cnt)}

val ar = new AssociationRules().setMinConfidence(0.8)

val results = ar.run(freqItemsets)results.collect().foreach { rule => println("[" + rule.antecedent.mkString(",") + "=>" + rule.consequent.mkString(",") + "]," + rule.confidence)}

However, my code returns a dozen rules with confidence equal to 1 ... which makes little sense!

Does anyone know if I lack some parameterization?

sample.txt
(4.6 kB)

Comment

Best Answer

**Answer** by jfrazee ·

@Pedro Rodgers Three things jump out at me:

- There are a ton of examples in your sample data with just one or two occurrences of the pattern. For this particular algorithm and its usual application they're not going to be very useful.
- Your confidence is quite high, considering the size of the sample data and the evidence for the different patterns.
- The learner came up with 1165 rules for 1185 data points.

I re-ran your code including a .filter(_._2 > 2) after the .reduceByKey(_ + _) and lowered the confidence to 0.6 and I get 20 or so rules now with confidence varying between 0.6 and 1.0.

I suspect if you carefully go through the results you were getting before you might see that it was just learning a 1-to-1 mapping between input and output, so the confidence of 1.0 is justified, but the generalization of the model is bad.

Hi jfrazee, Many thanks for your response :) I've some questions about this:

- 1) The structure of my data (Each line corresponds to a set of products_id) is correct to this algorithm?
- 2) The ".filter(_._2 > 2)" filter the products that have occurrence smaller than 2?
- 3) When I submit "val freqItemsets = transactions.map(_.split(",")).flatMap(xs => (xs.combinations(1) ++ xs.combinations(2) ++ xs.combinations(3) ++ xs.combinations(4) ++ xs.combinations(5)).filter(_.nonEmpty).filter(_._2 > 2).map(x => (x.toList, 1L)) ).reduceByKey(_ + _).map{case (xs, cnt) => new FreqItemset(xs.toArray, cnt)}" I'm getting the following error:
**<console>:31: error: value _2 is not a member of Array[String] .**Do you know how to solve it? Many thanks for your help and explaination about the association rules algorithm :) And sorry for this questions.

@Pedro Rodgers Sorry, I meant to write to put the .filter() after the .reduceByKey(). Edited it in the original answer. It should run now. Yes, it's filtering out/eliminating occurrences with counts less than or equal to 2. If your data/training time isn't too big, you can probably tune that and your confidence level empirically using a grid search.

@jfrazee The normal is to reduce / remove products with little occurrence, right? It is reasonable to think about eliminating the products that appear only in 20% of all transactions?

@jfrazee but I can define the cut-off the values that are higher than the average, right?

HCC Guidelines | HCC FAQs | HCC Privacy Policy | Privacy Policy | Terms of Service

© 2011-2019 Hortonworks Inc. All Rights Reserved.

Hadoop, Falcon, Atlas, Sqoop, Flume, Kafka, Pig, Hive, HBase, Accumulo, Storm, Solr, Spark, Ranger, Knox, Ambari, ZooKeeper, Oozie and the Hadoop elephant logo are trademarks of the Apache Software Foundation.

- Anonymous
- Login
- Create
- Ask a question
- Create Article
- Post Idea
- Add Repo
- Create SupportKB
- Tracks
- Community Help
- Cloud & Operations
- CyberSecurity
- Data Ingestion & Streaming
- Data Processing
- Data Science & Advanced Analytics
- Design & Architecture
- Governance & Lifecycle
- Hadoop Core
- Sandbox & Learning
- Security
- Solutions
- Explore
- All Tags
- All Questions
- All Articles
- All Ideas
- All Repos
- All SKB
- All Users
- Leaderboard
- All Badges