Skip to content

GitLab

S
spark_apps
Commit 3e8434ea authored by Sigmund Augdal's avatar Sigmund Augdal
Browse files
Options

Replace two maps and a union with a flatMap. Seems to be faster

parent 0ac1f5ac
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 3 deletions
pythonApp/netflowAlgs.py
View file @ 3e8434ea
...@@ -31,7 +31,9 @@ def ports_count_by_ip3(csv): ...@@ -31,7 +31,9 @@ def ports_count_by_ip3(csv):
def ports_count_by_ip(csv): def ports_count_by_ip(csv):
srcs = csv.map(lambda x: ((x[DEST_PORT], x[SRC_IP]), 1)) # srcs = csv.map(lambda x: ((x[DEST_PORT], x[SRC_IP]), 1))
dsts = csv.map(lambda x: ((x[DEST_PORT], x[DEST_IP]), 1)) # dsts = csv.map(lambda x: ((x[DEST_PORT], x[DEST_IP]), 1))
ips = srcs.union(dsts).reduceByKey(add) # ips = srcs.union(dsts).reduceByKey(add)
ips = csv.flatMap(lambda x: (((x[DEST_PORT], x[DEST_IP]), 1),
((x[DEST_PORT], x[SRC_IP]), 1))).reduceByKey(add)
return ips.map(lambda x: (x[1], x[0])).sortByKey(False).take(20) return ips.map(lambda x: (x[1], x[0])).sortByKey(False).take(20)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment