A fork of wasted/netflow to provide netflow parsing as a library on top of akka-stream. The original code was a standalone service using Netty. Since so much code was dependent on netty-buffer, it's still a dependency, but the network feature is now built on Alpakka/UDP.

Include in build:

libraryDependencies += "com.codemettle" %% "netflow-stream-lib" % "(version)"

Get a flow that parses tuples of InetSocketAddresses (representing the netflow source) and akka.util.ByteStrings as NetflowPackets:

package io.netflow {
  def netflowParser[In, Mat](
      incomingPackets: Flow[In, (InetSocketAddress, ByteString), Mat],
      v9TemplateDAO: NetFlowV9TemplateDAO
  )(implicit system: ActorSystem): Flow[In, FlowPacket, Mat]
}

Include in build:

libraryDependencies += "com.codemettle" %% "netflow-receiver" % "(version)"

Start a listener bound to the given interface / port which parses incoming UDP netflow packets and passes them along to the given flow:

import java.net.InetSocketAddress

import io.netflow.lib.FlowPacket
import io.netflow.NetFlowV9TemplateDAO

import com.codemettle.netflow.NetflowReceiver
import com.codemettle.streamutil.IngestingResult

implicit val system: ActorSystem = ???
val bindAddress: InetSocketAddress = ???
val packetHandler: Flow[FlowPacket, _, _] = ???
val templateDAO: NetFlowV9TemplateDAO = ???

val resultF: Future[IngestingResult] = NetflowReceiver(bindAddress, packetHandler, templateDAO)

// IngestingResult is one of:
// case object BindFailure
// case class OtherFailure(error: Throwable)
// case class Ingesting(boundTo: InetSocketAddress, ks: KillSwitch)
//
// Assuming everything was successful, the Ingesting.boundTo address will inform the caller
//  of what port was bound if a random (0) port was specified, and Ingesting.ks allows the
//  stream to be terminated.

=======

This project aims to provide an extensible flow collector written in Scala, using Netty as IO Framework. It is actively being developed by wasted.io, so don't forget to follow us on Twitter. :)

• NetFlow v1
• NetFlow v5
• NetFlow v6
• NetFlow v7
• NetFlow v9 (RFC3954) - options not handled yet

• NetFlow IPFIX/v10 (RFC3917, RFC3955)

Databases supported:

• Redis
• Apache Cassandra

• NetFlow v8 - totally weird format, would take too much time. Check yourself…

If we get a pull-request, we won't refuse it. ;)

Can both be found in the Issues section up top.

  ./sbt compile

  ./sbt run

First, setup Redis or Cassandra in the configuration file. After, start netflow to create the keyspace and required tables.

Go inside the project's directory and run the sbt command:

  ./sbt ~run

If you think it's ready for deployment, you can make yourself a .jar-file by running:

  ./sbt assembly

There are paramters for the configuration file and the logback config. To run the application, try something like ths:

java -server      								\
	-XX:+AggressiveOpts      					\
	-Dconfig.file=application.conf				\
	-Dlogback.configurationFile=logback.xml		\
	-Dio.netty.epollBugWorkaround=true			\ # only useful on Linux as it is bugged
	-jar netflow.jar

A more optimized version can be found in the run shellscript.

We are open to suggestions for some more optimal JVM parameters. Please consider opening a pull request if you think you've got an optimization.

Once it has successfully started, you can start adding authorized senders using the HTTP REST API. To do this, you need to use the admin.authKey and admin.signKey provided by the config. In case you haven't configured them, netflow.io will generate random-keys on every start. The authKey is like a public-key, the signKey more like a private key, so keep it safe.

In order to authenticate against the API, you have 2 possibilities:

• Generate a SHA256 HMAC of the authKey using the signKey
• Generate a SHA256 HMAC of the payload using the signKey (only if you have a payload)

To generate the SHA256 HMAC on your console:

echo -n "<authKey>" | openssl dgst -sha256 -hmac "<signKey>"

It's easiest to use like this:

export NF_AUTH_KEY='<authKey>'
export NF_SIGN_KEY='<signKey>'
export NF_SIGNED_KEY=$( echo -n "${NF_AUTH_KEY}" | openssl dgst -sha256 -hmac "${NF_SIGN_KEY}" )

This will setup the NetFlow sender 172.16.1.1 which is monitoring 172.16.1.0/24 and 10.0.0.0/24

curl -X PUT -d '{"prefixes":[{"prefix":"172.16.1.0","prefixLen":24}]}' \
    -H "X-Io-Auth: ${NF_AUTH_KEY}" \
    -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
    -v http://127.0.0.1:8080/senders/172.16.1.1

Of course we also support IPv6!

curl -X PUT -d '{"prefixes":[{"prefix":"2001:db8::","prefixLen":32}]}' \
    -H "X-Io-Auth: ${NF_AUTH_KEY}" \
    -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
    -v http://127.0.0.1:8080/senders/172.16.1.1

Please make sure to always use the first Address of the prefix (being 0 or whatever matches your lowest bit).

To remove a subnet from the sender, just issue a DELETE instead of PUT with the subnet you want to delete from this sender. This also works with multiples, just like PUT.

curl -X DELETE -d '{"prefixes":[{"prefix":"2001:db8::","prefixLen":32}]}' \
    -H "X-Io-Auth: ${NF_AUTH_KEY}" \
    -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
    -v http://127.0.0.1:8080/senders/172.16.1.1

To remove a whole sender, just issue a DELETE without any subnet.

curl -X DELETE \
    -H "X-Io-Auth: ${NF_AUTH_KEY}" \
    -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
    -v http://127.0.0.1:8080/senders/172.16.1.1

For a list of all configured senders

curl -X GET \
  -H "X-Io-Auth: ${NF_AUTH_KEY}" \
  -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
  -v http://127.0.0.1:8080/senders

For information about a specific sender

curl -X GET \
  -H "X-Io-Auth: ${NF_AUTH_KEY}" \
  -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
  -v http://127.0.0.1:8080/senders/172.16.1.1

Querying for statistics (more to come here)

curl -X POST -d '{
  "2001:db8::/32":{ 
    "years":["2013", "2014"], 
    "months":["2013-01", "2013-02"], 
    "days":["2014-02-02"], 
    "hours":["2014-02-02 05"],
    "minutes":[
      "2014-02-02 05:00",
      "2014-02-02 05:01",
      "2014-02-02 05:02",
      "2014-02-02 05:03",
      "2014-02-02 05:04"
    ]
  }}' \
  -H "X-Io-Auth: ${NF_AUTH_KEY}" \
  -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
  -v http://127.0.0.1:8080/stats/172.16.1.1

Querying for more detailed statistics (about dst AS 34868 and src AS 34868)

Tracked fields:

• all
• proto (6 for TCP, 17 for UDP, see /etc/protocols)
• srcip, srcport, srcas
• dstip, dstport, dstas

curl -X POST -d '{
  "2001:db8::/32":{ 
    "years":["2013", "2014"], 
    "months":["2013-01", "2013-02"], 
    "days":["2014-02-02"], 
    "hours":["2014-02-02 05"],
    "minutes":[
      {"2014-02-02 05:00": ["all", "dstas:34868", "srcas:34868"]},
      {"2014-02-02 05:01": ["all", "dstas:34868", "srcas:34868"]},
      {"2014-02-02 05:02": ["all", "dstas:34868", "srcas:34868"]},
      {"2014-02-02 05:03": ["all", "dstas:34868", "srcas:34868"]},
      {"2014-02-02 05:04": ["all", "dstas:34868", "srcas:34868"]}
    ]
  }}' \
  -H "X-Io-Auth: ${NF_AUTH_KEY}" \
  -H "X-Io-Sign: ${NF_SIGNED_KEY}" \
  -v http://127.0.0.1:8080/stats/172.16.1.1

NetFlow v9 and v10 (IPFIX) consist of two packet types, FlowSet Templates and DataFlows. Templates are defined on a per-router/exporter basis so each has their own. In order to work through DataFlows, you need to have received the Template first to make sense of the data. The issue is usually that your exporter might need a few minutes (10-60) to send you the according Template. If you use IPv4 and IPv6 (NetFlow v9 or IPFIX), the router is likely to send you templates for both protocols. If you want to know more about disecting NetFlow v9, be sure to check out RFC3954.

Basically the same as above, while your collector was down, your sender/exporter might have updated its template. If that happens, your netflow.io misses the update and cannot parse current packets. You will have to wait until the next template arrives.

We encourage everyone to use FreeBSD ng_netflow or OpenBSD pflow (which is a little bit broken in regards to exporting AS-numbers which are in the kernel through OpenBGPd). We advice against all pcap based exporters and collectors since they tend to drop long-living connections (like WebSockets) which exceed ~10 minutes in time.

Our suggestion would be to check your Switch's capability for port mirroring.

Mirror your upstream port to a FreeBSD machine which does the actual NetFlow collection and exporting.

This is also beneficial since the NetFlow collection does not impact your router's performance.

Not yet, but we are heavily developing towards our first stable public release!

We had it implemented into two classes before (LegacyFlow and TemplateFlow), but we were unhappy with "per-flow-version" debugging. We believe that handling each flow separately gives us more maintainability in the end than having lots of dispatching in between.

First rule for debugging: use tcpdump to verify you are receiving flows on your server.

If you need a little guidance for using tcpdump, here is what you do as root or with sudo:

# tcpdump -i <your ethernet device> host <your collector ip>

As a working example for Linux:

# tcpdump -i eth0 host 10.0.0.5

If you suspect the UDP Packet coming from a whole network, you can tell tcpdump to filter for it.

You might want to subtitute the default port 2055 with the port your netflow.io collector is running on.

# tcpdump -i eth0 net 10.0.0.0/24 and port 2055

Just grab the source-ip where packets are coming from and add it into the database.

By the way, tcpdump has an awesome manual!

  Copyright 2012, 2013, 2014 wasted.io Ltd <[email protected]>

  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.