Grok patterns in Logstash

After posting a set of postfix and sendmail Grok patterns, I’ve had a request to explain how to implement Grok patterns in Logstash. So, here we go.

I’m going to make some assumptions about your installation of Logstash – on a typical Debian install, assuming you installed from the Elasticsearch PPA (hint: you should be doing this), your Logstash configs will be in /etc/logstash/conf.d/ or something of the sort.

|/etc/
|-logstash/
|--conf.d/
|---inputs.conf
|---filters.conf
|---outputs.conf
|--patterns/
|---sendmail.grok
|---postfix.grok

As you can see, the Grok patterns go in /etc/logstash/patterns/ in my case. You can realistically put them anywhere you like, providing the logstash process can access the location.

Anyway, if you want to make use of any Grok pattern (assuming it’s valid), it’s a fairly straightforward operation:

filter {
  grok {
    patterns_dir => "/etc/logstash/patterns"
    pattern => "%{SENDMAIL}"
    named_captures_only => true
  }
}

The named_captures_only will tell Grok to only capture fields that are named in your Grok pattern and discard the cruft – useful when you want to extract a few key metrics from an event.

If you look at the Sendmail grok file I posted here, you can see how it’s being referenced above. This method is deprecated, but it still works for me, so that’s what I’m sticking with.

You can get creative and only invoke Grok for certain types of events by using an

if [type] == "something" {
  grok {
    patterns_dir => "/etc/logstash/patterns"
    pattern => "%{SOMEPATTERN}"
    named_captures_only => true
  }
}

Hope this helped!

Grok patterns in Logstash was originally published on antispin

Zimbra 8.5 EWS

Zimbra have recently released Zimbra Collaboration Suite 8.5 (JudasPriest).

One of the most-hyped features of 8.5 is native support for Exchange Web Services (EWS), meaning Macs and Outlook clients can natively connect to EWS instead of using the Zimbra Connector or IMAP.

After laying the groundwork for doing the upgrade – moving data around, system updates, installing the LTS Enablement pack from Canonical on Ubuntu 12.04 to upgrade the kernel among other things, we were ready for the upgrade to 8.5.

Watch out for Puppet

If your Zimbra box is managed by Puppet and if Puppet manages your /etc/sudoers file, you’ll want to stop the agent before running the upgrade. The reason for this is that 8.5 has additional commands that the Zimbra user needs to be able to run with elevated privileges. It’ll update your sudoers file as part of the upgrade, but there’s a danger that Puppet will roll it back which can seriously cock up your installation.

EWS and Touch Client

Do not work out of the box.

It turns out that if your license was purchased before the 1st of September 2014, you are not eligible to use the EWS functionality, despite enabling it in the CoS settings in Zimbra.

Needless to say, I am not currently best-pleased with Zimbra for effectively locking out users who have purchased perpetual licenses and a support pack which is supposed to provide access to upgrades. According to them, however, it seems that since EWS was not available as a feature at the time of purchase of the license, you need to pay extra to get it.

Anyone else having this problem with Zimbra? Anyone else been bitten by Zimbra’s sneaky pricing tactics?

Update

Zimbra have come back to us and have said the following:

Outlook for Mac is not an upgrade feature but an add-on.  Because there is a royalty associated with it, Zimbra chose to not increase prices for the product editions for everyone to accommodate that expense but rather to simply allow those companies who needed it, the ability to purchase it separately.  

Which is interesting, given that it’s described as a “New Feature” in their 8.5.0 upgrade guide

Update 2

Twitter user @jorgedlcruz has done a bit of asking around and has let me know the score.
Seems that EWS is an optional feature and the feature matrix reflects this. But it’s only available to resellers and internally.

Zimbra 8.5 EWS was originally published on antispin

We were grabbing a bite of lunch at a small cafe, in a mall, right across from a booth that sold jewelry and where ears could be pierced for a fee. A mother approaches with a little girl of six or seven years old. The little girl is clearly stating that she doesn’t want her ears pierced, that she’s afraid of how much it will hurt, that she doesn’t like earrings much in the first place. Her protests, her clear ‘no’ is simply not heard. The mother and two other women, who work the booth, begin chatting and trying to engage the little girl in picking out a pair of earrings. She has to wear a particular kind when the piercing is first done but she could pick out a fun pair for later.

"I don’t want my ears pierced."

"I don’t want any earrings."

The three adults glance at each other conspiratorially and now the pressure really begins. She will look so nice, all the other girls she knows wear earrings, the pain isn’t bad.

She, the child, sees what’s coming and starts crying. As the adults up the volume so does she, she’s crying and emitting a low wail at the same time. “I DON’T WANT MY EARS PIERCED.”

Her mother leans down and speaks to her, quietly but strongly, the only words we could hear were ‘… embarrassing me.’

We heard, then, two small screams, when the ears were pierced.

Little children learn early and often that ‘no doesn’t mean no.’

Little children learn early that no one will stand with them, even the two old men looking horrified at the events from the cafeteria.

Little girls learn early and often that their will is not their own.

No means no, yeah, right.

Most often, for kids and others without power, ”no means force.”

from "No Means Force" at Dave Hingsburger’s blog.

This is important. It doesn’t just apply to little girls and other children, though it often begins there.

For the marginalized, our “no’s” are discounted as frivolous protests, rebelliousness, or anger issues, or we don’t know what we’re talking about, or we don’t understand what’s happening.

When “no means force” we become afraid to say no.

(via k-pagination)

(via villiljos)

Oops. The internet ran out of routes

Yesterday, internet service providers around the world and the services that used their networks started going offline and experiencing abnormal levels of packet loss and latency. The issue was widespread and affected a great many services. What happened?

The internet ran out of routes.

Update: 

Here is a more thorough explanation of the issues of a couple of days ago: http://www.bgpmon.net/what-caused-todays-internet-hiccup/

The IPv4 public address space (the available publicly-routable IP addresses) has become more and more fragmented as companies have broken down the address blocks they own more and more and sold small chunks of their IPs on. Each distinct IP block that is connected to the internet needs to be routed to and from by routers all over the world and so requires an entry in the global BGP routing table. This is data stored by routers that ISPs operate.

Most routers, especially older pieces of hardware, have a limit of 512,000 odd routes that they can hold in their global routing table. This limit was mostly defined by the available memory the router had and when IPv4 was conceived, 512,000 routes seemed like a ludicrously high number.

As a new address block is carved off in the IPv4 address space, another route is advertised for that netblock. Yesterday, we exceeded the hard limit of 512k routes that most routers could hold.

Date Prefixes   CIDR Aggregated
06-08-14   511103   280424
07-08-14   511297   280432
08-08-14   511736   280442
09-08-14   511719   280722
10-08-14   511762   280563
11-08-14   511719   280860
12-08-14   511648   280869
13-08-14   512521   280918

Modern routers didn’t all have this limitation – in fact Cisco and other people posted an advisory about the impending issues that the growing number of IPv4 routes were going to cause – but routers have typically been deploy-and-forget devices that are set up and then run with minimal interaction. Older devices were mostly configured with a default of 512k. When the number of advertised routes exceeded 512k, in the words of redditor DiscoDave86,

“Upon further investigation it appears that the IPV4 public address space exceeded 512k routes, and some older routers have that as their limit due to memory constraints, consequently a whole load of routers became expensive doorstops”

The fix is simple enough: reconfigure the routing table limits on your router, but it requires a reboot of the device and rebooting a core router is not a task undertaken lightly.

Many ISPs have been scrambling to reconfigure their hardware to make sure they aren’t stung by this again, but the effects have been far-reaching as a result.

Impact

Update: I’ve drawn some quick & dirty nodegraphs to illustrate what happens when routers reboot.

In this (very simplistic) illustration of the Internet, Node 1 is trying to connect to Node 7. The bold path is the path its network traffic takes across the ‘net.

Everything is normal – traffic is routed according to the hop distance (fewest nodes to target). This isn’t always how it works in reality, but for the purposes of this example, it’ll do.

 Node 4′s administrator notices the problem, applies the fix and reboots the router, causing all routes that are using Node 4 to fail and have to be re-calculated.

 While Node 4 is rebooting, Node 8, which is operated by someone else, also starts to reboot to apply the changes to the maximum size of the routing table. N1 > N8 > N7 is no longer valid, so route is recalculated to N1 > N2 > N6 > N7
Nodes 4 and 8 are offline pending a reboot, so the path from N1 to N7 is routed through N2 and N6. Any addresses behind N4 and N8 are offline and become un-routable. It’s as though they no longer exist. 

N4 is back online but now has to re-create its routing table and only adds N1 and N7, so it can no longer route to N3 and N5

N8 is back online and starts to recreate its routing table, adding N1 and N4 as its available nodes.

After the nodes reboot, this is the final state of the network. As you can see, N4 and N8 have not got their original routes back, necessarily.

This is a very simplistic representation of what happens when you reboot a core router attached to the Internet, such as those that the likes of L3, AboveNet, TiNET, NTT operate. I haven’t included link costs in this diagram, either.

Last night, when many ISPs were doing this, entire blocks of addresses simply became un-routable. You didn’t get timeouts or dropped packets or lag. They just didn’t exist, for all the Internet was concerned.

Oops. The internet ran out of routes was originally published on antispin

Import lumberjack events manually with stdin

A typical install of logstash-forwarder (lumberjack) is configured to watch a set of files in specific locations and often playing with that file is impossible. However, you might need to load a file into it that it doesn’t typically monitor.

In another situation, you may need to load historic logfiles into LSF. This can be problematic as LSF keeps track of its position in a given file and will often recognise the file as one it has already processed and won’t  reimport events it considers as “old”.

So here is a quick way of getting events in without interrupting your log shipping.

  1. Create a new config file somewhere where the user you run LSF as can read it e.g. /etc/logstash-forwarder/temp.conf
  2. Add a bare-bones config with your remote server and a single stdin input:
    {
      "network": {
        "servers": [ "10.0.0.10:5043" ],
        "ssl certificate": "/etc/logstashforwarder/ssl/logstashforwarder.crt",
        "ssl ca": "/etc/logstashforwarder/ssl/ca.crt",
        "ssl key": "/etc/logstashforwarder/ssl/logstashforwarder.key",
        "timeout": 15
      },
      "files": [
        {
          "paths": [ "-" ],
          "fields": { "type": "nginx" }
        }
      ]
    }
  3. cat your logfile into a new instance of LSF with the config above like so:cat /var/log/nginx/temp/server.access | /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/temp.conf -spool-size 100 -log-to-syslog
  4. You can watch syslog to see if your events are being shipped by tailing syslog.
  5. ???
  6. Profit.

You can shut down the temp instance once the flood of events dies down.

Cheers!

Import lumberjack events manually with stdin was originally published on antispin