Import lumberjack events manually with stdin

A typical install of logstash-forwarder (lumberjack) is configured to watch a set of files in specific locations and often playing with that file is impossible. However, you might need to load a file into it that it doesn’t typically monitor.

In another situation, you may need to load historic logfiles into LSF. This can be problematic as LSF keeps track of its position in a given file and will often recognise the file as one it has already processed and won’t  reimport events it considers as “old”.

So here is a quick way of getting events in without interrupting your log shipping.

  1. Create a new config file somewhere where the user you run LSF as can read it e.g. /etc/logstash-forwarder/temp.conf
  2. Add a bare-bones config with your remote server and a single stdin input:
      "network": {
        "servers": [ "" ],
        "ssl certificate": "/etc/logstashforwarder/ssl/logstashforwarder.crt",
        "ssl ca": "/etc/logstashforwarder/ssl/ca.crt",
        "ssl key": "/etc/logstashforwarder/ssl/logstashforwarder.key",
        "timeout": 15
      "files": [
          "paths": [ "-" ],
          "fields": { "type": "nginx" }
  3. cat your logfile into a new instance of LSF with the config above like so:cat /var/log/nginx/temp/server.access | /opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash-forwarder/temp.conf -spool-size 100 -log-to-syslog
  4. You can watch syslog to see if your events are being shipped by tailing syslog.
  5. ???
  6. Profit.

You can shut down the temp instance once the flood of events dies down.


Import lumberjack events manually with stdin was originally published on antispin

Casual Love


Friends, put on your flak jackets. It’s time to drop some honesty on yet another uncomfortable topic: love.
We use the word “love” to mean a lot of things. Throughout this post I’ll be referring to the romantic kind of love, the kind that usually involves sexual attraction, AKA “falling in…

Tips for setting up your home NAS

After a conversation with a friend about the best way to go about setting up a NAS for your home, I thought I’d relate some of the things I’ve learned over the few years that I’ve been working with ZFS at home and at work.


If you’re going to be building a homebrew NAS box to use as your media/file storage, streaming server and porn dump, you’ll need to think about what sort of kit you need. I’m going to be assuming an OpenIndiana-based setup similar to what I currently run.


Buy a HP ProLiant Microserver Gen8. The Gen7 are also decent bits of kit, but these units support more memory, have a faster CPU and are better-built and generally More Better.


You need a couple things:

  • A decent CPU. It doesn’t need to be blazing-fast since it’ll be doing mostly a lot of nothing 95% of the time. A moderately cheap i3 or i5 will probably do the trick.
  • RAM. Lots of RAM. As much as you can afford/the motherboard will support/you think is necessary. 8GB minimum. The reason for this is that ZFS loves RAM. The more memory it has, the more data it can load into RAM, meaning faster IO for you.
  • Disks. You want to get disks with as good IO performance as you can, since having disks that have shitty IOPS will make your NAS crawl.
  • SATA/SAS controller. Don’t buy a RAID card. You won’t need it. ZFS paired with a hardware RAID card can result in epic data corruption as the RAID card actually interferes with ZFS’ own data integrity magic and can lead to hilariously broken arrays. If you have a RAID card, set it to JBOD mode or the equivalent. ZFS needs direct access to the disks without a RAID controller being in the way.


I personally run OpenIndiana on my MicroServer and it’s been rock-solid for years now. I’d recommend it. You can get it here. Don’t waste a whole disk for installing it on, however, especially if you have a limited number of disks you can use.

The beauty of ZFS-based systems is that the OS is actually trivial. You care about your data, primarily. If the OS dies, you can reinstall, import your pool and you’re back up. With that in mind, I would install the OS on a USB stick or SDCard on your motherboard, if it supports it. This way you have the maximum number of available disks to use for storage.

Disk Config

Here, you have two options:

Use a set of mirrors.

                              capacity     operations    bandwidth
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
virt                       1.89T  3.54T      0  1.60K  15.3K  51.3M
  mirror                    324G   604G      0    267      0  8.27M
    c4t5000C50041FDDFEBd0      -      -      0    147      0  8.27M
    c4t5000C50041FDE0DFd0      -      -      0    146      0  8.27M
  mirror                    323G   605G      0    168      0  7.78M
    c4t5000C50041FDEA1Fd0      -      -      0     88      0  7.78M
    c4t5000C50041FDEB2Fd0      -      -      0     87      0  7.78M
  mirror                    323G   605G      0    217      0  9.10M
    c4t5000C500426D38D3d0      -      -      0    118      0  9.10M
    c4t5000C500426D3417d0      -      -      0    118      0  9.10M
  mirror                    323G   605G      0    206  15.3K  7.60M
    c4t5000C50057F93F57d0      -      -      0     80      0  7.60M
    c4t5000C50057FA2D6Fd0      -      -      0     83  15.3K  7.60M
  mirror                    323G   605G      0    295      0  7.69M
    c4t5000C50057FA2DC3d0      -      -      0    127      0  7.69M
    c4t5000C50057FA28C3d0      -      -      0    133      0  7.69M
  mirror                    323G   605G      0    226      0  7.11M
    c4t5000C50057FA273Fd0      -      -      0    142      0  7.11M
    c4t5000C50057FA2847d0      -      -      0    143      0  7.11M

This sort of setup gives you expandability further down the line, so if you decide you only need 4TB of storage, you can create a stripe of two mirrors of 2TB disks initially. If you decide that you need to add an extra 2TB of storage, you simply add a third mirror of 2TB disks, etc, etc. This gives you great flexibility and decent read/write performance to boot.

Use a RAIDZn

If you aren’t worried about your IO performance being blazing fast, you can instead opt for a RAIDZ1 or RAIDZ2which is the equivalent of a RAID5/RAID6. If you have a limited number of usable disks – for price or other reasons, this is often a good alternative. It will still provide you with some redundancy – you can lose a disk and keep going, providing you replace the failed drive sooner rather than later.

The drawback is that you can’t expand a RAIDZ pool as easily as ZFS doesn’t support adding disks to a RAIDZ. You can increase the size of the array by sequentially swapping disks out for larger ones – e.g. swapping your 2TB disks for 4TB disks – and letting the pool resilver (rebuild) on to the new drives. Once all disks are replaced, the pool will expand to the full capacity of your drives.

Hope this helps answer some questions about how best to set up your home NAS

Tips for setting up your home NAS was originally published on antispin

ELK Stack Retrospective

For the past six months or so, I’ve been running an ELK stack setup in our hosting infrastructure at work to monitor, among other things:

  • HTTP requests coming in
  • Nginx response times
  • System loads
  • Sendmail and Postfix activity
  • Disk IO and related metrics

To do this, I’ve had to evolve the infrastructure somewhat. Here’s a brief overview of what happened.


  • Logstash was installed on a single box using its built-in Elasticsearch server to store data.
  • Each host I wanted to monitor had a local instance of Logstash installed on it and configured manually to send events to the indexer.
  • The indexer processed the events and stored them in ES.


I quickly realised that this wouldn’t work in the long-term as event rates increased and a single box couldn’t handle the load of indexing and storing the logs in ES. So, I

  • Commissioned a new, dedicated ES instance to store the data and added it to the cluster that Logstash was running on its own.
  • Once the shards had replicated, I shut down the LS internal Elasticsearch instance and reconfigured LS to write to the dedicated ES cluster (of one device)
  • In addition, I replaced the Logstash instances on the client nodes with a lighter-weight Lumberjack/Logstash-forwarder instance. LSF is built in Go and will run on any platform with minimal requirements, unlike Logstash proper which needs a JVM and is much more memory-intensive.


The setup above was good, but it lacked two things that would be needed in a production environment:

  1. Resilience/availability
  2. Scalability

The main problem was that while the lumberjack instances would queue up events to a certain point, restarting the main Logstash indexer process would cause lost events and load would jump through the roof as soon as the LS process restarted.

Logstash is capable of using Redis or another message queue to handle variable event rates, so I set up a “logstash relay” box which ran a simple, lightweight instance of LS with minimal configuration (no event processing, just forwarding) which dumped everything into Redis and LS1 (the indexer) would then connect to the Redis instance and grab events off the queue to process them.

This system worked pretty well for a time until several lumberjack/LSF agents went crazy and started dumping millions of events into the queue from old log files that they decided to parse. This was a Bad Thing because Redis writes its queue to disk by default and when it runs out of disk space, there is no easy way (that I knew of at the time) to throttle incoming events. This ends up crashing the Redis instance and stops the relay dead.

So, some further re-architecture was done.


The current incarnation of the ELK stack looks like this

The main change was that I replaced Redis with RabbitMQ. RMQ is much “heavier” than Redis, but it is also more manageable, configurable and, crucially, it will look after itself and ensure that if it runs out of memory and/or disk space, upstream clients will be blocked or throttled until such a time that disk space falls below the warning threshold and events can be written to the queue once more.

This means that the LS relay instance can be made to not accept events from client nodes because it’s aware that it’s being throttled by RMQ. This wasn’t the case with Redis, as LS was attempting to blindly dump events into Redis and failing.

Performance Notes

Here are some recommendations for building out a resilient ELK stack

  • Memory. ES loves memory. The more RAM you give you ES instances, the better they’ll perform.
  • Set your Java heapsize to 50% of available RAM for ES – so if you have 24GB RAM, set the heap size to 12GB.
  • Storage: the faster your disks, the better ES does, especially as event rates climb and it’s having to index several thousand records a second.
  • CPU. You need lots of CPU and as little contention as possible. Depending on just how much processing you’re doing using Grok and other filters, the LS workers can become saturated and if your CPU is being stressed already, LS is likely to lock up and fail very un-gracefully.
  • You will need to balance fine-grained analysis of your events with CPU use.
  • Playing with the number of workers that LS starts can give you a boost in event rate at the cost of using more CPU. To set this, edit /etc/default/logstash (in Debian) and set the following:
    LS_OPTS="-w 6"

    Change 6 to whatever number of workers you want. You can experiment with this value to find the sweet spot between event rate and CPU use.

  • You can also use this command to get a good overview of what LS is up to.
    alias lstop='top -Hp `cat /var/run/`'

    Once you run that, you’ll be able to issue ‘lstop’ from your comandline and get a quick view of all the LS processes. If you begin to see many |worker processes constantly in bold and with high CPU use, you’re looking at the CPU being saturated and may need to back off the filtering or the number of workers or both

Hopefully this helps those of you who are playing with LS and are looking to improve your infrastructure. Let me know in the comments if you have any ideas of your own.

ELK Stack Retrospective was originally published on antispin

Multiple Steam Library Folders

Steam, Valve’s content delivery network, games marketplace and cloud savefile storage system, among other things has supported having multiple install locations for your games library.

As SSDs become more and more prevalent and cheaper, many people are looking to install their stuff on the much faster SSD.

To create a new installation location for future installs – this article doesn’t cover moving your existing Steam library – simply open your Steam settings, hit the “Downloads” tab and click “Steam Library Folders”

From here, you can click “Add Library Folder” and specify the location of your new library.

However – and this took me about a half-hour of searching – you can’t add a new location while a game is updating or a download is taking place. To get around this just pause all your downloads!

Multiple Steam Library Folders was originally published on antispin

Wordpress woes

It would seem that my wordpress instance over at is flipping the fuck out and isn’t posting things and images and stuff as intended. 

When’s Ghost 0.5+ coming out? I want to get off the horrible lumbering behemoth that WP has become. 

Also, it would seem that not using Varnish is actually beneficial to my infrastructure…

A fantastic take on the classic euphoric “crowd-pleaser” track, with a bunch of glitch thrown in. Strangers has been out for a while now and is my absolute favorite song to play live. Working with Myon & Shane 54 was huge for me as I have been a fan of theirs for quite a long time. Tove is doing BIG things right now and we were lucky to be able to work with her when we did! Keep an eye out for her ;)

New EP Worlds Apart available now on Casablanca Records.

Follow Seven Lions:


New Bonobo! ‘Duals’ is a new Bonobo track taken from the Ten Tigers EP, available on 12” and download, in stores now: Bonobo Store / iTunes
i. Ten Tigers (Bengal Edit)
ii. Duals
iii. First Fires (Maya Jane Coles remix)

If you buy a ticket from the Bonobo store for the final show on The North Borders Tour at Alexandra Palace then you will get a free download of Duals. Buy your ticket here
If you’ve already bought a ticket for the show, then you can expect to find Duals in your inbox today.

Watch the trailer for the new live album and DVD The North Borders Tour - Live. :











No, I’d say the show does a great job of representing the typical 18-34 male with Larry, with his constant need for validation, attention, and the world to revolve around him.


I wouldn’t care if there wasn’t a single damned man on the fucking show.

Are you fucking serious?

You have entire boxsets filled with men, straight white male only casts running 16 seasons deep

There’s me thinking its going to say Asian people, but nah, we forgot The Men.

But what about the menzzzzzzzzzzzzzzz?????????????!!!!!!!!????????

"Important group" lol don’t flatter yourselves

(via youbestnotmiss)