Monday, March 23, 2099

Greetings :)

Ave.

For those of you that know me, you'd know I like to email you with things I'm thinking on that I think you might find interesting or helpful.

So, rather than fill up your inbox with long winded thoughts, how-tos and tips and tricks, I thought I'd put it I'll in a blog instead!

This is really just so I have all my tech notes in one place but given that everyone tends to work on similar things at similar times I figured this was a good way to get some info sharing happening...

As I'm a network engineer, the focus will primarily be on routers, switches, firewalls, wireless, UC etc. but will occasionally wander into server / virtualisation land (gasp) as there is admittedly a fair bit of cross-over these days...

Other sections and random off-topic things will pop up as well just for something different :)

I'll try to include a quick video for in-depth posts that provides a overview of the topic wherever possible and please feel free to leave comments on each topic (I'll try and reply to them whenever possible).


If you find this page useful I strongly encourage you to create you own blog page as well!

Tuesday, April 25, 2017

NBN - How to Avoid a Crap Merchant RSP (A work in progress)

Introduction

Sooo.

You're a Government.
And one day in 1999 you are sitting around going, hmmm, we're rather crap at managing a budget and we need some money.
I know! Let's sell off our telecomms infrastructure (because that never ends badly)!
So you privatise your telecoms and who would have thought - it gets turned into a maximum profit, minimum effort outsourced pile of rubbish.
The solution? Let's build a NEW government telcomms network!
Except instead of controlling the whole lot and making sure it's a good quality service, let's leave the messy, risky stuff like supporting customers and making sure there's enough backhaul bandwidth to any business who wants to play and not hold them to any performance standards at all!
And it shall be known as NBN. 

I know all that sounds a bit harsh, but tell me why I'm wrong :p

My ADSL Experience with Exetel

As a bit of a history lesson, I'll give you some insight into the service I've come from.

I've been using an ISP called Exetel for the better part of 10 years.

Exetel used to be a great ISP (in that you got the performance that you paid for) run by a guy called John Linton who sadly passed away in 2012.

Exetel's concept was that they were an ISP for techies with very little support (with the idea that you should be able to figure things out for yourself) with the added bonus that the provided 1:1 backhaul contention ratios, meaning you got what you paid for.

Here I was in probably 2010 and was using an Optus resold connection through Exetel paying $45 / month for a full performance ADSL2+ connection.

That's right - I was sitting there getting ~20Mbps down. Any time of the day. Any file from any country, transferred at my sync rate.

You could tell Exetel was run by true engineers through their open publishing of MRTG graphs demonstrating the usage of their peer and backhaul links as follows (with some captures of the old login pages shown to demostrate the difference between how they were then and, well, take a look at the site now):









Unfortunately, following Johns passing, profit margins seem to have taken priority over network performance and the things that set Exetel apart such as 1:1 subscription ratios and visibility into network performance have disappeared.


My NBN Experience with Exetel

Woohoo. NBN is coming to my house!

It's 2017 and all this money has been spent by the Government so I must be getting an AWESOME connection that will absolutely fly.


I look at the RSP websites and they're all very glossy, filling me with hope that this is a polished operation.

The woes of yesteryear of different ISPs providing widely varying performance for services such as ADSL are behind us, right?


Well, here I sit on a supposed 25Mbps down / 5 Mbps up plan from Exetel (FTTN VDSL2 - the most common form of NBN delivery).

I just did an Ookla speed test to a Brisbane based server at 8:30pm on Saturday 23/04/17.

To make things a bit more scientific, I ran up SNMP monitoring of my routers WAN interface to capture the overall traffic and make sure something else on my network isn't chomping my bandwidth.

As they say, a picture is worth a thousand words.




Here we have a graph showing the WAN bandwidth. It averages it out so you don't see the speedtest but you get the general idea that the WAN interface isn't flogged.



 

Here's the super crap results of the bandwidth test.
Exetel - hang your head in shame.
Remember - this is LOCAL traffic within AU.




For good measure, just showing the sync rate and actual rate of the VDSL connection.

Let's try an international download:





73.3 KB/sec.
That's 0.58 Mbps for those of you playing at home.



So, just to recap, it's 2017 and I'm now paying $79 / month for an NBN connection which performs at only 10% of the (download) speed my service which cost $45 / month back in 2010.

Rather than read online chatter, I thought I'd take this monster apart and pinpoint where the issue is and what my options are to make my NBN connection perform like it's supposed to.



NBN Design and RSP Responsibilities Overview
When you hear "NBN" it immediately conjures up ideas of a giant Australia wide network all owned and built by the Australian Government.

Unfortunately, the reality is that the Government (NBN Co.) only really a portion of the network, specifically:
  • Purchase of last mile delivery infrastructure from Telstra (copper and HFC).
  • Installation of POI infrastructure
  • Blueprint for how RSPs provide core connectivity for NBN provided customers.
NBN performance pain points in detail - Down the rabbit hole we go!


On-Net and Off-Net POI

Domestic RSP Performance (RSP Peering)

International RSP Performance (RSP Backhaul) 

I did my homework and changed RSPs - now my connection works correctly.


How to Avoid Crap Merchant RSPs

 

RSPs - What they can do to PROVE they're not oversubscribing customers.
 

Thursday, March 30, 2017

NfSen on CentOS 7



Ahhh Netflow (and sFlow for that matter).

That magic service that let's you see what's happening on your routers WAN interface.

For new players, there's a fantastic open source product called NfSen that collects Netflow data and provides a GUI with input box to allow you to bring up flow information as and when required (going back as far as your storage can handle).

Now, like all Open Source products, they can be a bit of a PITA to setup.

I've sat down and followed three different guides to get this working on CentOS 7 and found one that was 99.9% there.

The guide I recommend is over at ProLinuxHub @
http://prolinuxhub.com/building-centos-7-netflows-monitoring-station-with-nfsen-and-nfdump/

Follow that guide to the tee with the following changes and you'll be ready to rock n roll:



Extra Packages
On the line that says:
 yum install perl-Data-Dumperu

Change this to
 yum install perl-Data-Dumper

NfSen.conf Settings File
On the line that says:
 $HTMLDIR = "/var/www/nfsen";

Change this to:
 $HTMLDIR = "/var/www/html/nfsen";


Final note:

If you see the following message when you hit up your NfSen URL @ http://x.x.x.x/nfsen/nfsen.php

 Frontend - Backend version missmatch!

Edit /var/www/html/nfsen/nfsen.php

Comment out the line:
 if ( !array_key_exists('backend_version', $_SESSION ) || $_SESSION['backend_version'] !=  $expected_version ) {

And enter directly below:
 if ( array_key_exists('backend_version', $_SESSION ) && $_SESSION['backend_version'] !=  $expected_version ) {

 Save the file.

Restart NfSen:
 /etc/init.d/nfsen restart

Tuesday, November 8, 2016

Dovado 4G Routers

Yeah yeah yeah.
I know what you're thinking...

USB routers suck.
And I used to agree.

But given that service providers in Australia seem to be spinning up services on new frequencies (e.g. Band 28 / 700MHz LTE) on a regular basis, you can understand why USB 4G modems are becoming popular.

I started thinking, that's all great but when you need to share the 4G connection to other devices over WiFi or Ethernet, then you need to get one of those horrible routers that works with the USB modem until it locks up, forgets it has a USB port etc. etc.

If only someone made a router specifically designed to work with USB modems that had some sort of technology to detect when the USB modem locks up and intelligently reboots the USB port to re-establish connectivity.

Well, that day has come.

Enter Dovado.

http://www.dovado.com/en/products



Cisco 3850 QoS Hardmax and Softmax Buffers

Here's a fun one kiddies.

If you're working with a Cisco 3850, I strongly recommend you check out Cisco's page they've created especially to explain how QoS buffers work on this particular model as it's different to anything I've ever worked with (for better or worse).

And of course, with excellent names like "hardmax" and "softmax" buffers I had to put together a little article picture :)



Check it out at:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

Sunday, April 24, 2016

Media Center on Windows 10 Fixes (DVB-T, Scheduled Recording and ehrecvr.exe Fix)

Gday again.

Well the time has hit to use pleb TV again in Australia (thanks Masterchef) so I thought I'd get good old Media Center going on Windows 10.

As it turns out, Media Center is no longer part of Win10 by default.

Luckily, you can still manually install it.
Head to the http://lifehacker.com/get-windows-media-center-running-on-windows-10-with-a-f-1729919907 for more details.

Once installed, you'll hit a few small issues.
Below is a list of these issues and how to resolve them.


Live TV Doesn't Work (Codec Issue)
By default Windows 10 doesn't have the necessary codecs to decode a DVB-T stream.

This is easily fixed by downloading and installing the LAV filters pack available at:
https://github.com/Nevcairiel/LAVFilters/releases

Recordings Don't Start (Permissions Issue)
It appears Media Center uses task scheduler to manage recordings.
Win 10 uses different privileges for different users and it appears the standard account doesn't have rights to create scheduled tasks.

This is easily fixed by changing the program shortcut settings under the compatibility tab to make sure that the program is always run as an administrator.

Machine Unresponsive (CPU gets hammered by recording service)
It appears the gem that has plagued media center from day one still exists.

Anyone who has used MediaCenter has probably experienced the issue where you get 100% CPU usage on all cores by a little bastard process called ehrecvr.exe.
As an added bonus, it also sets it's priority to above normal, which makes it even more prone to CPU chomping.

This is the service used by Media Center to record shows, so  it's sort of important.
It's clearly a bug that has never been resolved by MS.

Luckily, there's absolutely no need for this service to require that much CPU so we can limit how many cores it can use (one) and set its priority to normal (instead of all).

Note - through my testing, you have to have ehrecvr bound to core 1.
You can give it more than one core, but one of the cores must be core 1.
If you don't, recordings won't start.

To bring this sucker under control, first you need the ehrecvr service to start.
The only way to do this without confusing Media Center is to let Media Center be the reason it starts.

I put a shortcut to Media Center (set to run as administrator) in my Startup folder to make sure it kicks off at boot.

Next, create a batch file with the following lines and save it somewhere:

wmic process where name="ehrecvr.exe" CALL setpriority "normal"
PowerShell "$Process = Get-Process ehrecvr.exe; $Process.ProcessorAffinity=1"

Note - the processor affinity value indicates which cores to use.
For this exercise, one core is fine but if you want you can give it two cores.

Values are as follows:
Core 1 Only: 1
Core 1 and 2: 3
Core 1 and 3: 5
Core 1 and 4: 9

Next, create a scheduled task with the following settings.

General Tab: 
Run whether user is logged on or not: Checked
Run with highest privileges: Checked

Triggers:
Begin the Task: At Startup
Delay Task For: 1 Minute

Actions:
Point it to your batch file.

And that's it :)

You should now have a full functioning install of Windows Media Center on Windows 10 that doesn't cause weird performance issues.

Enjoy :)