More Go Playing

I've been getting more into building compiled apps thanks to how easy I'm finding the Go programming language and the Fyne library for cross-platform GUI development. For a while I've been frustrated with the state of the art for free Apache log analysers. There's the old faithful AWStats and Webalizer... but I'm finding their layout and information a bit... old. I'd like something new but I'm a hobbyist and I'm not going to pay for Splunk or similar just to get fast log searching. So why not build my own?1

With a week away with the family in the middle, I've spent some time now playing with building a Go+Fyne+MySQL log file visualiser. So far I've got the following breakdown:

  • Digest: Parse through log files (currently in Apache log format) and process them into a standard struct layout
  • Decypher: Figure out what the log files mean for data recording and splinter them out into summary tables, perform IP->Country mapping, etc.
  • Display: Visualise end point.

3D visualising! Marketable.

I've spent a non-insignificant amount of time playing with SVG world maps to create a visualisation of the top, near top, and also-ran of countries that visit my site. Wikipedia has a few good worldmaps with CC0 licensing2, and there was a great one with country labels in SVG format. Scalable! Unfortunately, it was also processor heavy in Fyne so I ended up grabbing a simpler CC0 map I could translate into a PNG and then laboriously figuring out the x/y locations of all the countries using the original SVG. Owie. On the plus side, I did somewhat extend the original SVG by wrapping the <path> elements for countries with multiple paths in <g> tags so you could use country code IDs for finding their respective <path> or <g> in the code for styling. A real shame it was too slow. Oh well.

Aside: A mysql trick

The MySQL section was also entertaining, figuring out partitioning and best indexing; culminating in the previous MySQL section - but I did find something else. When trying to do the IP address -> Country map I used a database of ip min/max ranges to country. The integer value of an ip address is the first number * 256 * 256 * 256, second number * 256 * 256, third number * 256 and last number as it is - sum them all together to get a unique number. When running a select on that table to get a between, though, it wasn't hitting indexes and was slow. The trick is to use LIMIT 1 on the statement and it hits the indexes. The explain plan proved it so... wow. Weird.

Anyway, still having a lot of fun coding this and figuring out little best practices for my app. I'm enjoing adding a status bar to the bottom of all the GUI apps I'm building that show progress, status, little notes and that sort of thing. Useful space, that footer.

  1. Because there are tonnes of other tools that do this. And better. With more time and dedication. Hoy. ↩︎

  2. This is a personal "thing" so I again find it hard to justify paying for a licensed world map, but I'm not going to use that to justify stealing one. ↩︎